Presenting Predictive Models to Advisers

Iryna Johnson
Associate Director for Analytics, Institutional Research
Auburn University
 

This visual display is an interactive tool, implemented at Auburn to help advisers understand factors that put a student at risk of departure or low first semester GPA. Entering a student’s ID triggers populating the tool with student official data, responses from the Beginning College Survey of Student Engagement (BCSSE), and predicted first semester GPA and the probability of returning next fall. First semester GPA and the probability of return are calculated based on predictive models hidden behind the tool.

For most items on the tool, there are two values initially prepopulated from student records and survey responses. However, one of these values can be changed, while the other remains the same unless a user enters a different student ID. The predicted GPA and probability of return are calculated based on actual student records and responses. The adjusted GPA and probability of return are calculated based on values entered on the form. As long as the values on the form and the actual student records and responses match, the predicted and adjusted GPAs and probabilities of return are the same.

 View larger version of model (PDF opens in new tab). 

LgvddIJ.JPG
Once the values on the form are changed, the adjusted GPA and probability of return next fall will change accordingly, thus giving advisers an opportunity to see how a different response would have affected this student’s odds of success.

For example, if, instead of selecting “2” on a scale from 1=Not very certain to 6=Very certain, this student reported that she is certain she will finish something she has started, this would not change this student’s probability of coming back next fall, but it would increase this student’s expected fall 1 GPA from 3.26 to 3.36, as shown below.

MedvddIJ.JPG

Figure 2: Effect of change in the values on student outcomes
 

To evaluate an effect of a single variable, a user should make sure that the values of other variables on the form match actual student records and responses.

Because some first-time freshmen either did not complete BCSSE or did not provide sufficient identifying information, most of the values on the form are missing for these students. When an adviser enters an ID of such a student, the following warning appears in the top right corner of the form: “The student did not complete BCSSE/Match not found. Manipulating values of BCSSE variables will not work.”

The benefits of the form are twofold. First, it helps advisers get to know each of their students better by reviewing individual responses to the survey. Second, the interactivity of the form makes it easier to identify factors that put each student at risk of getting low grades or not returning next fall.

This tool might be helpful to institutional researchers who are struggling with presenting and explaining predictive models to non-statistical audiences.

Note: A paper describing this visual display and predictive models has been accepted for publication in the Proceedings of the 12th annual National Symposium on Student Retention, October-November, 2016.

 

 

 Comments

 
To add a comment, Sign In
Total Comments: 4
 
Cate posted on 12/19/2016 2:46 PM
Do the students know their responses will be used for this purpose when they fill out BCSSE? This could have a chilling effect if students assume their survey response is confidential.
Iryna posted on 12/21/2016 5:09 PM
Thank you for your comment, Cate. During BCSSE administration each student receives an Informed Consent Letter that states that her ID and name are requested so that her responses can be matched with the university records to provide individualized information to her academic advisor. Hope this addresses your concern.
Matthew posted on 1/3/2017 9:33 AM
What is the model fit and confidence interval of these predictions? Is it narrow enough to display as only a single point?
Iryna posted on 1/4/2017 10:58 AM
Thank you for your comment, Matthew. I agree that these predictions are far from perfect. For example, standard error of estimate for the linear regression used to predict fall GPA based on student records and BCSSE responses is 0.66. One can infer confidence intervals from this. (R Square for this model is 0.39.) Despite its limitations, identifying at-risk students is useful for early interventions. “Essentially, all models are wrong, but some are useful” (Box and Draper 1987).