Defining Student Success Beyond Traditional Measures

Ask eAIR invites questions from AIR members about the work of institutional research, careers in the field, and other broad topics that resonate with a large cross-section of readers. Questions may be submitted to eAIR@airweb.org

This month’s question is answered by Jim Woehrle, Associate Director of Institutional Effectiveness, Research and Planning, Antioch College.     

The ideas, opinions, and perspectives expressed are those of the author, and not necessarily AIR. Members are invited to join the discussion by commenting at the end of the article.

Dear Jim: How do you define and measure student success beyond the traditional graduation rate and retention rate measures?

JimW1.pngThat’s a great question, and I can use my own institution, Antioch College, to provide a detailed example. Antioch College is a small, private, liberal arts college located in Yellow Springs, Ohio. Antioch closed in 2008, and re-opened in 2011 as a non-successor institution to the previous iteration of Antioch. As such, Antioch College was technically a brand new institution, and was required to go through the process of initial accreditation with the Higher Learning Commission. This rigorous process entailed providing evidence that Antioch College was capable of fostering student success. However, as a new institution, the Office of Institutional Effectiveness and Research (IER) was lacking a common measures of student success, such as post-graduate outcomes and graduation rates. Additionally, longitudinal data on retention rates were not available and were limited to 2-3 cohorts. The onus was on the office of IER, then, to identify and utilize non-traditional measures of student success, including institutional surveys and course-level data from course evaluations and assessment of student learning.

Upon hiring an IER coordinator in early 2013, the College began regularly collecting data on first-year students through the Higher Education Research Institute’s (HERI) two first-year student surveys, The Freshman Survey (TFS) and the Your First College Year Survey (YFCY). These surveys allowed IER to gather data on academic behavior, goals, and expectations of incoming students (through TFS), as well as to assess the level of academic engagement that first-year students experienced during the initial year at Antioch College (through YFCY). The TFS provides a tool that predicts graduation rates based on survey data called the Expected Graduation Rate Calculator. This calculator uses demographic, high school, and academic behavior data to predict 4, 5, and 6-year graduation rates (see DeAngelo et. al., 2011 for a full description of the methodology underlying the calculator). This tool allowed for an empirically-based estimate of graduation rates for the 2014, 2015, and 2016 entering cohorts. For the previous cohorts, the office of IER utilized the basic version of the Expected Graduation Rate Calculator, which relies only on demographic and high-school academic (high-school GPA, standardized test scores) data to provide an estimation of anticipated graduation rates (The basic Expected Graduation Rate Calculator can be found here).

In addition, YFCY allows insight into the frequency at which students engage in academic behaviors that are correlated with objective student success outcomes, such as persistence to graduation and career outcomes. These behaviors include regularly asking questions in class, looking up research articles, and revising papers, among others (Note: these behaviors are included in the HERI construct “Habits of Mind”). Antioch student responses to the YFCY indicated that our students were likely to be successful in graduating from the institution as well as achieving desirable post-graduate outcomes.

Course-level data were also a valuable source of information regarding student behavior that underscores post-graduate success. Course evaluations allowed insight into the level of rigor that students perceived in their classes. Academic rigor and time spent engaged in academically rigorous tasks are significant predictors of student learning. Course evaluations also provided additional data regarding behaviors related to academic engagement, such as faculty-student interactions in and out of the classroom. Although IER was lacking in graduation rates and post-graduate outcomes, we had clear evidence that students were engaged in academically rigorous coursework. Also, course evaluations demonstrated that students were generally enjoying their classes and would recommend them to peers; a finding that is predictive of persisting at the institution.

Course-level data on assessment of student learning outcomes were also valuable in determining the likelihood of producing positive student success outcomes. Course-level assessment was required for every course at Antioch College during the period of applying for initial accreditation, and the data on student learning outcomes were useful in providing evidence that students were indeed learning in their courses. Working with the assessment committee, IER examined student learning outcomes related to written communication and compared outcomes across cohorts and classrooms. Several faculty members also utilized pre-/post-test designs when gathering student learning outcome data and demonstrated that students improved their scores in learning outcomes over the course of an academic term. Evidence that students were learning during their time at the College was an additional predictor that students were likely to advance towards graduation and be successful following graduation.

Despite lacking graduation rates, post-graduate outcomes, or longitudinal data on student retention, Antioch College was able to rely on alternative indictors and predictors of student success. Based on the gathered data, IER could determine the areas where Antioch students were excelling and areas where they needed more support. They also enabled us to have a good overall sense of likelihood of persistence to graduation and post-graduate success. These data were cited frequently as evidence in our accreditation argument and contributed to the HLC granting Antioch College early initial accreditation in the summer of 2016.

In conclusion, it can be beneficial to IER offices to look beyond the traditional measures of student success, and supplement these measures with other indicators to gain a richer understanding of student and institutional factors that underscore student success.

 

 Comments

 
To add a comment, Sign In
There are no comments.