Program Retention Report

​David Onder, Director of Assessment &
Alison Joseph, Business and Technology Applications Analyst
Western Carolina University

The Program Retention Report is a tool used by Western Carolina University (WCU) for tracking student outcomes by cohort within a program over time. To accomplish this, each cohort of students is followed within a given program, and at each interval a determination is made as to how many were graduated or retained within their program, how many were graduated or retained at the university but outside of their program, and how many were lost completely to attrition. The full report includes very detailed term-level data, which is then rolled up in progressively more condensed forms and finally graphed in comparison to university averages of programs at the same level. The graphs are inspired by the Voluntary System of Accountability's Student Success and Progress and are very similar to the new Student Achievement Measure. This report is built in Microsoft Excel so that it can be produced in a highly automated fashion year after year. A downloadable version and related PowerPoint presentation are available on WCU’s Office of Institutional Planning and Effectiveness website under Staff Presentations.

View Larger PDF Version


Please provide your comments on the visual display, below.

 

 Comments

 
To add a comment, Sign In
Total Comments: 18
 
Betsy posted on 9/11/2013 4:02 PM
The graph is clear and I like the color scheme. It very well for the average figures since they are more steady from year to year and you can easily see that those who leave mostly doe so in the first year.

For the specific program assessed, there are real issues with the tiny numbers, and the fact that probably a few people who weren't retained after 2 years seem to have returned for the third year. A shift in three people makes a big and perhaps meaningless bump in the graph when the cohort is this small. For large programs, I think this could be very effective, but for the smaller ones it may be a little less so.
John posted on 9/11/2013 4:18 PM
It isn't clear how you define "average graduate program." Since graduate and retention values are computed using cumulative counts, perhaps the name should be "nominal graduate program" or "all WCU programs".

Would it make sense to aggregate across all cohorts for both the focus program and WCU, and compare two bars (one for the program and one for WCU) for each year? It would help answer the question "Does this program retain or graduate students at rates different from the rest of WCU?"
Jon posted on 9/11/2013 5:15 PM
Great Job!I think the stacked bar chart is great for this application. I am not sure if the automation aspect would support labels in the bar charts, but that we make them clearer and reduce the back and forth between the table and the chart. You might also think about adding some source reference. Some indication of what we are looking at (Distinct student count of...)and from where the data is derived (Source: Colleague(Informer: Program Retention...)would be very helpful if people have questions about the information contained within the report.
Jeff posted on 9/11/2013 5:27 PM
For our college, we always display all time-related trend graphs to show time on the X-axis, and the dependent data that varies over time on the Y-axis. That makes it easier for users to figure out what your graph is all about. If you turn this page on its side, you can see that these are all trend graphs with associated trend tables that reveal how outcome mix effects vary and hopefully trend upward over time.

Another suggestion is to to use colors that imply positive versus negative outcomes. Graduation should be the darkest green, with retention being a lighter shade of green. There are slivers in the middle for non-program retention and graduation that might follow similar color shades. The Not Retained means those who left your school before they graduated. That should be in bloody red to show how much you hate to see that. I use shades of grey for ambiguous items -- not really applicable here.
Christinia posted on 9/11/2013 8:56 PM
The table should say "Number of Cohort Student" and not "New Cohort." A show of numbers for the Program Retention would have been better than percentages, because people could vividly see the true picture with numbers. The "Program Name" and the "Program Department" should have been more specific. I did not see the relevance to display WCU Retention percentages. The table shows the "Not Retained" percentages increasing except for one year, which denotes a concern.
Daina posted on 9/12/2013 6:44 AM
I agree with many of the comments already made. Usually admin is more comfortable with time/year on the horizontal axis and not the verical one. In addition, I would add the actual count to the graph - small N's could really skew the data presentation. the choice of colors is also odd, the eye is immediately drawn to the darkest color program graduation. Is that what is intended?
Ghenet posted on 9/12/2013 9:14 AM
The color scheme is good with the exception of WCU Retention color. It seems a little off in terms of contrast to the rest of the colors.
When we say @1 year, @2 years & @3 years, which one is the 100%, 150% or 200% of the expected graduation time because different programs might have different length of time? That means without the knowledge of the expected graduation time (or key to figures) it would be difficult to understand the visual information.
Generally, if we are dealing with such small numbers presenting count is preferred than creating graphs and percentages unless the numbers such as 16, 25, 18, 23 are representing thousands.
Gail posted on 9/12/2013 9:17 AM
Great comments! I concur that the bar graphs are very nice visuals and I really like how you have added WCU retention next to program retention. I have always put them on separate lines but I like your set up much better. It is great that you added the comparison numbers but I would prefer the percents to be computed using all graduate students rather than report the average program percents and if you do report the average is it a weighted average or unweighted?
I also noticed the decline in the % not retained which is confusing when you first see it. If some students have returned I would footnote how many.
Tim posted on 9/12/2013 9:55 AM
I think the colors and display of the information is good. I agree that the "Average Graduate Program" title is confusing and seems unclear for me. The symmetry of the specific program versus the university trends is easy to follow and to be able to compare.
Nikisha posted on 9/12/2013 1:38 PM
I think this is an excellent tool for tracking graduation and retention rates in a simple and easy to read format. However, I agree with Besty that this may not be the best tool for very small programs. In addition, I wonder if rotating to a portrait layout and putting the averages side by side with the corresponding program data would allow for easier comparisons.
Gary posted on 9/12/2013 1:50 PM
Great job, overall. I like the organization on the page and use of a single legend per row of plots. Titles appear to run together, so perhaps break them all as you did with "Not Retained." The stacked bars are sexy, but make it tough to visually compare everything except Program Retention. A dodged bar plot would resolve the issue, and make comparisons like John mentioned possible. Then you would have a little space to add counts as well. Still, it's a useful summary - thanks for sharing!
Terry posted on 9/12/2013 5:29 PM
Great! This should cut down on questions from the program people. If they can get themselves and understand it. That makes IR work that much easier.

Terry
Scott posted on 9/12/2013 11:12 PM
I find that this report works quite well. The tables give enough details for those who want specifics, and the graphs provide an effective summary of students' status at each point in time. I especially like how you've provided great comparison point in the lower half for every program figure in the upper half. Ordinarily I might have a minor quibble about using stacked bars, as sometimes it is hard to judge the proportions of the middle segments when the size of the end segments changes. But in this case, the categories are few enough that the graphs identify a variety of patterns that might not jump out from the tables alone.
Tim posted on 9/13/2013 11:51 AM
This visual does a nice job explaining what has been a tricky dynamic at our institution--making a clear distinction between retention within a program, and retention within the institution. It seems like it would work best with a relatively stable, cohort-based program. It might not work as well with a program like undergraduate behavioral science, where students are moving in and out of the program constantly for 4 years. For graduate programs it probably works well. It would be interesting to reorganize the bars to show one cohort per column, rather than per row. That would allow you to easily watch the progression of students to graduation looking down across bars. Of course, if the point is to compare cohorts at common points, the way it is currently organized works better.

I also have questions about how the "average graduate program" was defined--do all programs last exactly three years? If the purpose is really to compare with the institutional average, it may be worth moving those "average" bars up next to their "program" counter parts. (And visually distinguish them somehow--making them lighter, or giving them a background).
Overall, very effective and attractive. Especially considering it was built in Excel!
Teri posted on 9/13/2013 1:22 PM
This is a great display! I have one technical question about how/when you define your cohorts, though. That was always the tricky part for us - do you track students from when they first declare the major, which might be as a freshmen or might be as a junior, or do you set a "census" point - so whoever is a major at the beginning of their sophomore year is tracked? (And thanks for the shout-out for the VSA and SAM!)
Sam posted on 9/16/2013 11:59 AM
Only a few things I would suggest beyond those of my colleagues: 1) use % as column headers to cut down on the number of percent signs one has to see (also adding N below "New Cohort"; 2) Differentiate N's from percentages by extending percentages to one decimal place. 3) Not sure using decimals for N's is advisable (in the Comparison table). 4) Visually, it took me a while to understand what the greyed out text was conveying. I think an even more simpler way to convey this comparison is displaying the difference between the program and the college over all. This would provide a clear thumbs up or thumbs down. As it is now I have to go to the tables to see if this particular program is out/under performing the college overall. 5) Increase font size to promote readability. What we all need to get used to is conveying information in displays which are very easily interpretable. That is, the top piece of information should clearly stand out. In this case, is this particular program doing better or worse over time as compared to the college over all?
Pam posted on 9/16/2013 1:13 PM
This is a very comprehensive look at retention and graduation; however, it may almost be too much information. Maybe it works for your institution because it is a standard report that people are used to looking at. If I were to send something like this out to staff and faculty at my institution I'm not sure if they would completely understand it. I have had trouble with colleagues not being able to fully interpret visual displays of data with a lot of information in a small space.

If I were starting to distribute a report like this at my institution, I would also include an interpretation guide to answer most basic interpretation questions.

It may also be a good idea to include recruitment rates somewhere in the report. That would provide the full student cycle recruitment, retention, and graduation. Then this graphic could be used for full extent of program planning as it relates to moving students into and through a program successfully.

If the goal of this report is to display a lot of information, effectively, in a small space, then the goal was achieved; But the untrained eye may need a little help adjusting to the representation of this data and the many interpretations it allows.
Mark posted on 9/20/2013 11:29 AM
Has anyone suggested providing details regarding the programs that WCU Ret. students are moving into?