UT System and Census Collaborate for Better Data on Student Outcomes
It is no longer business as usual for institutional researchers when creating metrics that describe student access, progress, and success. Policymakers, media, and the public are asking complex questions pertaining to student outcomes. There is also a strong consumer advocacy movement asking, “Is higher education really worth the cost?” and demanding transparency and data on the value of a degree and the return on investment (ROI).
In response, some postsecondary institutions are linking higher education data with state unemployment insurance (UI) wage records to demonstrate how higher education institutions are preparing students for a successful career. Accurate, timely data on student outcomes and post-graduate earnings are a critical piece of any state effort to close equity gaps in college access and success, boost attainment statewide, and strategically align education and workforce goals. However, without federal student-level data, states and other key stakeholders do not have all the information they need to inform state and institutional efforts and best serve students.
Within most states, students' workforce outcomes are limited to students who remain in the state after graduation. In the IHEP report A Roadmap to Better Data: Developing a Census Bureau Partnership to Measure National Postsecondary Earnings Outcomes, Stephanie Bond Huie and I describe a first-of-its-kind collaborative between the University of Texas (UT) System and the U.S. Census Bureau to provide national earnings data for students after graduation, including for those students who leave the state. The partnership provides accurate, timely, and secure data on student earnings outcomes by institution, degree level, and field of study within the UT System. With information to assess the success of all graduates, the UT System is better equipped to create evidence-based policies and practices and promote the success of all students.
This article outlines the history of UT System/Census collaboration, the ways the UT System is using the data for transparency and improvement, and advice for individuals in the AIR community interested in embarking on a project of this nature.
Collaboration
Using Texas Workforce Commission (TWC) UI wage records, we developed a free, online tool called seekUT. The tool provides students and families information on 1st-, 5th-, and 10th-year earnings after graduation in the context of average loan debt, by degree level and major. For certain degree types (bachelor and professional) we obtained high match rates (70-84%) when linking with TWC data. However, match rates fell dramatically when focusing on master’s and Ph.D. degree (50% and 33%, respectively). We were not satisfied with that result because we wanted to provide all types of students earnings data for their programs. This was one of the main reasons why the UT System decided to pursue a collaboration with the U.S. Census Bureau’s Longitudinal Employer-Household Dynamics (LEHD) program. The LEHD data maintains national UI wage records that accounts for over 96% of employment in the U.S. At the onset, we felt it was important to capture as many students as we could, since not everyone who earns a degree at a Texas institution stays in-state.
One of the main reasons we were able to successfully build a partnership with the Census is that we had a proof of concept for how the data would be used. We met with the Census representatives and demonstrated the seekUT tool, and they appreciated how we visualized the data to tell a comprehensive story. Also, as they had limited access to higher education data, they saw a way to use our visualization model to improve and enhance their Census products. This partnership became a win-win for both parties. We established a 10-year agreement with them based on their recommendation, and we spent several months educating them on higher education data while they shared UI wage data in aggregate form. Also, they helped us as IR professionals to better define earnings outcomes for our students.
Using the Data
Through our partnership with Census, we learned some valuable lessons that are useful to other IR professionals. For example, at the UT System we had always taken a conservative approach to calculating earnings data by using only graduates with wage records for all four quarters within a calendar year. However, an analysis by the Census showed there were few differences in earnings when using three quarters versus four quarters. Since using three quarters of wage records significantly increases the sample size, we are now using use the three-quarter model to measure annual earnings. The Census analysis also showed little difference in the annual earnings for May graduates compared to December graduates, allowing for more records to be added to the model without compromising the results.
It is important for us to make the earnings information understandable and actionable for students, such as providing information in a way that high school counselors can talk with prospective students about how the selection of a specific institution or academic program will impact the typical amount of student loan debt as compared to expected earnings after graduation. We have learned that student loan debt information is more helpful to students when the total debt is broken down into monthly payments over a 10 year period and compared to expected monthly earnings. Students about to graduate are also beginning to use this data during job offer negotiations.
We know that expected earnings for graduates can vary based on the location of employment. For example, there are significant differences in the cost of living between El Paso and Austin or Silicon Valley and Dallas. Although we try to educate students about the differences in cost of living, the current earnings data are limited. We are looking forward to 2020, when Census will add W2 information so that we can provide more targeted earnings data for different regions of the country.
We’re always on the lookout for additional data points to fill in the information gaps for students and help answer their questions. One big example is internship data. The number one question we get from students is “How does my internship impact my earnings outcome?” While we don’t have the answer to that yet, we are working toward better data in this area.
Advice for Others
The roadmap in the IHEP report provides state and other data leaders with guidance at every critical step in creating a partnership with Census, including building buy-in and approval from leadership, developing necessary legal agreements, and ensuring the quality and security of data. In addition to these concrete steps, I believe you must build strong relationships with career services and student groups within your institution or system to understand and capture their needs within the data systems and tools being developed. Once you have the data the work is not over, you must figure out how to disseminate it to your primary audiences. You have to bring it to life! Having a vision for where you want to go will greatly contribute to the success of your project.
Overall, I believe our secret to success is that we built a level of respect and an open line of communication between the two agencies – the UT Systems and the Census. If you don’t have respect and effective communication, it is difficult to work through the inevitable issues or setbacks that you might encounter along the way.
From a data privacy standpoint, we haven’t had one complaint from a parent, student, dean, president, or politician since we began the work in 2013. Instead, stakeholders appear to want more information, not less. This positive relationship and sense of trust from stakeholders is likely due in part to the care and attention from the UT System and the Census to ensure the proper safeguards are in place. For example, Census uses differential privacy methodology so we will never be able to identify an individual student within our dataset. The privacy protocols do create challenges. One choice was to match UT System student data with Census earnings data at the four-digit CIP* level for baccalaureate and professional degrees and at the two-digit CIP level for master’s and doctoral degrees, as we know there are fewer students with master’s and doctoral degrees. With very few students, the earnings amounts can be unstable and may create a disclosure risk. We also had to make sure we received information back from the Census, because anything with small cell sizes are suppressed and would be returned to us with an N/A. That is why we were strategic with the choice to use the four-level versus the two-level CIP earnings calculations.
As a member of the AIR Board of Directors, I am reminded that every IR office is at a different data maturity level based in part on how much data they have, how long they’ve been analyzing it, and the resources they have available. It is likely that many small offices do not have the resources to participate in this type of project as one institution. The Census, however, is beginning to work with more states to build partnerships so that higher education institutions within the state can have access to state level data for aggregate reporting that may help to alleviate the pressure many of these small offices are facing. For institutions, systems, or states with the resources, support, and infrastructure to consider a partnership with Census, I recommend contacting the U.S. Census Bureau, contact: CES.PSEO.Feedback@census.gov.
The goal of the IHEP report is to help institutions pursue strategic data partnerships within their states and elevate the utility of student outcomes data nationwide. Ultimately, the UT System-Census partnership shows that it is possible to protect student data and also provide timely information to help students and their families make better, more informed decisions.