IR and Inconvenient Public Data

Ask eAIR invites questions from AIR members about the work of institutional research, careers in the field, and other broad topics that resonate with a large cross-section of readers. If you are interested in writing an eAIR article, or have an interesting topic, please contact eAIR@airweb.org.  

This month’s question is answered by Ishuan Li, Associate Professor of Economics, Minnesota State University-Mankato

The ideas, opinions, and perspectives expressed are those of the author, and not necessarily AIR. Subscribers are invited to join the discussion by commenting at the end of the article.

ishuan-li.jpgDear Ishuan: I recently came across a particular website that generates employment measures and portrays graduates from my institution unfavorably compared to other institutions. What is your perspective on IR use of/approach to public data on graduates’ employment outcomes?

About four years ago I came across a website which reported longitudinal labor market data from the Minnesota Department of Employment and Economic Development.[i] As a faculty member with research interests in the economics of education, I was extremely excited about its potential use in assessing effectiveness of programs and institutions in preparing students for the labor market. On this website, readers can view aggregated reports on employment measures (unemployment status, earnings) of graduates from all institutions of higher education in the state of Minnesota by degree levels and majors.[ii] Two other similar public access data tools are provided by Virginia Longitudinal Data System[iii] and Florida College System Report[iv].

My perspective as a college faculty member and applied economist in a state institution might differ from institution administrators, full time staff, and IR personnel. Obviously, administrators face external stakeholders who determine public funding. IR personnel, on the other hand, stand between administrators and faculty. As a faculty member, I teach primarily undergraduate courses for the degree programs in my department (within a large and public university in the Midwest). I have assessed program outcomes of our graduates. I value IR as a source for clean or “reliable” statistics used for program assessments. I have utilized IR reports, such as student “success/failure” rates by race (and underrepresented population) categories across College courses. While assessing my program’s degree outcomes, I struggled to find objective measures linking socioeconomic variables to the academic performance of students and outcomes in the labor market. My experience in the latter made me realize the difficult but critical role for IR offices, sandwiched between administrators to whom it generates analysis and reports, and the demands from faculty (departments or programs) who are not directly accountable to external stakeholders. It is especially difficult for IR offices at public institutions given the divergent interests of different stakeholders with varying urgency of priorities.

Despite their potential usefulness, public data, particularly those generating negative statistics of an institution’s graduates in the labor market, create several issues for IR offices. I argue that IR offices have a critical role and ethical obligation to confront those issues directly. Except for resource constraints, objections to confronting the data generally fall in one of the following groups: (1) unfamiliarity with proper analytical methodologies; (2) imperfect data tools; (3) imperfect measures; or (4) expediency. The first two objections can sometimes be justified by limited institutional resources (better endowed institutions are able to hire staff with the quantitative research skills). I am sympathetic to the objection regarding data and methodology imperfections. The most objectionable approach is the latter, expediently ignoring the data and hoping that nobody notices. The “ugly” data, which reflect unfavorably upon an institution is ignored and consequently unavailable for decision making.

Honest assessment of a university’s contribution to their graduates’ labor market outcomes is critical for efficient public funding of higher education. IR offices face the critical responsibility of steering institutions towards a proactive, careful analysis, interpretation, and presentation of data. When a public website negatively portrays an institution’s graduates' labor market outcomes, the best course of action is to present the institution’s own analysis of the data within the context of the institution’s mission, constraints, values, and strategies. What is the alternative? In the absence of the institution’s own narrative, the public may incorrectly interpret seemingly harmless statistics without proper context and arrive at a less than favorable conclusion.

[i] The Graduates Employment Outcomes in Minnesota is an online data tool that provides aggregated reports of labor market outcomes for graduates from institutions of higher education in the State of Minnesota. The reports show median earnings (hourly and annual) and employment status of students with valid social security number and are employed within the state of Minnesota. The reported data are aggregated at the program (four- and two-digit CIP levels), by award types (AA, bachelor, certificates, graduate), of public, private, and for profit institutions, by year of graduation (2006-2009, and subsequent cohorts, the most recent is July 2013-June 2014.)

ii] Graduate Employment Outcomes in Minnesota - Results

[iii] Virginia.gov Wages of Graduates, Degree Levels by Institution

[iv] FL DOE SAS Full-Time Employment Earnings Gain Report (Log in required)

 

 Comments

 
To add a comment, Sign In
Total Comments: 2
 
Amy posted on 2/17/2018 11:47 AM
I would like to see the Florida Report but I cannot login to SAS. Who has access to SAS to view the report? I thought AIR members did, otherwise this information you are providing cannot be used.

Amy
Ishuan posted on 2/21/2018 5:03 PM
Hi Amy: the report is public, no log-in is required. The link provided is for the specific report. To get to this report, go to https://edstats.fldoe.org/SASPortal/main.do, on that page click on FETPIP tab, then click on FETPIP portal link. Scroll all the way down and click on FETPIP ADDITIONAL FETPIP REPORTS. Once on the the FETPIP Reports page, click on the FLORIDA COLLEGE SYSTEM REPORTS. You have a choice of reports to view, including employment earnings by majors, for instance, for all students earning a bachelor degree, over the reported years. The EDSTATS.FLDOE.ORG site allows the public to view various reports on k-12 and higher ed institutions.