Organizing for Evidence-Based Decision Making and Improvement

This Resource Review was peer-reviewed by members of the AIR publications volunteer panel, and the ideas, opinions, and perspectives expressed are those of the author, and not necessarily AIR.  Members are invited to join the discussion by commenting below.

Christina Leimer (Change, July/August 2012, pp. 45-51) 

Reviewed by Jonathan D. Fife 

Since the early 1990s, there has been an increasing recognition that higher education intuitions were significantly behind other major organizations in creating a decision making culture that values the use of qualitative and quantitative data. Increasingly, external organizations, such as accreditation and state agencies, that control the funding of—and even the existence of—institutions have pressured institutions to utilize data to validate their decisions and as evidence of the achievement of meeting their stated institutional values, goals, and mission. The reputations of institutions as reported by the ranking publications, such as US News and World Report Best College Rankings and the 2013-2014 Baldrige Education Criteria for Performance Excellence (U.S. Department of Commerce, National Institute of Standards and Technology), base their evaluations on data supplied by institutions.  

These external pressures have created a need for higher education institutions to develop a “culture of quality” and a “data driven decision making process.” Institutions’ responses to these pressures have been slow and inconsistent. The reasons for this are simple to identify, but enormously difficult to correct. The first reason is that faculty members are very skeptical, even hostile, to the idea of basing academic decisions on data produced by non-academic administrators, i.e., institutional researchers or external evaluations. The second is that, on the whole, higher education administrative structures lack the skills and organization needed to create a data-driven decision making cultures that meet the needs of both external and internal stakeholders. 

In Organizing for Evidenced-Based Decision Making and Improvement, Christina Leimer provides an analysis of the changing role that institutional research offices are beginning to play in higher education institutions’ efforts to create environments that value data-driven and evidence-based decision making. Two factors are identified that make new administrative approaches desirable: (1) the need for presentation of data, findings, and application that motivates people to want to act on this information; and (2) an administrative structure that is embraced for its ability to be helpful to those who are affected by how information is used.  

Leimer details the dilemma concerning the role of IR offices. Is the purpose of this administrative office only to assemble data to which others react (a decision making passive role)? Or is the role of IR to identify, collect, and present data that facilitates action (an active role)? Increasingly, IR offices see themselves as joint participants in the collection and analysis of data with the intent of informing institutional decision making related to linking institutional short-term outcomes with long-term purpose and goals.  Specifically, IR has two major roles: (1) to create data gathering processes that link institutional activities with current results, and (2) to present findings so they stimulate ground-level enthusiasm and personal needs to act on these data. However, for IR to be effective in these roles, IR’s position in the organization must evolve in a way that it is considered a legitimate part of both academic and administrative decision making processes. IR must be viewed as a partner in the achievement of institutional values and goals, and not as an external assessor.  

Leimer presents a study of how some institutions have developed different organizational models to foster evidence-based decision making. She found that from 1995 to 2010, the number of IR offices charged with some aspects of institutional effectiveness (IE) increased from 43 to 375. While modest when compared to the growth in the number of traditional IR offices (672 to 1,499), it shows rising awareness of the need to change the focus of IR activity. Leimer used institutions’ websites to analyze the responsibilities of 30 IE and 30 IR offices. She found that some offices only experienced name changes whereas others were primarily responsible for assessing learning outcomes. Neither change involved an IE role for IR offices. Significantly, it was evident that many offices had been assigned multiple responsibilities related to assisting other units with evidence-based decision making. These areas included planning, assessment, academic and administrative program review, and accreditation.  

Nineteen of these potential IE offices were interviewed to determine their offices’ origins, organization types, and responsibilities. She used her analyses to refine her model of an IR office as evolving from a traditional role to the integrated model (IM), which involves myriad responsibilities, such as:

  • Creating an institution-wide understanding of the use of evidence in decision making.
  • Identifying external trends and issues relevant to the institution’s mission and helping appropriate units understand their relevance to activities.
  • Working with units throughout the institution to “help make data use part of the culture.”
  • Continuously measuring activities and outcomes deemed key in the achievement of the institution’s goals, and participation in the review of these measurements with those responsible for the successful execution of the activities.

What is involved in changing the data gathering role of IR offices to a role in which IR professionals are facilitators who help create and support a culture of evidenced-based decision making is very complicated. These complexities are identified in the remaining portion of the article. Unfortunately the space available is insufficient for an in-depth analysis of these issues. 

What is missing in this article is recognition of the importance that organizational cultural values play in accepting IR offices and their staff as valued partners in the evidence-based decision making process. There is a great deal of literature related to the impact of organizational culture and how it can be changed. 

As Leimer noted in the beginning of her article, a 2011 Inside Higher Ed survey reported that two-thirds of responding college presidents felt that “their institutions are not particularly strong in using data for making decisions.”  This observation suggests the existence of a culture that does not support evidence-based decision making. To change this type of culture to one that values evidence-based decision making requires more than changes in administrative structure. What is needed is for the involvement of all the institutional units to: 

  • develop an understanding of how the basic goals and mission of the institution, as defined by internal and external stakeholders, relate to their particular activities;
  • embrace change as a positive and continuous value;
  • be a part of the development and use of measurement tools that link process with the fulfillment of organizational and personal goals; and
  • understand that the data gathering and analysis skills of institutional researchers and the IR office can be a positive force if they are to be more effective in basing decisions on fact and not past practices. 

The Baldrige Performance Excelling Program 2013-2014 Education Criteria for Performance Excellence (U.S. Department of Commerce, National Institute of Standards and Technology) criteria address many of the areas addressed in Leimer’s article. An analytic framework that has been 20 years in the making, the Baldrige criteria consists of a set of questions about critical aspects of an institutions’ management and performance that can greatly inform the role of IR—specifically the part IR offices play in how institutions organize themselves to create a culture for evidence-based decision making and improvement.  

In addition to the Baldrige criteria, two publications that may be useful in the development and implementation of the integrated model that Leimer describes are A Culture for Academic Excellence: Implementing the Quality and Principles in Higher Education (Freed, Klugman, & Fife, 1997) and Tools for Improving Institutional Effectiveness (Fife & Spangehl, 2012). 

Jonathan D. Fife is Visiting Professor, Higher Education and Student Affairs, Virginia Tech.



To add a comment, Sign In
Total Comments: 1
Bill posted on 4/11/2013 5:07 PM
Tina's article is a pivotal resource for our profession and for higher education generally. Every institution in the country needs to be discussing these issues.