Secret Shoppers: ​Secrets Revealed

Got a secret?  

Twenty-eight AIR members had secrets at the 2013 Forum in Long Beach when they served as AIR Secret Shoppers.  

leadimage.JPGHave you ever been frustrated while waiting in a long line or while navigating a circuitous telephone menu system? Have you questioned whether management cares about the quality of service delivered to you? Most of us have had these kinds of experiences, but have you ever wondered how they might also inform your work in institutional research?  

IR and assessment professionals find themselves involved in all kinds of evaluation projects. A request to explore users’ satisfaction with services could certainly land in your inbox as the next component of “other duties as assigned” that require data and evaluation skills. Use of secret shoppers to evaluate college and university services should be in your assessment tool kit and at the ready. 

The secret shopper concept applies to any transaction between your institution and an individual. You might be asked to explore how faculty members experience human resource services; how students experience the library reference desk, business office, registration or orientation processes; or how recent graduates experience alumni services. What makes this secret shopper concept special is that the staff members involved in the delivery of services set the standards for evaluation. The power is in having providers define acceptable effectiveness and satisfaction levels of the services they deliver.  

The key steps: 

First, staff members in a service unit identify the “high value” transactions that shape the consumer experience. The first criterion for selection must be the importance of the transaction in bringing about the desired outcome. A secondary criterion is to select items that the average consumer can evaluate or measure without any special tools or knowledge. Good examples are length of time in line, whether the service provider addressed the consumer by name, or if a specific “deliverable” was actually delivered.  

Next, the service providers determine the level of service they consider acceptable. For example, rather than ask secret shoppers if the line for service was “too long,” which means that every shopper establishes what “too long” means to him or her, the service providers present a defined set of answers for the secret shopper to use. In the example of time spent waiting in line, actual time is the desired measure rather than the shopper’s subjective opinion of what is too long or is just right. (Consumer satisfaction and opinions matter, but that is a different process!)  

The most important work is complete at this stage because the delivery model has been formalized and the service providers have clear goals. The setting of team performance goals is a powerful action in and of itself, but there is still value when performance is checked with independent measures to confirm that what was hoped for was actually achieved. 

The data collection phase requires reaching out to service users to ask a sample of them to test the service with use of a set of questions and answers on a secret shopper survey. Secret shoppers are directed to focus their attention on the items selected rather than broadly evaluate whatever strikes their fancy. This is not an overall evaluation process; rather, its target is to improve a few key performance measures.  

Some secret shopper models assign evaluators to specific time bands or conditions, but many allow consumers to go about their business as usual and leave the collection time to chance. Obviously, the more important the activity being evaluated, the more important it is to control the evaluation model. This process works best when it is for general service improvement and does not carry high stakes, such as pay adjustments or disciplinary actions. 

Finally, a plan for review and analysis of secret shopper data should be in place before the project commences. Whether secret shopping is used as an occasional review or is incorporated as an ongoing component of services provided, data gathered have limited value if they are not used to inform review of services. 

So what about those sly 28 AIR members with a secret? The AIR staff members who managed the 2013 Forum registration desk and the breakfast service prior to the Monday keynote agreed to be part of a secret shopper evaluation. We also secret shopped concurrent sessions 1. We asked 43 AIR members who attended the Forum to be secret shoppers, and 65% completed their assignments. Check out the results

eAIR invites your comments. Have you used this model before? Do you have ideas for its use in the future? Enter your comments about past or proposed use, or any variations in the model you wish to share in the comments box below.  

1 We confess that the concurrent session evaluation fails the true test of secret shopping because presenters did not set the evaluation components. Most assessments have to compromise a bit to be “doable.” 

P.S. No AIR staff member or secret shopper was endangered or harmed during participation in this activity! Don’t try this at home without IRB oversight. 



To add a comment, Sign In
Total Comments: 5
Heather posted on 8/15/2013 9:03 AM
This article is a great example of how diverse the IR professional's opportunities are (besides being full of unlimited fun). I would like to see this go a step further and have a series on IR occupations that are outside the norm. Excellent Job.
Tony posted on 8/15/2013 3:05 PM
I was one of the secret shoppers. This is a great idea, and was really fun. The results are interesting and well presented.
Mark posted on 8/16/2013 11:39 AM
It was fun to be one of the secret shoppers, a good idea and worth continuing.
Eric posted on 8/16/2013 11:51 AM
What a great way to learn more about effectiveness! I have never used this approach, but I can definitely see the value of the secret shopper technique especially to evaluate the quality of service for student support areas.
Jamie posted on 8/16/2013 4:04 PM
Assessment is more rewarding when you see that the results are being used - or at least looked at. I think it was wise to have secret shoppers focus on something pretty specific like a single event or one function over a few days. Good to see AIR trying something a little different.