When a Survey Can Ask Why: AI Follow-Ups in Student Affairs Assessment

Anyone who has coded open-ended survey data knows the frustration of a response that signals something important but does not provide enough detail to inform action. A student might share, “The sessions weren’t helpful” or “I didn’t feel like the program was for me,” but a traditional survey cannot ask what made them feel that way or what would have improved the experience. Historically, gaining that context has required a focus group or interview, valuable approaches that demand additional time from students and staff.

At Carnegie Mellon University’s Office of Institutional Effectiveness and Planning, we have been piloting BlockSurvey, a platform with built-in generative AI capabilities, as a promising alternative. What started as a way to strengthen open-ended feedback has grown into a tool for gathering richer insights and reaching students who might not otherwise participate in our research.

AI Follow-Up in Action

BlockSurvey’s AI Follow-Up Questions feature generates a targeted, conversational prompt in real time based on a respondent’s open-ended answer. The prompts are grounded in Open questions, Affirmations, Reflections, Summaries (OARS), a framework borrowed from motivational interviewing, and survey designers can set one to five follow-ups. Additionally, the system stops when a respondent signals that they are finished, helping the interaction remain conversational rather than intrusive.

If a student responding to an orientation survey writes, “Some of the sessions weren't useful,” the AI might ask, “What about those sessions felt less useful to you?” So instead of ending with a vague statement of dissatisfaction, stakeholders benefit from additional context that can inform real programmatic decisions.

We are also applying this approach to an assessment of neurodivergent student experiences. Students can choose between an in-person focus group or a BlockSurvey session combining voice-to-text responses with AI follow-up questions. This offers another way to participate for students who may be uncomfortable in a group setting while allowing us to gather rich qualitative data without scheduling additional focus groups.

Ultimately, an AI-assisted survey tool brings some of the depth of an interview into a survey, at a scale difficult to achieve through human follow-up alone. It offers a much richer context to responses from standard survey questions.

All of that said, these benefits do not eliminate the need for rigorous research methods. We are intentional about which questions warrant follow-up and whether probing will produce meaningful and actionable insights without contributing unnecessarily to survey fatigue. For example, I often trigger follow-ups only for negative sentiment or constructive feedback. A response such as “Nothing” or “I had a great time” may not warrant additional probing, particularly when another question already asks about program strengths. Before launch, we test potential responses to ensure the prompts are appropriate. Good output still depends on sound research design and human judgment.

Analyzing Adaptive Responses

Because respondents receive different follow-up questions based on their initial answers, analyzing the data requires a unique approach. BlockSurvey’s AI-assisted analysis supports this process by allowing stakeholders to track aggregate themes and insights in near real time. I am transparent that these emerging insights should be viewed as preliminary rather than final findings, but even so, they provide a valuable view of patterns as responses are submitted.

After data collection, I review the AI-generated themes against the original responses to ensure they preserve context and accurately reflect what students shared. Stakeholders then receive a final analysis of the open-ended data that I have personally reviewed and validated.

Conclusion

For IR and student affairs assessment offices already stretched thin, these GenAI-assisted follow-up questions offer a way to deepen qualitative insight without adding staff workload or survey length. My advice: start small. Pilot an AI follow-up feature on a simple pulse survey, and see what your students tell you when someone—even an AI—simply asks them to say more.