Designing Your AI Co-Analyst: Principles for a Trusted Qualitative Analysis Workflow

Qualitative analysis remains one of the most time-consuming tasks in institutional research. Generative AI tools can quickly synthesize large volumes of open-ended responses, but many IR professionals are understandably cautious. If we are unable to explain how an insight was generated, verify where it came from, or protect the underlying data, we risk undermining the trust that makes our work valuable.

Before using AI for qualitative analysis, it is important to establish a secure, human-centered process. That means using institutionally approved AI tools, removing unnecessary personally identifiable information (PII), and treating AI as an analytical assistant rather than a decision-maker. The goal is not to replace researcher judgment. The goal is to spend less time processing data and more time interpreting it.

The following four principles have helped us build a more reliable approach to AI-assisted qualitative analysis.

1. Start with a Prompt Architect

One lesson I learned quickly is that AI can be surprisingly helpful in building prompts for other AI tools.

Rather than creating every instruction from scratch, I use a separate AI agent called a "Prompt Architect." In practice, this is simply an AI conversation dedicated to designing and refining prompts before those prompts are used for analysis. Its job is to ask clarifying questions, identify ambiguities, and help structure prompts before they are used with actual qualitative data. I then use the Prompt Architect to build an “Analyst” agent. And yes, I use an LLM to help create the Prompt Architect.

This approach improves consistency and reduces trial and error when developing prompts and building agents.

2. Write a Core Truth Directive

The most important part of our analyst agent is what I call a "Truth Directive."

We instruct the agent to base its findings only on the data provided. Whenever appropriate, every claim should be linked to a respondent identifier or data label. Any inference should be clearly identified. Most importantly, the model must be directed to admit when it does not know something.

If the data does not support a conclusion, the AI should respond with something like, "I cannot verify this from the information provided."

This encourages transparency and makes findings easier to review before sharing them with stakeholders.

3. Test, Fail, Fix, Repeat

Prompt development is an iterative process. The first version rarely works as intended.

When something goes wrong, identify the specific point where the AI became confused. Then add instructions that address that issue directly.

For example, we found that AI struggled to interpret PDF visualizations. Switching to structured tables improved accuracy significantly.

Treat prompt development like any other analytical process. Test it, review the results, and refine it over time.

4. Separate Extraction from Assembly

One of the most effective ways to improve reliability is to separate data extraction from report writing.

In our workflow, the first prompt extracts and categorizes comments while preserving respondent identifiers. A researcher then reviews those results before moving to the next step. Only after that review is complete do we use a separate prompt to draft summaries or findings.

This review point is critical. It helps prevent unsupported conclusions, reduces the risk of losing important context, and keeps researchers actively engaged in the analysis.

Conclusion

Generative AI can help institutional researchers process qualitative data more efficiently, but efficiency should never come at the expense of rigor. Using approved tools, building verification into prompts, and saving successful prompts as templates can help create a more trustworthy workflow. AI can assist with organization, categorization, and synthesis, but researchers remain responsible for interpretation, judgment, and the final conclusions.

Kirby Livingston, Ph.D. (kirbyl@andrew.cmu.edu) is the Data Strategy & Institutional Research Manager at Carnegie Mellon University’s Tepper School of Business, where he focuses on leveraging text analytics to enhance the student experience, managing rankings, and designing secure AI integration workflows. A graduate of the University of Wisconsin-Madison, he has recently presented his practical frameworks for augmented intelligence in IR/IE at national and regional conferences for AIR, NEAIR, and NASPA.