Automating Surveys with Python, Qualtrics API, & Windows Task Scheduler
Last fall, I had the opportunity to present my survey automation project using Python and Qualtrics at the Pacific Northwest Association for Institutional Research and Planning (PNAIRP). With my friend, we have since turned my original code into an open source Python programming package along with a step-by-step tutorial. However, for this Tech Tip, I will just provide a general overview of the process, what is needed, and the long-term benefits for automating your institutional surveys. This tip is intended to be a companion guide. For the full step-by-step tutorial, please visit my Medium blog.
Why Automate?
One of the most labor-intensive tasks I oversaw was disseminating surveys. How many of us have quarterly or annual surveys that go out to the same sample of students, staff, faculty, and/or alumni each quarter or year? Not only did I have those responsibilities, I initiated a daily satisfaction survey to first time visiting patients at the campus clinic. I knew right away that I was not going to manually run a list of emails; upload that list to Qualtrics; and then schedule surveys to go out each day… day after day. I didn’t want to be the ‘click!’ ‘click!’ person, no one should really want to be. That’s the beauty of automating things in Python. In fact, there’s actually a free e-book called, “Automate the Boring Stuff with Python.” Don’t be the ‘Click!’ ‘Click!’ Person - Start Automating!
What is Python?
Python is a general purpose open-source programming language whose popularity is considered one of the fastest-growing in recent years. You can see in the following diagram that as of September 2017, Python is the programming language which the most questions posted on Stack Overflow.
What You Need
First, verify you have access to a Qualtrics account with API add-on, which is not including in as part of Qualtrics’ Research Suite. If you do not have the API add-on, please reach out to your Qualtrics Product manager to purchase it.
Second, install Python 3.6 or newer, which can be downloaded.
Third (optional, but highly encouraged), install the PyCharm Professional Edition IDE, which is available for free for ‘.edu’ email addresses, from JetBrains, per the instructions provided.
The Real Magic is in the API
APIs or Application Program Interface are one of the things that makes modern day web-based software solutions so popular. Modern software solutions with a well-documented API essentially give permissioned users full “behind-the-scenes” access to the application. Let’s look at the case with the Google Maps API. Normally, if you want to see how far a destination is, you look up an address by going to maps.google.com, enter the address, and then click search. However, what if you have thousands of addresses to calculate? For example, for one project, I had to calculate the average distance students lived away from school. Instead of searching it one by one, you can write a script to make an API call or ‘request’ to Google Maps to ‘get’ the distance for each address.
The way I understand it, a web API is just a complex URL that contains additional information, actions, and/or instructions to query information from a website or web application. This is an example of what an API call to Qualtrics might look like:
In the above code, what your doing is “getting” (requesting) from Qualtrics your mailing list (mailinglists/ML_3w9zbKyxLflYNDc) with your super-secret access token (x-api-token) as your credential. Yup so that 8 lines of code will allow you to fetch any of your mailing lists from Qualtrics.
The Survey Automation Process
The actual process to automate survey distributions follows the diagram below.
We first get all the essential setups in Qualtrics, like generating your API login token, building your survey, creating your email message template, etc. We then run the Python Qualtrics-Mailer package and enter parameters unique to you like your survey name, email subject line, date and time you want to send out the survey, etc. And then finally, we setup the Python script to run by itself using (for Window’s user) the Task Scheduler tool.
The Window’s Task Scheduler is something that works behind the scene on your computer. It is used to create and manage common tasks that your computer will carry out automatically at the times you specify. Even though you will typically not see it, it is running behind the scenes. For example, if you have something like Dropbox or Google Drive installed on your Windows PC, then your Windows Task Scheduler will probably show some sort of ‘check for updates’ task for those already scheduled. The best way to bring it up is to search for it in the Start Menu or in the Cortana button (for Windows 10 PCs).
Long Term Implications
For my project, I automated a daily survey to new patients. But I think there’s a huge opportunity for something bigger and more relevant to the most of you – automating course evaluations at the end of each term. With the Qualtrics API, or really any other modern web-based survey tools with an API (e.g. Survey Monkey, Typeform, SurveyGizmo), you can essentially do it yourself with in Python. Basically, you’d just query your student database at the end of each term and send your course evaluation survey to all registered student.
The Qualtrics API is also well documented and robust. So, it’s not only possible to distribute surveys, but also to import survey results on demand. This allowed our office to quickly update our survey dashboards by routinely exporting results into our Microsoft Power BI dashboard tool (will work in Tableau as well).
Some Takeaways
Thank you for reading my Tech Tip. I want to leave you with a few tips and comments.
- If you’re looking to purchase any web-based solution for your IR office, do look for ones with well documented APIs (if you’re not sure, ask your IT department to review it). I think automation and integration between mixed systems will become a necessity soon, so purchasing solutions with APIs and “web-hooks” will really help in the long run.
- For more information about using Python, I recommend the official Python Software Foundation’s tutorial. Another excellent resource is The Hitchhiker’s Guide to Python!
A big thank you to my Python programming colleague and fellow IR professional, Isaiah Lankham for collaborating on the ‘qualtrics-mailer’ Python package. For those interested in Python, we co-host the Python Out Loud podcast about the Python programming language and the process of learning software development. You can also follow us on Twitter @PythonOutLoud.
Here are some other resources for working with Excel & Python.
