How to analyse open answers, with or without AI

Open answers hold what a number can't say: the why. They are also the part that often goes unread, because it takes a lot of time to read hundreds of them and make something sensible of it. This method comes from the literature on thematic analysis and from how evaluation officers do it in practice.

The eight steps

1. Clean and anonymise. Remove names, student numbers and other traceable details from the text before you share or analyse it. Think of indirect details too, like a teacher's name or a rare course.
2. Read a sample. First read a random part of the answers, without sorting. That gives you a feel for the tone and what is going on.
3. Make a first list of topics. For each answer, note in a few words what it is about. An answer can have several topics.
4. Group into themes. Merge topics that mean the same. Give each theme a name that says what is going on, like "feedback comes too late" instead of "feedback".
5. Count. How many people mention each theme? Counting isn't everything, but it shows where most words are. Careful: a theme with many mentions isn't necessarily the most important.
6. Pick quotes. Put one or two verbatim quotes with each theme that capture it well. Quotes make a theme recognisable and show it isn't made up.
7. Check. Take a sample from the raw answers and see whether they fit the theme where you put them. Also look for answers that fit nowhere: those are often the most interesting.
8. Write the "now what". A theme isn't finished until you've added what it could mean and what you'll do with it, or why not.

Where AI helps, and where to check it

AI is good at steps 3 and 4: recognising and grouping topics, in several languages too. That saves a lot of reading. But there are a few pitfalls.

• Summarising is not counting. An AI that writes "students find the feedback too late" doesn't say how many students think so. Always show the count and the source.
• Check the quotes. A quote must appear verbatim in the data. Ask for it and verify it.
• Watch what falls away. A summary smooths what is rough in the raw data. Read the outliers yourself too.
• Know where your data goes. A public AI tool is not the same as processing in the EU with a data processing agreement. See the privacy checklist.

When it's better by hand

With fifteen answers you don't need a tool: read them all and write down the themes. A systematic approach pays off from a few dozen answers, and certainly across several courses or rounds at once.

A better question is the best analysis

Much of the effort in analysis comes from shallow answers like "good" or "fine". A follow-up in the moment ("what makes it good for you?") yields answers you can work with. That is the idea behind a conversation instead of a form.

Febaso does steps 3 to 6 for you: it reads every answer, groups them into themes with counts and verbatim quotes, and shows with each theme which answers sit behind it, so you can do step 7 quickly.