Survey coding turns open-ended responses into data you can count: each verbatim is read, assigned one or more codes from a codeframe, and tabulated like a closed question. This guide covers how to build a codeframe, how to code open-ended responses by hand and with AI, how to measure inter-coder agreement with Cohen’s kappa, how to handle nets, multi-coding, sarcasm and multiple languages, and how to run survey coding privately when the data sits under an NDA.
Open-ended questions capture what a closed list cannot anticipate: the reason behind a score, the brand a respondent names unprompted, the complaint nobody thought to include. Coding time has always been the price, and it keeps many questionnaires to one or two open-ends. Language models lower that price. In a 2024 paper in Research & Politics, Mellon and colleagues compared six language models on “most important issue” answers from the British Election Study Internet Panel, coded into 50 categories. The best model, Claude 1.3, reached 93.9% accuracy against 94.7% for a human coder, and the authors concluded that language models may allow wider use of open-ended questions.
Accuracy on a benchmark is a starting point. The process around the model, from the codeframe to the QA sample, decides whether coded data is fit to report, and that process is what the rest of this guide covers.
What Survey Coding Is
Five terms carry the whole process:
- Verbatim. One respondent’s answer to an open-ended question, exactly as typed or transcribed.
- Code. A numbered category, such as “price too high” or “friendly staff”, with a written definition.
- Codeframe. The full list of codes for a question, including nets and the residual codes for “other”, “don’t know” and “no answer”.
- Net. A group of related codes reported as one figure, such as a “Value” net covering price, discounts and value for money.
- Multi-coding. Assigning more than one code to a verbatim that covers more than one theme.
Coded results go back into the data file as variables. A multi-coded question is usually stored as a set of yes/no variables, one per code, so it tabulates alongside the rest of the survey.
How to Build a Codeframe
A clear codeframe makes every later step faster, whether people or a model do the coding.
Start from a random sample
Read a random sample of verbatims before writing any codes. Savanta’s guide to coding open-ended survey responses suggests reading at least 10% of them, selected at random so that early or late completes do not skew the frame. Note every distinct idea, then group the ideas into candidate codes.
Write a definition for every code
A two-word label leaves room for coders to read it differently. Give each code a one-sentence definition, two or three example verbatims that belong, and one example that looks similar but belongs elsewhere. Those definitions double as the instructions a language model follows, so time spent here pays off twice.
Plan nets, residual codes and code numbers
Decide the nets before coding starts, and include residual codes so every verbatim lands somewhere. On a tracker, keep code numbers stable from wave to wave and add new codes at the end of the list, so wave-on-wave tables line up without remapping.
Pilot and revise
Code a fresh batch of verbatims with the draft codeframe before the main run. Codes nobody uses are candidates for merging, and an “other” bucket that keeps growing usually means a code is missing.
How to Code Open-Ended Responses: Manual vs AI-Assisted Coding
Manual coding uses trained coders working through verbatims in coding software or a spreadsheet, with a coding lead answering queries and double-coding a sample to check consistency. It gives the team full control, and it suits small studies, new topics and questions where judgment carries most of the weight.
LLM-assisted coding keeps the same codeframe and the same QA, and changes who does the first pass. The model receives the question text, the codeframe with its definitions and examples, and the verbatims one at a time or in small batches. For each verbatim it returns code numbers, a confidence score and a short reason. A typical workflow has four steps:
- Code everything. The model codes every verbatim against the approved codeframe.
- Route by confidence. Low-confidence assignments and anything coded “other” go to a human coder.
- Sample the rest. Coders also review a random sample of high-confidence assignments, so QA covers the full range of the model’s output.
- Feed corrections back. Each correction updates a code definition or adds an example, and the model recodes the affected verbatims.
Two other approaches sit between fully manual and LLM-assisted coding. Rules-based auto-coding maps keywords and patterns to codes. A supervised classifier learns from verbatims coded in earlier waves and predicts codes for new ones. Our machine learning consulting team builds classifiers like this when a long-running tracker has the coded history to train on. The table compares all four approaches.
| Approach | How it works | Best fit | Where people stay involved |
|---|---|---|---|
| Manual coding | Trained coders read and code every verbatim | Small studies, new topics and judgment-heavy questions | Every coding decision |
| Rules-based auto-coding | Keyword and pattern rules map text to codes | “Other, please specify” lists with a fixed vocabulary, such as brand names | Writing the rules and coding what they miss |
| Supervised ML classifier | A model trained on coded past waves predicts codes for new verbatims | Long-running trackers with a stable codeframe and coded history | Labelling training data and coding new themes |
| LLM-assisted coding with human QA | A language model applies the codeframe definitions and returns codes, a confidence score and a reason | Most studies, including new codeframes and multi-language fieldwork | Approving the codeframe, reviewing low-confidence cases and a random sample |
Inter-Coder Agreement and Cohen’s Kappa
Inter-coder agreement tells you whether two coders, or a coder and a model, apply the codeframe the same way. Percent agreement is the simplest measure. McHugh’s 2012 review in Biochemia Medica notes that it does not account for coders agreeing by chance, so it can overestimate the true agreement. Cohen’s kappa corrects for chance agreement, which matters most when one code dominates a question.
The same review reports Cohen’s suggested scale for kappa: 0.41–0.60 moderate, 0.61–0.80 substantial and 0.81–1.00 almost perfect agreement. McHugh considers that scale too lenient for health studies and argues that any kappa below 0.60 indicates inadequate agreement. Survey teams can set their own bar for each question, with a stricter one for the codes that drive headline findings.
Three practices make intercoder reliability figures more useful:
- Calculate kappa per code on multi-coded questions. Treat each code as a yes/no decision, so one weak code cannot hide behind strong ones.
- Treat the model as a coder. Measure model-to-human agreement exactly as you measure human-to-human agreement, on the same sample of verbatims.
- Use Krippendorff’s alpha for more than two coders. It handles several coders and missing codes in one figure.
Nets, Multi-Coding and Hard Cases
Netting
Nets roll related codes into one reported figure. A respondent counts once in a net even when they mention two of its codes, so a net is usually smaller than the sum of its codes. Define nets in the codeframe and let the tabulation software calculate them from the coded variables, which keeps the counting rule identical across tables and waves.
Multi-coding
“Great coffee, but the queue took twenty minutes” carries two codes, one positive and one negative. Allow as many codes per verbatim as the content supports, and say so in the model’s instructions. A prompt that asks for “the best code” pushes a model toward a single answer.
Sarcasm and irony
“Oh great, another price rise” reads as positive word by word, which is how a keyword rule would code it. A language model reads the whole sentence, and it codes sarcasm more reliably when the codeframe includes a sarcastic example under the right code. Route verbatims the model marks as ironic to a human coder, since they often carry the strongest feeling in the study.
Multiple languages
Multi-country studies can code every language against one master codeframe. A multilingual model codes each verbatim in its original language, which keeps nuance that translation can lose, and the report can show the original text with an English gloss. Measure agreement separately for each language. An arXiv study of German open-ended responses by von der Heyde and colleagues found that results depended on the model, the prompting approach and the topic, and that a fine-tuned model performed best.
Short and empty answers
“Nothing”, “n/a”, “idk” and keyboard mashing need their own codes so they never land in a substantive category. Pre-coding these with simple rules before the model sees the file saves review time for the verbatims that need judgment.
Running Survey Coding Privately
Verbatims hold the most personal text in a survey. Respondents name employers, products, staff members and sometimes themselves, and the study as a whole usually sits under a client NDA or a data processing agreement. Running the coding model inside your own environment keeps that text where the contract expects it to be.
A private setup runs an open-weight model such as Llama, Mistral or Qwen on your own servers or in your cloud tenant. The codeframe, the verbatims, the coded data and every model call stay there, and the coded variables go out as labelled SPSS .sav or CSV files for your tabulation tools. Our private AI deployments follow this pattern, with access set per client or per study and a log you can share with a client on request. Model choice matters, so benchmark the candidate models on one of your own coded studies before choosing.
Starting With a Shadow-Mode Pilot
Pick a past study with a finished codeframe and human-coded data. Run the model on the raw verbatims, compare its codes with the delivered codes, and calculate kappa for each question and each code. Review the disagreements with the original coders. Some will be model errors, and some will be inconsistencies in the human coding that the comparison brings to light. Set the agreement bar for each question, then run the next live wave with the model on first pass and your coders on QA.
Coding open-ends for clients under NDA? NeuralChainAI builds survey data analysis with private AI, with LLM-assisted coding to your codeframe, human QA with agreement reporting, and exports for your tabulation tools, all inside your own environment. Bring a past study and we’ll scope a shadow-mode pilot.
