One standout application of LLMs is their ability to automate the coding and analysis of open-ended responses. Traditionally, this work has been labor-intensive, but AI can now categorize and interpret textual data more efficiently than humans—especially at scale. Like Natural Language Processing (NLP) models, LLMs have succeeded remarkably in replicating the quality of human coders while reducing direct costs.

This has the potential to reduce the workload for researchers and accelerate the analysis of qualitative data—a significant breakthrough that can revolutionize one of the most challenging and labor-intensive survey analysis tasks. Even when benchmarked against traditional machine learning approaches, LLMs do just as well and require less statistical and programming knowledge to implement. 

Generative AI can also transform how we design surveys. By analyzing large datasets and previous survey responses, LLMs can help draft more effective, pointed questions and even suggest variations to reduce bias or improve clarity. This idea, presented by multiple research teams at AAPOR, is built around the concept that LLMs can be trained to learn the principles of good questionnaire design and aid researchers in writing questions with that advice.

While it doesn’t seem likely that LLMs will replace trained survey methodologists any time soon, a well-trained LLM can generate first-draft questions for standard topics when trained on question design best practices. The idea would not be to replace standard survey methodologists with LLMs, but to help guide people designing surveys to best practices without requiring additional oversight: a situation where LLMs might raise the floor on research quality, not the ceiling. 

LLMs can help impute missing data to complex, multicategory, and even open-ended questions. By analyzing patterns from existing responses, LLMs can estimate answers that were never collected. For instance, I’ve seen LLMs predict public opinion trends—like attitudes toward same-sex marriage—using historical survey data, even from periods when such questions weren’t asked.

This ability to “fill in the gaps” has opened up new ways to study long-term trends, giving us a clearer picture of societal change over time, even with incomplete data. Having the ability to retrodict public opinion, especially when the LLMs are trained on decades of public opinion data and individual socio-demographic trends around those opinions, should also allow us to explore perspectives from underrepresented or hard-to-reach populations.  

Another exciting frontier is using AI-powered chatbots to guide respondents through surveys. This has the potential to be a game changer for improving response quality and completion rates. These chatbots can clarify questions, prompt for more detailed answers, and create a more interactive survey experience. This could help reduce respondent fatigue and improve the depth and quality of the data we collect.

Further, this development would allow for more personalized tailoring of the survey experience to specific respondent needs and limitations, potentially improving the respondent experience during the survey and the quality and consistency of the responses provided. There are some concerns about how respondents will react to the chatbots—will they view them as annoyances like customer service agents? Will they react more favorably? While the early evidence is good that they’re an improvement, this is still a burgeoning field of study. 

AI also has a role to play in ensuring data quality, which is one of the major challenges in large-scale surveys. LLMs can be used to detect fraudulent or lower-quality responses by identifying patterns in open-ended answers that don’t match genuine human behavior.

I think we’re only beginning to tap into this area, but it’s crucial for maintaining the integrity of survey data, especially in the digital age, where responses generated by AI could skew results. Ironically, AI is part of both the potential problem and solution here. 

I’m optimistic about the future of AI in surveys, but I also believe that a balanced approach is necessary. LLMs are potent tools that can greatly enhance what we do. Still, they shouldn’t replace learned expertise from trained survey methodologists and social scientists who balance technical skills with human insight and years of practical experience.

As we continue exploring these technologies, we must remember the limitations outlined above and proceed cautiously. But at the same time, we must be aware that AI models will be another important tool in the methodologist/researcher’s toolbox. We need a lot of experimentation and rigorous testing to learn how to use it effectively and to understand the biases it introduces. 

As researchers, we need to think through some of these big questions and design research agendas around answering them. Some promising areas include understanding: 

• How adaptive chats and surveys impact respondent behavior
• What biases are introduced when coding open-ends with LLMs
• Whether we can build LLMs that can generate completely synthetic respondents that capture the nuances and complexities humans are known for

Doing these things rigorously and with an eye toward transparency is the only way to advance the state of knowledge in this field honestly and is the way forward for AI and survey research. 

As we can already see in AI, there will always be people interested in promising the moon with each innovation and, simultaneously, people approaching each new technology with the unmoored skepticism of modern-day Luddites. The future of AI in survey research is undoubtedly exciting, but as with any transformative technology, we need to approach it with a clear-eyed view of its strengths and weaknesses. If we can stay grounded in the real-world complexities of human behavior while leveraging AI’s efficiency and scalability, I’m confident that we can unlock new insights that benefit both researchers and the public.