Skip to main content

The Survey Surge: Balancing Quantity with Quality in Research 

Expert View

Author

David Dutwin
Director, Center for Panel Survey Sciences

October 2024

Survey research has become one of the most vital intelligence tools of the 21st century, but like many things, you get what you pay for. There are smart ways to field surveys efficiently, but at the end of the day, quality matters, as do real people. 

There is no shortage of political horserace surveys these days. In fact, researchers have shown that over the last 30 years, the number of political surveys has exploded from just a few dozen per election cycle to many hundreds, even thousands, fielded at every level and for nearly every contest.  

Of course, this is part of a larger trend: We have become a data-driven world, and this is very much a good thing. Data, gathered and analyzed with best practices and rigor, find the truth (or very close to it) and shine a light on what is hidden: hidden opinions, hidden behaviors, hidden relationships between things (for example, the rise of social media and declines in mental health)—the list goes on and on. Good data lead, far more often than not, to good decisions, good policy, and better lives—if we take smart and logical actions from what the data tell us.

“Data, gathered and analyzed with best practices and rigor, find the truth (or very close to it) and shine a light on what is hidden: hidden opinions, hidden behaviors, hidden relationships between things.”

Director, Center for Panel Survey Sciences

“Data, gathered and analyzed with best practices and rigor, find the truth (or very close to it) and shine a light on what is hidden: hidden opinions, hidden behaviors, hidden relationships between things.”

But the enormous growth of survey data is not always a good thing. There are countless examples in other domains where appetites for something get so acute that the void is filled with products made quickly, cheaply, and quite often, in a fashion where corners are cut and quality takes a backseat.

This is what Nate Cohn's New York Times newsletter about online polls reports. The most trustworthy polls find respondents via random sampling, a method by which each American has an equal likelihood of being surveyed. In contrast, Cohn details the rise of online convenience polls, which eschew random sampling and instead get respondents through internet advertisements, convenience lists (think airline miles partners), and internet technologies. 

While such approaches can be done better or worse, the trend has been toward lower quality due to, as Cohn reports, machine responses (bots), bogus respondents, internet segmentation, and other factors. It's reached the point that most professional associations of pollsters and market researchers have launched efforts to combat the rising tide of bad data and invalid results.

Cohn reports that there have been, and still are, highly inexpensive convenience polls that can be relatively accurate. One major challenge, however, is a lack of transparency in the methods by which many of these polls are conducted. Without transparency, there is no way to judge the validity of the research, and therefore, no way to know whether we can trust the findings. And it seems the more quality declines, the less transparent organizations are about how they conducted their research. In the end, quality matters, and quality simply cannot be assessed without transparency.

And at the end of the day, it is the quality of the research that matters most. Research that is prone to bots and bogus respondents will always struggle to be valid and reliable. If anything, random sampling is even more important today than it was a half-decade ago, given how segmented society has become and how specialized our interests are. Now more than ever, we need tools that capture as much of the full diversity of Americans’ attitudes, opinions, and behavior as possible.


Suggested Citation

Dutwin, D. (2024, October 9). The Survey Surge: Balancing Quantity with Quality in Research. [Web blog post]. NORC at the University of Chicago. Retrieved from www.norc.org.


Tags

Research Divisions

Departments, Centers & Programs



Solutions

Experts

Explore NORC Research Science Projects

Analyzing Parent Narratives to Create Parent Gauge™

Helping Head Start build a tool to assess parent, family, and community engagement

Client:

National Head Start Association, Ford Foundation, Rainin Foundation, Region V Head Start Association

America in One Room

A “deliberative polling” experiment to bridge American partisanship

Client:

Stanford University