Are Surveys Dead? Long Live Surveys
Dan Gaylin
President & Chief Executive Officer
July 2024
In today’s rapidly evolving data landscape, the role of surveys in research has been both challenged and validated in unexpected ways.
So, as a researcher and leader of an organization whose roots are in survey research, how do I think about the utility and viability of surveys today; their limitations in a dynamic data landscape, and their irreplaceable value in certain contexts?
A Surprising Endurance
Ten or 15 years ago, my colleagues and I at NORC believed that surveys would become a much smaller part of our business by 2024, with the proliferation of other data sources and rapid development of capabilities to access and analyze these data. We anticipated that federal funding for surveys would diminish as part of this larger trend, and that researchers in general would move away from surveys.
To paraphrase Mark Twain, the rumors of surveys’ demise have been grossly exaggerated! It is true that new data sources and methods have grown rapidly, and NORC and similar organizations are doing very well deploying our data science and analytics expertise on a plethora of different types of data. We are working with both new and long-standing clients on projects using many different types of data and analytic techniques, and as we expected, these newer parts of our research portfolio have grown extensively over the past 10 years. But at the same time, we have seen sustained—and in some cases growing—interest in survey research from both our traditional clients as well as new partners.
The resilience of our survey research portfolio has surprised us, but it is also very encouraging—it shows that the research marketplace is increasingly embracing the concept of “fit for purpose” in which we use the right data for the right questions to get the answers and insights we need. Sometimes traditional survey data will be the best, sometimes newer types of survey data will be best, and sometimes administrative data or claims data or other types of data generated by processes and captured by computers will be best. And sometimes, a combination of these data will be best.
This makes it a very exciting time to be a data scientist! There is a wealth of available data, and a powerful array of tools and methods to analyze these data. So, while surveys are not the predominant data source they once were, they still hold a very important place in research, especially research aimed at improving the lives of people, which, of course, is what NORC is all about.
“While surveys are not the predominant data source they once were, they still hold a very important place in research, especially research aimed at improving the lives of people, which, of course, is what NORC is all about.”
Why Surveys
One reason surveys remain essential is because they provide insights that other data sources simply cannot. For instance, administrative and transactional data can tell us about past behavior, but if you want to know what people plan to do, or really understand the attitudes, opinions, and beliefs that underpin current and future behavior, you need to actually ask them questions about these things.
Consider the example of energy use: before smart meters, we relied on surveys to gauge household energy consumption. Today, smart meters provide precise data on energy usage, and credit card transactions show purchases of green technology. However, these data sources can't tell us if someone plans to buy an electric car, or their views on a willingness to have a wind farm in their community. For those questions, surveys are indispensable.
This example highlights the "fit for purpose" approach. The best data source depends on the question at hand, and sometimes, surveys are the most effective method; sometimes computer-generated data is more useful. Often, a combination of data sources yields the most useful research.
Declining Response Rates, Higher Costs
Despite their utility, surveys do face challenges. An important aspect of surveys that gets a lot of attention is response rates, which have been declining for years, exacerbated by recent factors such as political polarization and growing audience reluctance to participate in surveys. People are also less likely to respond if they believe the survey is biased (or simply are administered by an organization that doesn’t match their social or political beliefs), which further complicates effective data collection.
Fortunately, to date, we have seen only limited evidence, in particular situations (such as very basic, often lower quality political polls), that the decline in response rates has produced a corresponding increase in response bias.
Nevertheless, because of declining response rates we need to have larger samples and more contact attempts to yield a similar number of completed cases as in the past. And this, combined with a variety of other economic reasons, has increased the cost of conducting surveys over time as well. It simply takes more time and money to obtain sufficiently large, representative samples that high quality survey research requires.
Survey Research Adaptations
Survey research has adapted by looking for efficiencies that balance data quality with survey costs. Three significant trends helping strike this balance are the shift to using more multi-mode surveys, using non-probability samples, and linking to external data sources.
Survey researchers have increasingly found that conducting surveys in a multi-mode format leads to superb results across many dimensions. And respondents often have preferences for participating in a survey via telephone, in person, web, text. In fact, NORC created AmeriSpeak, our multi-mode national probability panel, to conduct rigorous multi-mode surveys that are representative of the American general population, at reasonable costs. We’ve also added specialty panels such as Foresight 50+, the largest high-quality research panel of Americans aged 50 and over, and Amplify AAPI, the largest, most representative panel of Asian American, Native Hawaiian, and Pacific Islander communities.
Another trend in survey research is the increased use of opt-in surveys, which are part of a larger group of “non-probability” surveys. These surveys can be quite useful for understanding the group of people who participated, but they are often used carelessly, where the researchers claim that the results are representative of a more general population than the survey actually includes. Often the surveys are “corrected” using basic weights to reflect the proportions of various subgroups in the underlying population, but that doesn’t address the coverage bias issues these surveys may contain. Again, there are solutions. For example, NORC has developed TrueNorth, which uses more sophisticated weighting techniques based on AmeriSpeak probability surveys to calibrate non-probability surveys, so they are representative of the general population. This also can be very cost-effective since one of the main reasons non-probability samples are so popular is that they are inexpensive to conduct.
A final trend is toward more integrations of survey data with external data sources through data linkage. Combining survey data with administrative, transactional, or other datasets can provide a more comprehensive understanding. This holistic approach often yields insights that wouldn't be possible with a single data source. A classic example here is linking survey data on self-reported health status with medical claims data, which can help us understand the degree to which people’s self-assessment of their health is correlated with their underlying health conditions.
Linking survey data with administrative data also allows surveys to focus on the most essential questions that need to be asked—and cannot be obtained from another source—so that surveys can re-allocate limited resources more effectively to ensure higher sample sizes and better data quality.
The Future of Surveys
In a world awash with data, it is essential to be careful and thoughtful in using any data source. Embracing the concept of fit for purpose is also key. While surveys may not dominate policy research as they once did, they remain an indispensable tool. Like any method, surveys have evolved and play a different role in effective research today than they did just five years ago, and they will continue to evolve. But this evolution demonstrates that traditional methods can adapt and thrive even as they face new challenges. I am convinced that surveys will continue to make very valuable contributions to our understanding of the world around us for many years to come.
I’m curious to hear your thoughts on surveys or any of the other topics we explore in this forum. Plus, if you’d like to stay up to date with the great work happening at NORC, follow along on LinkedIn. You are also welcome to follow me on LinkedIn for more musings like this one in the future.
