NORC Reviews College Ranking Methodologies
This article is from our NORC Now newsletter. Subscribe today.
October 2024
Vanderbilt University asked NORC to examine college ranking systems.
College rankings profoundly influence how prospective students and their families make educational decisions. Given that influence, Vanderbilt University asked NORC to independently, comprehensively examine how college ranking systems distill multiple abstract qualities about a college or university into easily digestible, numbered lists.
“We used a construct validity framework—which determines if a measure accurately represents the theoretical concept it's intended to measure—to assess whether the methodologies of five popular college ranking systems accurately measure the qualities that form an overall school ranking,” said Susan Paddock, an executive vice president and chief scientist in NORC’s Research Science division. “We discovered that many aspects of college rankings suffer from a lack of construct validity.”
Specifically, we found that:
- Ranking system methodologies and concepts are unclear.
- The rationale behind the relative weights of college attributes is unknown.
- Some critical components of the ranking process are highly subjective
- Data quality is inconsistent, hindering accurate assessments of various measures.
“We hope that our report will better inform students, parents, and institutions of higher education about the strengths and limitations of ranking systems and ultimately lead to changes that will improve the quality of information available.”
“Poor data is particularly problematic because it can undermine the ability to capture even well-defined concepts—like graduation rates, student debt, and value-added earnings—potentially skewing the scores,” said Soubhik Barari, a research methodologist in NORC’s Methodology & Quantitative Social Sciences department. “Whenever possible, we suggest implementing processes to improve data quality, such as a review of data sources, more rigorous statistical correction procedures, and institutional audit mechanisms.”
Our report recommends enhancing the clarity and transparency of the ranking processes.
Our report makes several other recommendations to address these validity issues and enhance the clarity and transparency of the ranking processes. They include enriching the rankings by adding visual elements; convening higher education experts and other stakeholders to establish common standards, measures, and definitions for future college rankings; or abandoning rankings altogether in favor of a tiered rating system. (While NORC is an independent affiliate of the University of Chicago, the university was not involved in this research at any stage, including commissioning the analysis, conducting the methodological review, synthesizing findings, or forming recommendations.)
“We hope that our report will better inform students, parents, and institutions of higher education about the strengths and limitations of ranking systems and ultimately lead to changes that will improve the quality of information available to those seeking to make a substantial investment of time and financial resources in their futures,” Paddock said.
This article is from our flagship newsletter, NORC Now. NORC Now keeps you informed of the full breadth of NORC’s work, the questions we help our clients answer, and the issues we help them address.
