CHICAGO, September 9, 2024 — A new report examining methodologies used in various rankings of U.S. colleges and universities provides recommendations to support transparent evaluation of higher education institutions.

For decades, college rankings have sought to distill many abstract qualities into a digestible, numbered list. For this analysis, researchers reviewed the methodologies of five popular rankings to assess whether they accurately capture the qualities that form an overall “ranking.”  

The analysis was independently conducted by researchers from NORC at the University of Chicago, a 501c3 organization. Funding for the analysis was provided by the Vanderbilt University Office of the Chancellor. 

Key findings include:

• Methodologies are unclear, and the rationale for the relative weights of various attributes included in rankings is unknown. The concepts captured by the rankings also are not clear. Researchers say this makes it impossible to know exactly what is being measured and how much it should “count” in a final assessment. 
• Data quality is inconsistent, which hinders accurate assessments of various measures, even openly defined ones like graduation rates, student debt, and value-added earnings. 
• Some factors assessed are highly subjective but are critical components in the ranking process, which makes it difficult to establish definitive comparisons between institutions.

NORC’s analysis provides recommendations to significantly enhance clarity and transparency in the assessment and ranking process, such as convening higher education experts and other stakeholders to establish common standards, measures, and definitions for future college rankings.

NORC is an independent affiliate of the University of Chicago. The University of Chicago was not involved in this research at any stage, including commissioning the analysis, conducting the methodological review, synthesizing findings, or forming recommendations.