HRSA/MCHB rely on strong performance measures to maximize the value of services to mothers and families.

To achieve its vision of an America where all mothers, children, and families prosper and reach their fullest potential, the Maternal and Child Health Bureau (MCHB) of the Health Resources and Services Administration (HRSA) relies upon evidence-based strategies to implement programs and monitor their effectiveness. Their efforts include supporting states and jurisdictions through the Title V Maternal and Child Health Block Grant Program. They assess the performance of those grants using the Discretionary Grant Information System (DGIS).

NORC scientists helped MCHB ensure that their measures addressed critical issues, were feasible to collect and use, and were valuable to the agency and its grantees.

MCHB contracted with NORC at the University of Chicago to assess the validity, reliability, and utility of the DGIS performance measures and data. Between September 2020 and September 2021, we conducted a comprehensive, mixed-methods evaluation of the relevancy, appropriateness, and validity of a subset of the DGIS performance measures and developed recommendations for revisions, enhancements, and additions. Our approach to assessing the Core, Capacity Building, and Population Domain measures and Financial and Demographic forms was guided by the National Quality Forum (NQF) performance-measure evaluation criteria:

• usability and current use
  
• importance to measure and report (e.g., prevalence, evidence base)
  
• scientific acceptability of measure properties (e.g., validity)
  
• feasibility for grantees to report
  
• comparison to related and competing measures
   

To answer questions about the importance and feasibility of the measures, we:

• reviewed the total number of grantees reporting each measure
  
• determined the frequency and distribution of each measure type and its alignment with DGIS program goals
  
• assessed variability in performance rates among select measures
  
• examined patterns of missing data (e.g., low response rates)
  
• assessed the form logic to examine whether the forms collect data at the right level and follow best practices
   

To address questions about the current usability, importance, and scientific acceptability of the DGIS measure set and to identify related or competing measures, we reviewed background documents provided by MCHB, conducted 10 discussions with MCHB stakeholders, and participated in seven virtual internal workgroup meetings with MCHB staff.

MCHB received actionable suggestions on improving the set of measures, supporting grantees’ collection and use of them, and longer-term strategic refinements.

NORC generated a list of recommendations for improving the DGIS measures, ranging from short-term solutions to longer-term strategic planning initiatives that may involve an extensive redesign of the measure set. In the short-term, NORC recommended that MCHB focus on improving the reliability and validity of existing measures and reducing burden where possible through these revisions to the 2022 OMB package resubmission:

• develop DGIS-focused training programs and ongoing technical assistance (TA) for staff and grantees to provide an orientation to the measures and teach them how to submit data
  
• provide standardized definitions and instructional guidance for confusing measures and terms
  
• provide more informative response categories for specified measures
  
• develop and implement a process for validating the DGIS measures
  
  

In the long-term, NORC recommended that MCHB engage in strategic planning with leadership and staff to overhaul the DGIS measure set to make them more useful, valid, and less burdensome to stakeholders. These steps were also aimed at improving stakeholder buy-in:

• clarify the overall purpose of the DGIS measure set.
  
• reduce the number of DGIS measures.
  
• develop new measures that better capture grantee activities and MCHB core values.
  
• further explore and pilot alternate structures.
  
• create a more user-friendly data system so the DGIS data are usable and accessible.