Introduction to Implementation Capacity Assessments
For leaders and managers, reliable and valid measures of implementation capacity are essential for 1) planning, 2) supporting, and 3) evaluating the use of effective innovations in complex human service systems and organizations. For scientists, reliable and valid measures are necessary for conducting rigorous research on implementation variables. For several decades, members of the National Implementation Research Network (NIRN) have been developing implementation capacity in human service systems and organizations. These experiences coupled with ongoing reviews and syntheses of the implementation evaluation literature provide a rich experiential and data base to inform the development of the implementation capacity assessments.
Current attempts to assess components of implementation tend to use 1) general measures that do not specifically address core implementation components (e.g. Landenberger & Lipsey, 2005; Mihalic & Irwin, 2003); 2) measures specific to a given innovation that may lack generality across programs (e.g. Olds, Hill, O'Brien, Racine, & Moritz, 2003; Schoenwald, Sheidow, & Letourneau, 2004); or 3) measures that only indirectly assess the influences of some of the core implementation components (e.g. Klein, Conn, Smith, Speer, & Sorra, 2001; Aarons, Cafri, Lugo, & Sawitzky, 2012). Based on the Active Implementation Frameworks, the Implementation Capacity Assessments have been developed to measure the core implementation components that are generalizable across human service domains and programs.
Figure 1. Implementation Capacity Assessments developed to monitor progress and implementation outcomes at each level of a complex state education system.
As shown in Figure 1, assessments include the State Capacity Assessment (SCA), Regional Capacity Assessment (RCA), District Capacity Assessment (DCA), Drivers Best Practices Assessment (DBPA), and Observation Tool for Instructional Supports & Systems (OTISS). Data gathered from these assessments provide a wealth of information that can be used to support the dynamic bi-directional relationships in the cascade between the state, region, district, building, and classroom.
Importantly, the capacity assessments are “action assessments.” That is, each item in each assessment is relevant, readily assessable, and immediately actionable. Baseline assessments inform planning by pointing to aspects of implementation capacity that are strengths or need improvement. Action planning is based on prioritizing items, establishing timelines, and assigning capacity development tasks to individuals and teams. Repeated use of the capacity assessments enables a system or an organization to monitor changes over time. The result is improved implementation capacity to support effective innovations. The goal is to achieve improved outcomes for children, families, individuals, communities, and society on a useful scale.