When you think about sustaining an evidence-based practice, what comes to mind?

We have been working through this ourselves at the SISEP Center. According to Rabin (2008), sustainability is:

“To what extent an evidence-based intervention can deliver its intended benefits over an extended period of time after external support from the donor agency is terminated.”  

We get asked to define sustainability all of the time. What does it mean? What does it look like? What do we need to do to get this practice to be sustainable? So we thought, why not ask ChatGPT (an artificial intelligence chatbot)!

Implementation Specialist: ChatOn, define sustainability of an evidence-based practice in schools with citations.

ChatOn: Sustainability of an evidence-based practice in schools refers to the ability of the practice to continue over time, with fidelity to the original implementation, and to produce positive outcomes for students. It involves the establishment of systems and structures that support the practice, including ongoing professional development for teachers, administrative support, and resources for implementation. Sustainability also requires a commitment to monitoring and evaluating the effectiveness of the practice, making necessary adjustments, and sharing successes with stakeholders to build support for continued implementation. Ultimately, sustainability ensures that evidence-based practices become a routine part of the school culture, contributing to improved student outcomes and long-term success.
(Fixsen, et.al., 2005) (Durlak, et.al., 2008) (Greenhalgh, et.al, 2004) (McLeod & Weisz, 2010)

Not bad actually, and we were pretty impressed. But it does not get to the heart of the questions on sustainability for you as an agency and that is because sustainability is specific to your context and is a dynamic process, not linear.

Defining Sustainability

As you begin your journey to defining sustainability (without ChatGPT), take into consideration the five key constructs that describe organizational sustainability (Moore, Mascarenhas, Bain, et al., 2007).

1. After a period of time,
2. the program, practice, strategy, and/or implementation strategies continue to be delivered and/or
3. individual behavior change (i.e., teacher, student) is maintained;
4. the program/practice and individual behavior change may evolve or adapt while
5. continuing to produce benefits for individuals/systems.

As an implementation team, consider how long you have to implement and what target behaviors you want to see and by when. Our guess is, as you address these questions, you will discover that sustainability is not about the end goal (linear) as much as it is about maintaining the implementation and continuous improvement (dynamic process).

When thinking about sustainability from the linear perspective, sustainability is considered more of an outcome or reaching a final stage (Lennox, 2020). The linear perspective is the perspective on which sustainability research focuses the most. One of the main limitations of this perspective is that it may limit the results of a practice or program because it does not focus on how and why outcomes were reached (e.g., needed adaptations and adjustments).
 
The dynamic perspective views sustainability as a process that considers context, adaptations or modifications in the implementation process, and system interventions. It supports building adaptive capacity for the implementing bodies to make changes based on need (Lennox, 2020). But there are limitations as well – without outcome variables, it is difficult to determine when a program or practice is considered sustained.

Why sustainability?

The SISEP team also realizes sustainability's importance in implementation research and practice. It’s one of the biggest challenges in implementation work, no matter the industry or the setting. For researchers, having a standard definition provides a focus for the research. It creates a path for developing measures that can support implementation practice. A clear definition can help practitioners support organizations by actively thinking about sustainability. If sustainability is not kept at the forefront, we can end up in the “implement and forget” mindset. Many of us have experienced “implement and forget” - when all the time and effort is spent getting an innovation started and then, over time, it fails and is forgotten. This is frustrating to all those involved in the process and a waste of time and resources. Constantly introducing new initiatives without working to sustain them can also damage future partnerships/relationships between individuals, organizations, communities, and funders/investors that have a stake in the implementation process.

What might a linear approach to sustainability look like?

So what happens when an organization “reaches” sustainability as defined in the linear perspective? Let’s look at the SouthWest Regional Center and its implementation of a new literacy coaching program.

Over the last four years, the SouthWest Regional Center has focused on implementing literacy coaching across six local educational agencies. The center partnered with the six school districts, their state department of education, an institution of higher education, and the local reading associations to ensure the involvement of various critical perspectives and ongoing support. The implementation team addressed barriers and examined their capacity (using the Regional Capacity Assessment) and fidelity data (coaching walkthrough data). At their last monthly meeting, they celebrated that their trending implementation data showed positive growth and could not identify any specific barriers to address. They had “reached” sustainability, so they planned to switch their meeting cadence to twice a year and examine data collected on a quarterly basis asynchronously.

With this linear approach to sustainability, let’s take a look at what happens.

In June, at the end of the academic year, the Superintendent of Instruction and Curriculum and champion of the literacy coaching program retired. Within three months (September), the center had yet to fill the superintendent position. The Assistant Superintendent of Instruction did not take the lead with the implementation work since he was not part of the implementation team and had not been part of any communications regarding the program. With no interim superintendent in place, it was not until December that an implementation team member approached the regional center’s assistant superintendent and newly hired superintendent regarding concerns with the literacy coaching program. The team reconvened and discovered that, within the time of the superintendent leaving, no fidelity data was collected at the six school districts, two school districts had stopped utilizing their regional center literacy coach, and district leadership was unclear on the purpose and core components of the coaching program.

While turnover in organizations is often a challenge to sustainability, it is not the only barrier from a linear perspective. The team may also later encounter issues with changes to funding, educational policy, participant demographics, and/or competing initiatives that were not considered before they reached their linear definition of sustainability. 

How could this look different with a dynamic approach?

If the SouthWest Regional Center team would have considered a dynamic approach to sustainability, the team may have celebrated their data while also continuing to discuss potential gaps or policy changes that could hinder the program from being continuously implemented at its expected level. The focus of sustainability would be an evolving one, allowing the team to focus on sustainability by reviewing team membership and communication processes. They could also use their data to make decisions regarding the need to revisit previously explored implementation stages (i.e., exploration, installation, or initial implementation) to address barriers.

Next steps for you

Rather than utilizing ChatGPT to define sustainability for you and your organization, take some time as a team to define what sustainability looks like for each of the programs or practices your organization is implementing. As you do this, reflect on timelines, the behavior changes that need to occur, what processes need to be written down, and discuss potential gaps in your current structure.
 

References

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.

Durlak, J. A., DuPre, E. P., & DuBois, D. L. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3-4), 327-350.

Greenhalgh, T., Robert, G., MacFarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581-629.

Lennox, L. (2020). Sustainability. In P. Nilsen & S.A. Birken (Eds.), Handbook of implementation science (pp. 333-366). Edward Elgar Publishing Limited.

McLeod, B. D., & Weisz, J. R. (2010). The therapy process observational coding system for child psychotherapy strategies scale. Journal of Clinical Child & Adolescent Psychology, 39(3), 436-443

Moore, J. E., Mascarenhas, A., Bain, J., & Strauss, S. E. (2017). Developing a comprehensive definition of sustainability. Implementation Science, 12(110). https://doi.org/10.1186/s13012-017-0637-

Rabin, B. A., Brownson, R.C., Haire-Joshu, D., Kreuter, M. W., & Weaver, N. L. (2008). A glossary for dissemination and implementation research in health. Journal of Public Health Management and Practice, 14(2), 117-23. doi:10.1097/01.PHH.0000311888.06252.bb.