What is Systems Change?
As an external evaluator I’ve done heaps of service evaluations – whether it’s a sports programme aimed at young people, an advice service for people navigating the benefits system or a peer support group for trafficked women. I find I am increasingly asked to evaluated systems change programmes. Sometimes a programme is specifically about systems change and often a programme blends both service delivery and systems change elements.
But what exactly does systems change mean? While there are several definitions, the common thread refers to addressing root causes of social problems, recognising that people and the mechanisms they operate in are interlinked.
Examples of recent evaluations I’ve worked on aimed at addressing systems challenges include:
Lack of legal aid for refugees seeking family reunion
High numbers of young people who drop out of the education system following exclusion from school
Poor experience of mainstream services amongst children and young people who’ve experienced trauma
This blog post outlines some of the differences I’m noticing between how I approach a system evaluation compared to a service evaluation.
Similarities between Service and Systems evaluation
Theory of Change still helps – it is still important to articulate an intended Theory for Change at the outset of a systems change programme. It is arguably more important for delivery partners to revisit Theories of Systems Change more regularly than for services because of ever-shifting contexts (more on this later!)
Outcomes still make sense – as with service evaluations, an outcomes approach is still helpful. Outcomes can be for people, organisations and ecosystems
Evaluation methods are broadly similar – surveys, interviews, focus groups and ethnographic methods are applicable in both service and systems evaluations. Often the fieldwork will take place with decision-makers, policymakers and system architects rather than people who access services, but the evaluation skill set required is very similar
Differences between service and systems evaluation
Assumptions are harder to pin down for systems change evaluations. It can take a considerable amount of time for stakeholders involved in systems change to get “on the same page” about which system(s) or part of a system(s) needs improving. Often, at the start of a programme, there is an assumption that stakeholders (often multi-agency) have a shared understanding of the change required. But this is often challenged once the programme gets into full swing, with different stakeholders having a different perspective on what is required or, indeed, what the problem is! This differs with service evaluations where typically stakeholders have a shared understanding of the need and some evidence about how best to address it.
Need to be clear about the scope of influence - in service delivery programmes, it is realistic to expect some short- and medium-term outcomes to be achieved within the scope of a programme. However, for systems change programmes (which typically run for 1-3 years) it is not always feasible to change an entire system within this timeframe. It can be helpful to think of systems change evaluations documenting how things are happening at a point in time but acknowledging that intended changes may (or may not!) happen in the future.
Indicators are often attitudinal - There are “invisible” parts of systems programmes that include influencing hearts and minds. It is important that evaluation efforts don’t take this work for granted. For example, understanding the power dynamics (who makes decisions about systems?) and the values that stakeholders have (are they aligned?). Understanding these things takes time. And influencing them is often a pre-condition to long term systems change.
The ground is moving! Within a 1-3 year time frame, contexts can change. New governments, global pandemics, economic downturns, Brexit….the list goes on! This can be the case for delivering (and evaluating) services and systems. There is no silver bullet for building these changes into an evaluation framework but I tend to find that a developmental approach (positioning evaluation as an iterative learning process, part of the team rather than an objective outsider) to evaluation makes more sense in systems change evaluation work.
Systems change programmes need more mechanisms for learning – all programmes and evaluations should be about learning. Of course! But I’ve noticed on systems change programmes that extra space and resource is needed for stakeholders to reflect. Systems change is not about “delivering to” an audience – it is more relational, there is a need for exchange. I’ve noticed that once systems delivery gets underway, there’s a flow of information coming back to delivery professionals from the systems professionals they seek to influence. The best systems change programmes make space to reflect on what they are hearing, how they are feeling and what opportunities this information flow provides.
In summary, service evaluation and systems change evaluation uses much the same skills and data collection methods. However, there are some important things to note when evaluating systems change programmes including supporting delivery teams to challenge their assumptions, getting clear on the intended influence of a programme and taking a reflective learning approach to evaluation. It is important to acknowledge shifting contexts and the flow of knowledge between stakeholders – including systems change “makers” who are often simultaneously learning more about a system while also seeking to influence it.
Emma Roberts specialises in qualitative research and evaluation for organisations that want to make the world a better place. For more information please contact emma.roberts@cloud-chamber.co.uk
Comments