In recent years the social sector has given increased focus towards equity in grant making and in how programmes are designed and delivered. The move towards rebalancing power within the social sector also needs to include holding up a mirror to how we design evaluation and how we collect and use data. This is what has brought individuals from the Charity Evaluation Working Group (ChEW), NPC and The Social Investment Consultancy (TSIC) together to establish the Equitable Evaluation Collective (EEC).
On 1st February 2023 the EEC hosted a free workshop, for people working in the UK social sector, to discuss why and how the sector can move toward greater equitable evaluation practice. The workshop, which was attended by over 40 participants, included breakout discussion groups, lightning talks and a panel discussion with people with experience in equity led evaluation approaches. Here we’ll draw together some of the themes.
Commissioning, planning and designing evaluation
Workshop participants shared that practitioners striving for equitable practices in their work are often limited by the terms on which the evaluation has been commissioned. This includes: tight timelines and budgets that preclude relationship building, community-led evaluation or meaningful involvement; prescriptive approaches using predetermined evaluation questions and methods; and evaluation plans that don’t allow sufficient time to see longer term outcomes and impact realised.
During the panel discussion, it was highlighted that not enough attention is given to context and systems in the design of evaluations. We need to consider whether the programmes themselves are perpetuating ideas that cause harm.
Language matters. For example, using jargon can reinforce the technical expertise of the evaluation team over other types of expertise and create barriers to engagement.
Data collection and analysis
The use of standardised and validated quantitative tools can cause harm if applied without consideration as to whether the questions are appropriate for the cohort and sensitive to their needs. A trauma-informed approach to design and delivery can reduce this risk by considering how questioning can replicate traumas, supporting the autonomy and control of the participant, and putting their wellbeing first.
Equitable approaches move beyond checking the credibility of findings with participants to centring those most affected by the issue in question. For example, peer-research, co-production, or community research that recognises the knowledge and experience within communities as an asset. These approaches can open space for people from the community to share ownership of the narrative.
Reviewing, learning and acting on findings
Top down evaluations are usually determined by the commissioner, or in tandem with the evaluator. In these scenarios the objectives, questions, methods and even the format of final outputs are pre-determined in order to serve the decision-making, or accountability needs of organisations.
Workshop participants raised that decision making can sometimes operate on a different timeline to the evaluation process. This means that communities involved give time to data collection and share their stories, only for decisions about the future funding of programmes affecting them to have already been made or be delayed. Recommendations may be ignored, and no action taken based on the needs highlighted.
These extractive ways of operating are disempowering for evaluation participants and reduce trust. They reinforce concerns of evaluation being divorced from learning and real influence on decision making. Equitable evaluation recognises that people that bear the brunt of the issue should have a voice in driving ideas and solutions.
The evaluation workforce
One of the challenges to improving equity in evaluation is the lack of diversity within the professional workforce. Having a profession that can draw on evaluators that are representative of the communities participating removes the stigmatising barrier between who is evaluated and who does the evaluating, and strengthens the work overall.
Evidence standards
The origins of evaluation in the social sciences drives our sector’s understanding of what is robust evaluation. Anyone familiar with designing or delivering evaluation will be aware of the focus on study design in the evidence hierarchy and the prime position given to quantitative and experimental designs. Over the past 20 years or so, the social sector has produced different versions of standards of evidence. Discussion during the workshop highlighted a need for more expansive thinking about what counts as robust evidence and a challenge to the perception that all quantitative approaches are naturally robust and methods that are qualitative or participatory are not.
Where next for the Equitable Evaluation Collective?
It’s important to learn from other sectors where we can, and to amplify existing efforts within the social sector. EEC is keen to do research and further scoping to build on what is already known and to ensure ideas come from diverse voices.
Workshop participants brought a wide range of ideas about what is needed in the sector, and there is a clear interest in seeing a sector-wide initiative like EEC catalyse the move towards equitable evaluation.
Suggested developments include:
Facilitate safe spaces where evaluators can share what they are trialling and learning – this early activity often remains hidden.
Facilitate learning sessions where evaluators can share ideas and resources.
Bring greater understanding of the experiences of marginalised groups in the evaluation process, and what they think needs to change.
Create opportunities for joint action from social sector evaluators on equity.
Be a voice in facilitating funders and commissioners to align their practices with equitable evaluation and for accountability.
Support evaluation teams with practical guidance
Our thanks to everyone that participated in the workshop for sharing their experiences and perspectives. The video of the panel discussion is available here.
Keep in touch with the Equitable Evaluation Collective
Please join our mailing list to keep informed about developments with this work and future opportunities for collaboration. You can sign up here
You can also email us at equitableevaluation@gmail.com
Development of the EEC has so far been on a voluntary basis and to build on meeting the needs highlighted we’d appreciate grant funding from equity-minded funders. We welcome any discussions on funding and/or collaboration opportunities.
Comentários