Faced with choosing whether to self-evaluate or commission an evaluator? Using a well-thought-out checklist can be a game-changer, giving you and your stakeholders a systematic, transparent and consistent process to make decisions with confidence. This blog provides a simple checklist to help you get started with deciding.
In the ever-evolving charitable sector, organisations of all sizes are continuously seeking
ways to understand and enhance their processes, services and impact. For some of these
organisations, an evaluation is just the thing to enable their enhancement. While internal
staff can handle many of the tasks associated with an evaluation, there are situations when an external evaluator may be more appropriate.
Deciding who should lead an evaluation can become a complex undertaking, requiring
careful consideration and weighing of different choices. And most often, the decisions fall
into three categories:
1. Ethics – the agreed norms, values or standards around what is acceptable evaluation practice.
2. Equity – the degree to which anyone affected by what is being evaluated has what they need to either shape, contribute to, or own, the evaluation and its findings.
3. Practicality – the availability of resources and state of readiness of the context necessary for evaluation.
These three categories are intertwined, and often present conflicting courses of action. But carefully questioning these categories enables more balanced decision making – giving everyone involved the confidence to move forward with the best possible choices. Below is a non-exhaustive checklist based on the three categories followed by an example of its use.
Checklist Part 1: Ethics
Checking ethics is the most sensible starting point for deciding whether to commission and who is ultimately best placed to lead the evaluation. This involves questioning expectations surrounding the decision to evaluate in the first place, as well as what knowledge will be of interest, who holds this knowledge, and who can access it.
Checklist questions:
What is the evaluation seeking to address or answer?
Who are the stakeholders?
What do they want, expect or value?
What information is needed to provide answers?
Who can provide this information?
Who can gain the most access to this information?
Who can ensure that the information is safely provided, protected and treated?
Checklist Part 2: Equity
The past few years have seen an increasing focus on the advancement of equity through
evaluation activities. Emerging initiatives like the Equitable Evaluation Framework and the
Equitable Evaluation Collective UK are challenging those who commission, lead and use
evaluations to recognise and rethink those evaluation practices which, historically, have
reinforced exclusion, discounted diversity, and led to harm and distrust. Decisions about
who should lead an evaluation must factor in who can help to build in and maintain focus on equity throughout the evaluation.
Checklist questions to build in equity when shaping the evaluation:
Who can enable the involvement of the most diverse groups of stakeholders?
Who can balance differences in power between stakeholders, including service users, so that differences in what matters most are respected and managed?
Who can design an evaluation that is culturally and contextually appropriate and
methodologically valid?
Checklist questions to build in equity when implementing the evaluation:
Who can earn the most trust among the people contributing to the evaluation?
Who are these people more likely to be honest with?
Who can ensure that data collection gathers a wide range of perspectives, including often unheard voices?
Who can examine the data to highlight differential experiences, outcomes and drivers, for different groups who contribute to the evaluation?
Who can include diverse people and perspectives in sensemaking and learning?
Checklist Part 3: Practicality
Practicality appears last by design for two reasons. The first is to recognise that for some
organisations, especially small ones without ample funding, practical limitations are the
most considered factor. The second reason is to encourage organisations of all sizes to still interrogate ethics and equity. All too often, organisations rush to decisions about
commissioning based on practicalities like budget, capacity and timelines without giving due diligence to issues central to the very people and practices that will be involved - issues which, in my experience at both resource-limited and well-funded organisations, will inevitably emerge to affect every aspect of evaluation.
Checklist questions:
Who has the necessary evaluation or research capabilities?
Who has or can enhance capacity to lead the evaluation?
Who can add valuable perspectives?
Who is most affordable?
Who can fit in with where the focus of the service or initiative is at right now?
Using the checklist
Consider a charity faced with deciding whether to commission an evaluation to learn how to enhance their support for vulnerable young people. Specifically, they want to understand service delivery and young people’s experience and reflect generally on their service. Their small team usually shares internal learning responsibilities, with no dedicated lead. The team values the involvement of young people, staff, and sector collaborators in shaping and contributing to evaluation. They believe fitting evaluation into existing routines can enable involvement.
By considering ethics, the team recognises that their closeness to the service and young
people make them best placed to access and collect information. They briefly consider that an external evaluator could provide the distance they need to reflect on their service. But they conclude that safe access and safeguarding young people matter more than objectivity.
Equity questions allow the team to consider if their sector expertise and trusted
relationships with young people and collaborators are enough to design inclusive evaluation activities and obtain honest data. They also acknowledge that additional expertise could ensure the quality and validity of data collection and increase attention to equity.
After also weighing practicalities, the team ultimately decide to self-evaluate. They bolster their capabilities by enlisting one-off advice from an evaluator on Theory of Change, evaluation planning and managing data. They also consult accessible evaluation knowledge platforms including Better Evaluation and Inspiring Impact. By fitting evaluation into their routines, they preserve their capacity to prioritise service delivery.
Deon Simpson, DPhil, is an experienced Social Researcher and Evaluation and Learning Specialist, who was part of the panel at ChEW's event 'To Commission or Not To Commission'. ChEW members can watch the event here. Not a ChEW member? Join us today.
References:
1. Resnik, D. B. (2015). What is ethics in research and why is it important?
2. Equitable Evaluation Initiative (2021). Shifting the evaluation paradigm: The Equitable
3. Centre for Evaluation Complexity Across the Nexus (2021). The Complexity Evaluation
Comments