Evaluation - Data Cymru

Our guide to EVALUATION

This guide will:

Support your understanding of what evaluation is and why it is important for your projects, programmes and policies.
Give you the 'basics', so you understand why and when you might undertake evaluation.
Provide you with direction as to the approaches and processes you might use to undertake effective evaluation.
Provide pointers to further guidance and support.

This is not a definitive guide to evaluation, but we do aim to convey the key steps in undertaking effective and proportionate evaluation and provide practical tips on how this may be done.

The Magenta Book – guidance for evaluation

For comprehensive UK guidance, we highly recommend that you refer to the Magenta guide, which is Her Majesty’s Treasury guide on evaluation and use it as your “go to” evaluation resource.

So, what is evaluation and when is it used?

- Magenta Book -

In summary, evaluation is a process carried out by organisations to understand:

the progress and quality of their work (specific activities, interventions, outputs, and ways of working);
outcomes and the impact or difference they are making (e.g. to service users, stakeholders or funders); and
opportunities for further improvement and attention.

Evaluation can be used to assess the quality and impact of all types of organisational activity and service delivery from large scale policies to smaller programmes, projects and activities. It can take place before, during or after a programme is delivered but they are most effective when designed early, alongside programme planning.

Evaluation Process

Here are the steps we suggest you might consider when planning and undertaking evaluation:

Visual frameworks or models to support evaluation

When developed early in a programme rather than specifically for an evaluation Logic and Theory of Change models can be dynamic and updated to reflect changes in plans processes and outcomes

To evaluate a programme, it is helpful to have an agreed reference against which to judge progress or success. What are we expecting to achieve and why? What changes will we expect to see? etc.

A visual representation of the planned activities, outputs and expected results can be a helpful way of planning and tracking progress and change. There are models or frameworks you can use to help you do this. These will set out or establish a programme “story” and describe how processes and results are planned sequentially, drawing clear causal links between each element in a programme. These are useful for planning programmes and interventions as well as understanding change once you come to evaluation. We cover:

Logic models: These are usually graphical representations of the chain of planned activities and processes, which show how desired outcomes come about.
Theory of Change: This is a detailed description and illustration of how and why a desired change is expected to happen from a “bigger picture” viewpoint. It focuses on attribution, and clarifying what interventions or activities are responsible for change.

Evaluation types

Your decision may be influenced by time available the level of evaluation planning you have done prior to implementing the project and what measures are most important to your organisation and stakeholders

A starting point in effective evaluation is deciding what type of evaluation you need to undertake. The questions you need answered and where you are in the project life cycle will determine the evaluation type required. We define five types of evaluation, which are:

Formative evaluation - these assess the feasibility and potential of a programme, policy or intervention.
Process evaluation – these assess how effectively a programme, policy or intervention is being, or has been, implemented.
Outcome evaluation – these assess the extent to which a programme, policy or intervention has achieved its planned outcomes.
Impact evaluation – these explore and assess longer-term effects, intended or otherwise, of a programme, policy or intervention. .
Economic evaluation – these assess a programme’s or intervention’s value for money.

Although there are different types of evaluations, very rarely does an organisation carry out one evaluation type. Many organisations need to use evaluations to gain an understanding of the processes; outcomes; impacts; and economic value of their work to inform and shape how they work in the future. It is a choice whether you measure one or two of these elements at one time or all of them at the same time.

For example, understanding the difference made to people’s lives and economic value may be a priority to you if you are commissioning a new intervention. This need would be supported by an impact and economic evaluation. Alternatively, if you want to improve the number of people that attend your events or programmes, you may want to look closer at your processes and carry out a process evaluation.

Data and evidence

Data and evidence are at the heart of effective evaluation. Having agreed the questions you want answered, the type of evaluation you plan to use, and your evaluation story, you will need to identify the data needed to support the evaluation.

Evaluation is distinct to programme monitoring. Monitoring involves collectiong data to make sure your programme is on track. This can be helpful in supporting evaluation exercisies and can provide some of the secondary data needed to measure change

There are two main types of data:

Secondary data: Secondary data is any pre-existing information you can use to help you answer questions and can include: administrative data, programme monitoring, attendance figures and registers, official statistics, programme reports, etc. These types of data can be helpful in any type of evaluation, but especially economic, outcome and process evaluations which often need to draw on existing information to evidence money saved, improved outputs and outcomes. The availability of suitable secondary data sources should be considered in the first instance to avoid unnecessary additional data work and cost.
Primary data: Primary data is the research and evidence gathered specifically for the current evaluation or study. It can be qualitative or quantitative and include: surveys; focus groups; interviews; and experimental and creative methods. Primary data supports all types of evaluations, especially when there is no pre-existing secondary data available to show change, impact and progress. In impact evaluation, where long and complex change needs to be conveyed, primary data can be an effective way of showing emergent and unplanned changed.

Agree evaluation arrangements

Once you have assessed all your evaluation needs, which include: your programme story and logic; your evaluation type; and your data and evidence needs, you may want to consider and plan your evaluation management and delivery arrangements. Thinking about all the skills needed to carry out an effective evaluation will help you determine the skills you have in-house and the ones you need to outsource and the resultant resource and time implications.

We suggest you think about:

Planning and management
Monitoring, and data collection and collation
Research skills (Qualitative and Quantitative)
Data analysis and insight (Qualitative and Quantitative)
Dissemination and reporting
Stakeholder and community engagement skills
Resources required, including finance, time and other provisions
Budget scope and limitations.

Capabilities framework

This framework offers some helpful information on the skills that support effective evaluation

Getting the most out of evaluation outcomes

Evaluation is effective when used to show how change and improvement occurs and to continuously strengthen service delivery. It should also be able to indicate whether a programme is ineffective or even counter-productive and the reasons for this and therefore whether it should be stopped or re-designed. For evaluation to be effective in these ways, you may want to think about the difference evaluation findings could make, your dissemination and communication plans, as well as opportunities to maximise lessons learnt. We suggest you think about:

Who needs to know about your evaluation findings and lessons and how you will effectively communicate and disseminate this information?
The ways in which your evaluation findings could make a difference to your organisation, and/or your service delivery:
- Would adjustments need to be made to current programme delivery?
- Would evaluation lessons inform future programmes and planning?
- How widely would you like these changes to be felt – at local delivery level; organisational level; across multiple programmes within one area, or regionally?
Who would need to be involved in deciding and making these changes – what stakeholders and staff, and how and when would you need to involve them?
What processes would you need to undergo to start planning and implementing changes? For example, would you go through action planning sessions with stakeholders; team idea generation sessions; “wash ups”, etc.?
Could these change processes be carried out in-house or would you need to commission them?
How and to whom would you communicate these changes?
Would you need to review and evaluate these changes? How would you do so and at what points?

Asking and addressing these questions could lead you to the beginning of another evaluation cycle.

Helpful resource:

Evidence for good - How charities use evidence to boost their influence and impact

This guide on the effective use of evidence demonstrates how evaluation and research can trigger meaningful change.

Visual frameworks or models

As set out earlier, there are several framework or models which can support your planning and evaluation. In terms of evaluation, they can help you:

Plan, map and identify the activities and inputs that lead to results, supporting process evaluations; and
Understand desired changes and who would be accountable for them, which supports outcome and impact evaluations.

Logic Model

Logic models clearly map out the different factors included in the life span of a programme and the order in which they need to be considered. Listing the various elements and the stages at which they need to be addressed allows you to understand what information needs to be measured in the evaluation process and when.
They are a good tool for supporting process evaluations, where linear activities and outputs are reviewed. They can also be applied to formative evaluation when mapping potential opportunities and barriers to a programme’s implementation. They are also useful in outcome evaluation, when assessing the outcomes achieved and the outputs and activities that supported them.

Theory of change

Theory of Change tells detailed and multi-dimensional story of change
It promotes a critical thinking approach to evaluation design
It encourages evaluators, staff and stakeholders to attribute success carefully by considering:
- external factors
- multiples interventions and assumptions in a success story. Doing so allows you to consider the complex policy context a programme is being delivered and means that realistic and net success can begin to be attributed to the programme being evaluated
Its detailed mapping of process allows it to support process evaluations, but it also lends itself very well to complex outcome and impact evaluations.

Helpful resources:

NHS guidance on logic models

Ecosystem Services for Poverty Alleviation (ESPA) guidance on Theory of Change

Evaluation types

There is no definitive list of evaluation models because they are always evolving, and it’s OK to design your own model (providing you know your evaluation type, and your evaluation story).

Formative evaluation

What they measure

These evaluations assess project feasibility and interim progress and take place before or during a programme’s delivery. They can take place before programme delivery to assess the likely impact it will have on a service and its users and its feasibility. They can also take place during programme delivery to provide timely lessons on what improvements need to be made and how to make them.

Approaches

These evaluations tend to draw on qualitative research methods. They typically ask whether a programme will work and explore the factors that will hinder or enable its success. If evaluation takes place during programme implementation, quantitative methods can also be helpful.

Example

The National Collaborative Outreach programme aims to increase the representation of disadvantaged students in Higher Education. The Higher Education Funding Council is commissioning a four-year formative evaluation of the programme to find out what is working well and why.

Process evaluation

What they measure

These evaluations assess how effectively a programme has been implemented. They track the progress made toward a programme’s planned activities, outputs and targets. This allows you to see where success does and does not occur and start to understand why. They can take place at the end of a programme, but more helpfully at certain points within a programme to learn lessons and improve delivery effectiveness. You will notice that this can cause an overlap between formative evaluations and process evaluations.

Approaches

These evaluations draw on all types of data, including primary and secondary, and you can also use mixed research methods to answer evaluation questions. They typically question the success of planned activities, including attendance figures, customer satisfaction, the number of activities delivered and their efficiency. They can also analyse effectiveness of resources (or inputs) used to implement the programme. Quantitative and qualitative methods support these evaluations, including surveys, interviews, case studies, focus groups, observations, empirical studies, and more.

Example

The Department for Work and Pensions commissioned a process evaluation of its Fit for Work Programme. The programme was aimed at employees on or at risk of long-term sickness absence and the management systems in place to avoid and decrease it. It explicitly questioned the effectiveness of those systems and processes.

Helpful resources

World Health Organization

The World Health Organization (WHO) guidance on process evaluations.

Outcome evaluations

What they measure

These evaluations are traditionally used in public funded programmes to assess whether and how much they have achieved the planned direct and immediate outcomes of a programme. Where process evaluations may tell you how many people attended a course, outcome evaluations can tell you how many people gained a qualification or obtained employment because of it. Outcome evaluations therefore tend to take place immediately after a programme has finished, or at the end of a delivery time frame.

Approaches

These evaluations benefit from making use of multiple methods and data to understand the results of a programme. For example, quantitative surveys can be used to capture how many people are now in jobs, as well as qualitative focus groups that can explore how people’s understanding of a topic has improved. Empirical and experimental studies that compare one groups’ progress to another group who did not receive an intervention can also be used. These options are not exhaustive.

Example

The National Offender Management Service had outcome evaluations carried out on two domestic violence interventions to measure their effectiveness in reducing incidents of reoffending.

Helpful resources

The Institute for the Study of Children, Families and Social Issues’ guidance on outcome evaluation is based on their evaluation of Sure Start programmes but can serve as a general introduction to programme outcome evaluation.

Measuring outcomes: guidance on outcome evaluation for Sure Start Local Programmes

You may also want to look at:

Evaluation of Families First

Evaluation of Flying start

Impact evaluations

What they measure

These evaluations assess the long-term and broad changes made by a programme. These changes can often not be observed or assessed until sometime after a programme has begun. Because of their long-term and often broad nature, it is not easy to attribute change to one particular programme or intervention and they may require sophisticated evaluation designs. It is advisable not to confuse them with the planned, immediate and short-term results of a programme, which should be assessed in outcome evaluations.

Approaches

These evaluations tend to ask high-level and open-ended questions that allow diverse and rich data and information to be collected. They require sophisticated designs that allow links to be made between an intervention and its impacts, sometimes years after the programme began. Theory based designs which require qualitative methods or empirical based which require quantitative evaluation designs are favoured.

Example

UNICEF commissioned an impact evaluation on Malawi’s National Plan of Action for orphans and other vulnerable children. The study aimed to understand the extent to which the National Plan made a difference in children’s lives.

Something to remember: A good impact evaluation understands that change may be attributable to multiple factors, not just one intervention.

Helpful resources

Introduction to impact evaluation

Bond - Impact evaluation - a guide for commissioners and managers

Economic evaluations

What they measure

These evaluations illustrate how the results of a programme justified the cost on time and money. They can also demonstrate long term value for money. There are different types:
Social return on investment (SROI): These evaluations help you to understand and quantify the social, economic and environmental value a programme is contributing.
Cost effectiveness analysis: In these evaluations, two or more programmes would be compared to identify the most cost-effective option. This would be achieved by analysing the cost of each programme against the value added by their relative outcomes.
Cost benefit analysis: These evaluations calculate the potential benefits of a programme, and compare them against the associated costs of running it.

Approaches

As these evaluations are based around an assessment, usually in monetary terms, of a programme and its costs, they require an in-depth financial analysis.

Interesting fact: Social return on investment also draws on logic models, such as Theory of Change to make sense of the economic impact of interventions. This means you may see a small level of qualitative techniques and questions within these evaluations.

Example

Berkshire Association of Clubs for young people requested a social return of investment evaluation that looked at the fiscal impact of interventions for their multiple stakeholder groups.

Additionality: Additionality considers the extent to which something happens as a result of an intervention or programme that would not have occurred in its absence. This guide gives a very comprehensive overview of key concepts in economic evaluation:

Additionality Guide

The UK Government’s Homes and Communities Department provides comprehensive guidance on how to assess additionality

Further guidance and support

We have deliberately tried to keep this guide short and succinct.

We hope it provides sufficient detail to give a clear understanding of what evaluation is and to get you started in undertaking an evaluation. We hope it might also stimulate you to want to know more. You may want further guidance on the detail or wish to read around the theory and practice.

In this part of the guide we have included links to other support material which you may find helpful.

We hope to further develop the guide over time with input from users.

The Magenta Book – guidance for evaluation

The Magenta Book produced by HM Treasury, is well recognised guidance on what to think about when designing and undertaking evaluation. It explains how results can be interpreted and presented, and what should be considered in this process. While it is mostly used by central government, it is also useful for policy-makers in local government, charities and the voluntary sectors.

Evaluation - Practical Guidelines

This is a guide for researchers seeking to engage general audiences with their subject to evaluate public engagement activities.

Center for Theory of Change

The Center for Theory of Change aims to support understanding of the Theory of Change approach and what is needed to support its successful implementation. Its website contains guidance on the model and its use.

Evidence for good - How charities use evidence to boost their influence and impact

This guide on the effective use of evidence demonstrates how evaluation and research can trigger meaningful change.

Glossary of terms

Attribution: In evaluation, attribution refers to the actors and interventions responsible for progress and change.

Co-production: This is a way of working that actively involves service users and citizens in the design, planning, delivery and improvement of public services. See the Co-production Network for Wales.

Counter-factual: The counter-factual asks “what would have still happened” or “what would have happened anyway” to groups, communities or individuals if the programme, policy or intervention being evaluated was not in place. It supports a Theory of Change approach to evaluation.

Empirical studies/evaluations: These studies use scientific methods and experiments to understand the impact of an intervention or programme, including Randomised Control Testing and Quasi-Experiments.

Mixed methods research: This typically involves using research methods that allow qualitative and quantitative data to be collected.

Primary data: Primary data is the research and evidence gathered specifically for the current evaluation or study. It can be qualitative or quantitative and include: surveys; focus groups; interviews; and experimental and creative methods - there are many options. These types of data support all evaluation types. Primary data supports all types of evaluations, especially when there is no pre-existing (secondary) data available to show change, impact and progress. In Impact evaluation, where long and complex change needs to be conveyed, primary data can be an effective way of showing emergent and unplanned change.

Qualitative data: Qualitative data describes rather than measures. Typically, it describes the quality of something, or individuals’ and groups’ experience or perception of something. Words, imagery, and creative means are typically used to convey findings.

Quantitative data: Quantitative data is any data that can be measured numerically. It can measure people (for example, the number of people in one area), things, characteristics (e.g. the number of people claiming Job Seekers Allowance, the temperature of something), and experience (the number of people who rated a service highly). Numbers are used to convey findings.

Quasi-Experiments: Quasi-Experiments are a proxy way of determining the counter-factual, which is “what would have happened anyway” without the existence of the programme or intervention under evaluation. Often, administrative and existing data is drawn on to see the difference a programme has made in a population or group where the programme has been trialled in comparison to a population or group without the intervention. Where it is not possible to carry out Randomised Control Trials, Quasi-Experiments are often used.

Randomised Control Testing: Randomised Control Trials are an empirical and experimental approach to testing the effectiveness or impact of an intervention or programme. People with similar characteristics are randomly assigned to an intervention or non-intervention group. Both groups are assessed to determine the impact of a programme or intervention on the intervention group in comparison to the non-intervention group. This is a widely accepted “gold standard” way of establishing the counter-factual, which conveys what would have happened without an intervention or programme in place.

Secondary data: Secondary data is any pre-existing information you can use to help you answer questions and can include: administrative data, programme monitoring, attendance figures and registers, official statistics, programme reports, etc. These types of data can be helpful in any type of evaluation, but especially economic, outcome and process evaluations which often need to draw on existing information to evidence money saved, improved outputs and outcomes.

Theory based studies/evaluations: These designs are used to qualitatively assess the ways in which a programme or intervention has influenced outcomes and impacts. It assesses the extent to which a programme has influenced results and looks at the reasons for this, usually referring to a Theory of Change.

Our guide to EVALUATION

So, what is evaluation and when is it used?

Evaluation is an objective process of understanding how a policy or other intervention was implemented, what effects it had, for whom, how and why.

In summary, evaluation is a process carried out by organisations to understand:

Evaluation Process

Here are the steps we suggest you might consider when planning and undertaking evaluation:

Visual frameworks or models to support evaluation

Evaluation types

Data and evidence

Data and evidence are at the heart of effective evaluation. Having agreed the questions you want answered, the type of evaluation you plan to use, and your evaluation story, you will need to identify the data needed to support the evaluation.

Agree evaluation arrangements

Getting the most out of evaluation outcomes

Visual frameworks or models

As set out earlier, there are several framework or models which can support your planning and evaluation. In terms of evaluation, they can help you:

Logic Model

Theory of change

Evaluation types

There is no definitive list of evaluation models because they are always evolving, and it’s OK to design your own model (providing you know your evaluation type, and your evaluation story).

Formative evaluation

What they measure

Approaches

Example

Process evaluation

What they measure

Approaches

Example

Helpful resources

Outcome evaluations

What they measure

Approaches

Example

Helpful resources

Impact evaluations

What they measure

Approaches

Example

Helpful resources

Economic evaluations

What they measure

Approaches

Example

Further guidance and support

We have deliberately tried to keep this guide short and succinct.

Glossary of terms