Consider important elements of what is being evaluated

Available languages

What is being evaluated makes a difference to how it should be evaluated. It is helpful to identify particular aspects of what is being evaluated and check that these have been addressed in the evaluation design.

1. Check the stage of development of the project or program

Firstly, check the implications of the stage of development of the project or program that is being evaluated. Is it still being planned? Is it part–way through implementation? Or is it near the end – or has it in fact already ended?

Stage of development

Consequence

Possible implication for the evaluation design

Not yet started

Can set up data collection from the beginning of implementation

Possible to gather baseline data as a point of comparison and also to establish comparison groups or control groups from the beginning

Opportunity to build some data collection into administrative systems to reduce costs and increase coverage

Period of data collection will be long

Need to develop robust data collection systems including quality control and storage

Part way through implementation

Cannot get baseline data unless this has already been set up

Will need to construct retrospective baseline data to estimate changes that have occurred

Might be able to identify “bright spots” where there seems to be more success and those with less success

Scope to do purposeful sampling and learn from particular successes and also cases which have failed to make much progress

Almost completed

Cannot get baseline data unless this has already been set up

Will need to construct retrospective baseline data to estimate changes that have occurred

Depending on timeframes, some outcomes and impacts might already be evident

Opportunity to gather evidence of outcomes and impacts

Completed

Cannot get baseline data unless this has already been set up

Will need to construct retrospective baseline data to estimate changes that have occurred

Depending on timeframes, some outcomes and impacts might already be evident

Opportunity to gather evidence of outcomes and impacts

Cannot directly observe implementation

Will need to depend on existing data or retrospective recollections about implementation.

 

2. Is it complex or complicated?

Secondly, consider whether there are important aspects that are either complicated (with many components) or complex (emergent) that should be addressed in the evaluation design.

(i) Focus

Does everyone share the same objectives?

Homogeneity of objectives

Implications

Everyone shares a single set of objectives

Impacts to be included can be readily identified from the beginning.

There are different objectives valued by different stakeholders (competing objectives, different objectives at different levels)

Need to identify and gather evidence about multiple possible changes

Need an agreed way to weight or synthesise results across different domains to produce a judgement of overall performance.

The stated objectives are changing (often in response to changing needs or opportunities)

Need nimble impact evaluation systems that can gather adequate evidence of emergent intermediate outcomes or impacts

 

(ii) Management

Who has responsibility for management and decision making?

Who is responsible

Implications

Single organisation

Primary intended users and uses easy to identify and address in the development of Key Evaluation Questions and endorsement of the design

Multiple organisations (which can be identified) with specific, formalized responsibilities

Likely to need to negotiate access to data and ways to link and co-ordinate data

Might need to negotiate parameters of a joint impact evaluation, including negotiating scope and focus.

Changing list of organizations working together in flexible ways

Need nimble impact evaluation systems that can gather evidence about the contributions of emergent actors and respond to the different ways they value intended and unintended impacts

 

(iii) Consistency

How much variability is there in how activities are implemented?

Level of variability

Implications

Standardized – one-size-fits-all program

Quality of implementation should be investigated in terms of compliance with ‘best practice’.

Adapted – variations of a programme planned in advance and matched to pre-identified contextual factors.

Quality of implementation should be investigated in terms of compliance with the practices prescribed for that type of situation.

Adaptive – evolving and personalised program that responds to specific and changing needs.

Quality of implementation should be investigated in terms of how responsive and adaptive service delivery was.

 

(iv) Necessity

How many different options are there for solving the problem or producing the intended impacts? To what extent is this exact initiative needed to solve the problem?

Number of possible interventions

Implications

There is only one way to achieve the intended impacts.

Counterfactual reasoning appropriate.

The intervention is one of several ways of achieving the impacts, and the options can be identified.

Counterfactual reasoning not appropriate as it does not accept a causal relationship between the intervention and the impacts unless they would not have occurred in the absence of the intervention.

Possibly one of several ways of achieving the intended impacts (uncertain).

Counterfactual reasoning not appropriate as it does not accept a causal relationship between the intervention and the impacts unless they would not have occurred in the absence of the intervention.

 

(v) Sufficiency

To what extent will the problem be solved by the intervention alone?

Generalisability of the intervention

Implications

The intervention is enough to produce the intended impacts. Works the same for everyone.

Counterfactual reasoning appropriate

Reasonable to ask ‘Does it work?’

Works only in specific contexts which can be identified (eg implementation environments, participant characteristics, support from other interventions).

Impact evaluation question needs to be ‘For whom, in what circumstances and how does it work?’

Counterfactual reasoning only appropriate if the causal package of supportive context and other activities can be identified and included.

Works only in specific contexts which are not understood and/or not stable.

Impact evaluation question needs to be ‘For whom, in what circumstances and how does it work?’

Counterfactual reasoning not appropriate as the causal package of supportive context and other activities is changing and/or poorly understood and cannot be adequately identified.

 

Change trajectory

How are the impact variables expected to change over time? For example, straight line of increase, or J curve? To what extent are the relationships between variables understandable and predictable?

Relationship between variables

Implications

Simple relationship (cause and effect). Predictable.

Measurement of change can be done at a convenient time and confidently extrapolated

Complicated relationship that needs expertise to understand and predict.

Timing of the measurement of changes should be undertaken when it will be most meaningful – expert advice will be needed.

Emergent factors and multiple causes, sudden changes (tipping points) that are unpredictable. Can only be understood in retrospect.

Changes will need to be measured at multiple times as the change trajectory cannot be predicted.

 

Unintended impacts

To what extent are unintended impacts predictable?

Predictability of unintended impacts

Implications

Easily predictable and therefore can be readily included in the data collection plans

Need to draw on previous research and common sense to identify potential unintended impacts and gather data about them

Need expertise to predict and address.

Need advice from experts about potential unintended impacts and how these might be identified.

Unpredictable - only identified and addressed when they occur.

Need to include a wide net of data collection that will catch evidence of unexpected and unanticipated unintended impacts.

Source: Resource Hub for Evaluating C4D 2016 - adapted from Funnell and Rogers (2011), pp.90-91, Rogers 2016.

 

3. Identify issues to be addressed

Are any of the following issues present? They will need to be addressed in the design.

Issue

Possible implications for the evaluation design

Long time until impacts will be evident

Might need to gather data about intermediate outcomes (that will be evident during the timeframe of the evaluation) and use other research and evaluation evidence to predict the likely achievement of impacts

Difficulty observing implementation activities (eg conflict affected or remote areas)

Might need to gather data through remote sensing, key informants, big data or crowdsourcing

Difficulty observing results (outcomes, impacts) (eg sensitive issues, private behaviour)

Might need to gather data through key informant interviews, or unobtrusive measures (for example looking at patterns of wear from foot traffic) or techniques for gathering sensitive data (for example polling booth)

'Consider important elements of what is being evaluated' is referenced in: