Issues raised by participants in InterAction Forum
As part of the 'Impact Evaluation in Action' session at last week's InterAction 2011 Forum last week, participants were invited to consider a recent impact evaluation they had been involved in, and to identify one thing that worked well, and one remaining challenge.
Thank you to everyone who contributed to this exercise, generating rich information about the particularly important issues in impact evaluation.
The 150 items produced in this 'Sharpies and Sticky Notes' exercise are shown below, sorted into the 7 different components of evaluation that are used to structure the methods in the BetterEvaluation site - Manage & Engage, Define, Frame, Describe, Understand Causes, Synthesize, and Report & Support Use
One of the most striking aspects of the items is the strong emphasis on issues related to evaluation management - in particular, working with partner organizations to leverage the different resources needed for effective impact evaluation, including technical expertise and local credibility to open access to field sites.
The list of items will be helpful in informing the Impact Evaluation Guidance Notes, currently being developed by InterAction, and in further developing material for the BetterEvaluation site. They demonstrate the value of initiatives such as these which seek to share learnings across organizations about effective strategies for evaluation.
MANAGE
Effective management of an evaluation, ongoing evaluative activity, or the evaluation function within an organization is essential. It includes the following tasks:
- Ensure appropriate governance of the evaluation
- Identify and engage intended users and other important stakeholders
- Ensure the appropriateness of those conducting the evaluator
- Determine and secure existing and additional resources
- Develop Terms of Reference
- Ensure an ethical evaluation
- Ensure a quality evaluation
Worked well:
- Create shared value/goals/expectations in impact evaluation
- Cross institutional teams: * pull from multiple levels of expertise, * refinement of approach, * broader institutional buy-in because of cross institutional team not HQ driven
- Enthusiasm, interest and recognition that impact evaluation is important! Small group dedicated to regularly working on impact assessment
- The 3 key stakeholders – 1) program leadership, 2) evaluation team and 3) funders – spent sufficient time up front to allow full discussion and agreement on evaluation questions, understanding what would be learned and would not, and trade offs/open issues that would need to be revised as the evaluation played out.
- Convening – groups have been willing to come together and discuss issues and agree that there is a need. Structure for discussions
- Cross institutional teams: * pull from multiple levels of expertise, * refinement of approach, * broader institutional buy-in because of cross institutional team not HQ driven
- Developed an online tool tailored to our organizations’ way of doing business – contract software/GUI development, - solicit input/feedback from staff for ongoing improvement of the system
- Enthusiasm, interest and recognition that impact evaluation is important! Small group dedicated to regularly working on impact assessment
- Finding partner with common vision
- Good indicators – well thought out, lots of input (Also reported in DESCRIBE)
- (User of external evaluation) Useful outsider perspective help change some ideas
- Contract for independent external evaluation with academic professionals
- Found a data collection expert who could adapt to field realities – he did field testing to verify which methodologies could work
- My agency has recognized that conducting sound, appropriate IE is a different skill set than managing good humanitarian/development projects so we've hired a team of specialists/academics-social scientists familiar/skilled in IE to design and work with our country teams. They’re rolling out training, back stopping etc. to ensure solid IEs.
- Recruiting a sharp Fullbright scholar to do the field work and working with them to develop the methodological tools required
- Good involvement of program participants in project and evaluation design, implementation, analysis, reporting and use.
- Involving a variety of stakeholders in the design, evaluation and dissemination for increased ownership of results
- Involving multiple stakeholders in evaluation: domestic universities, school and govt actors on the ground. This enables efficient evaluations.
- Multigroup collaboration on primary data collection tool
- Participatory/stakeholder involvement in evaluation (all stages)
- Partnership and relationship building between field and HQ
- Team were experts in subject matter and country
- Working in collaboration with other organizations and the community to decide on the real goal of intended impact has made the evaluation process more poignant. Meetings, focus groups, think tanks/brainstorming (Also classified under DEFINE)
- Extreme attention to logistical planning to ensure enough time (and $) for the necessary process to be done well and communicated to stakeholders
- Training new in-country research assistants and enumerators (merging their local expertise with data collection techniques)
- Working in conjunction with partners to reach broader audience and gather more and better data – leveraging scarce resources
- Collaborating with other organizations who have better knowledge on how to do impact study and evaluations
- Putting in place the structure to actually start doing M & E – budget and staff
- Funding for impact evaluation
- Clear evaluation Terms of Reference with qualified but diverse evaluators carrying out the evaluation
- Truly randomized assignment of communities between treatment and control arms with a sufficient number in each group largely due to good partnerships with a research University. (Also classified as part of UNDERSTANDING CAUSES)
Challenges:
32. (User of external evaluator) – Evaluator did not come on site, worked remotely (interviews and paper documents)
33. Need adequate budget, staff person hours and time to undertake impact evaluation well.
34. (User of external evaluator) – Evaluator did not come on site, worked remotely (interviews and paper documents)
35. Capacity building and standardization in a global network of autonomous organizations
36. Capacity for data collection to ensure quality and accuracy of information (Also categorized under DESCRIBE)
37. Common understanding between evaluators and policymakers/program managers
38. Cost is the number one challenge
39. Getting to “Yes’ – difficult to get agreement around specifics, - i.e. measurements for success. (Also categorized under DESCRIBE)
40. Lack of appreciation on why the data is needed on the part of practitioners on the ground
41. Household surveys to measure impact indicators are resource intensive (time and money)
42. In collaborative projects (multi-agency) getting a common understanding of evaluation
43. Insufficient resources to conduct comprehensive/long term evaluation
44. Resource constraints
45. Resource limitation of implementing partners (particularly staff, time and communication)
46. Resources, especially for post-program/project evaluation
47. Scarce resources (human and financial) to collect data
48. Separating impact evaluation from strategic planning
49. Cost involved to carry out rigorous methods
50. Inadequate capacity to conduct evaluations
51. And resource constraints of course
52. Evaluation capacity of field practitioners (staff, local partners)
53. Finding resources be it cash or personnel to carry out the impact study of various programs
54. Funding/finding partnerships for impact evaluation
55. Managing the dynamics/relationships between NGOs (practitioners) and universities (academics)
56. Measuring impact when a program contract has closed (Also categorized under DESCRIBE)
57. Not enough time/funding to measure impact. (Also categorized under DESCRIBE)
58. To train the staff, the field staff to open their minds to embrace the new documentation required, collect the information in time
DEFINE
It is important to develop an agreed definition of what is to be evaluated. This can require ongoing review and revision it includes the following tasks:
- Develop a brief description of what is to be evaluated, including its objectives and intended impacts and rationale
- Identify potential unintended results (positive and negative)
- Develop a theory of change/logic model/program theory of how the program activities are understood to produce or contribute to the intended impacts
Worked well:
59. Building in impact indicators (not just outcome indicators) into the program design. This would also include unintended, but feared, negative indicators/results/consequences
60. Improve program logics
61. Use of theory of change
62. Good evaluation results - Long-term path of intervention. How commitment/sustainability was built with various actors over time
63. Good evaluation results - captured the relationship building that led to replication and intersectoral multifaceted commitment to project ends. (Also classified as part of UNDERSTANDING CAUSES)
64. Working in collaboration with other organizations and the community to decide on the real goal of intended impact has made the evaluation process more poignant. Meetings, focus groups, think tanks/brainstorming (Also classified under MANAGE)
Challenges:
65. Tried to evaluate impacts of a program without established or clear rules on who receives the benefit/intervention
66. Whether and how to measure the impact of the network or the individual practitioners
FRAME
Before designing the evaluation, it is necessary to decide on its purpose, the key evaluation questions it is intended to answer, and to clarify the values that will be used in the evaluation. Tasks include:
- Determine purpose
- Develop key evaluation questions
- Clarify values
Worked well:
67. Use of reflecting on Peace Practice Methodology to determine strategic and logical project/programme progression – 2 x 2 matrix and criteria for effectiveness
68. Worked well – Establish credibility with the communities we work in
Challenges:
69. Finding a balance between what the results actually turned out to be and what has been promised to the donor, and communicating a truthful and transparent set of outcomes that will satisfy the donor. Issues to consider include: (1) original indicators of success could have been tailored to donor needs, (2) donors are not as interested in “learning” anymore, but are heavily focused on “impact”, (3) donors expect to see impact in too-short a period of time – not feasible. (Also classified under REPORT & SUPPORT USE)
DESCRIBE
The design of an evaluation sets out how data will be gathered/retrieved and analysed in order to describe the activities undertaken by an intervention, its outputs, outcomes and impacts, and important contextual factors, including activities by other organizations that contribute to impacts. Tasks include:
- Sample
- Collect/Retrieve data
- Use specific measures and indicators
- Combine quantitative and qualitative data
- Look for patterns
Worked well:
70. Combining qualitative with quantitative
71. Full participation by the beneficiaries to provide requested information through formal interviews, surveys in written form
72. Good indicators – well thought out, lots of input (Also reported in MANAGE)
73. Use various qualitative, quantitative and tech evaluation to collect data and monitor impact
74. A qualitative study capturing extensive life histories. It was a way of capturing info the field workers knew, confirming programmatic changes on the ground, and giving good tool/data for analysis
75. By working with women’s advocacy groups, we have been able to increase participation of women in our programming and measure how many women participate
76. Case study sampling to describe changes to lives of children and families
77. Collecting and graphing epidemiological data about what were the human health outcomes
78. Considering evaluation designs that use existing administrative data
79. Focus groups
80. Getting the patient info as they come to the clinic the first time
81. Good baseline data on disease surveillance infections disease transmission (Evaluation of the impact of the Disease Surveillance Initiative in the lives of people by the Mekong Basin, Asia)
82. Having hub facilitators in each region worked well because they maintained contact with beneficiary communities that had completed projects early in the program and were able to get back in touch with them to arrange focus groups for the final evaluation
83. Involving the field in the formulation of indicators
84. Keeping track of numbers – well organized on a spreadsheet, quarterly/regular reports from field sent to information manager/coordinator who keeps them well ordered.
85. Looking at similar measures used by other organizations to build off their experience
86. Met with and go governmental level permission to access/enter orphanages to do sampling and testing
87. Our country teams work hard to contextualize questions and support valid translations in their local environments
88. Working with trusted local partners to collect data in a local community rather than having a complete outsider try to collect and interpret data
89. Client’s Centre 1. Set objectives 2. Achieve objectives – specific indicators to measure it. -> short and long term impact
Challenges:
90. Developing metrics/indicators to measure real gender empowerment and not just capacity strengthening
91. Communicating M & E tools to staff and familiarizing them with the terminology and helping them conceptualize their program with M & E in mind – e.g. identify outcomes, progress markers, indicators etc
92. Designing and maintaining quality, ongoing monitoring systems that feed into the evaluation
93. A measurement statistic for determining a realistic rise in living standards - i.e. does increase in income and number of people working in a region translate to higher living standards?
94. Capacity for data collection to ensure quality and accuracy of information (Also categorized under MANAGE)
95. Consistency in data collection and observation on social impacts
96. Determine sample size.
97. Disconnect between the timeframes for the outcomes of interest and the timeframe for the work being conducted/the program timeframe
98. Distinguishing between “outputs” and “outcomes” (impact)
99. Getting to “Yes’ – difficult to get agreement around specifics, - i.e. measurements for success (Also categorized under MANAGE)
100. Lack of consistency in data collection
101. Measuring global advocacy campaigns
102. Measuring capacity building since it’s hard to quantify and it’s never “done”.
103. Collect baseline data that methodologically is geographically ethnically etc representative before beginning our interpretation (in this case on nutritional status of 1,000,000 children in Chinese orphanages)
104. Aligning collaborative partners around common tools and methodology (especially across multiple cultures)
105. Baseline data (challenge that there wasn’t any or if it was available, wasn’t useful).
106. Changing indicators – early ones selected at the beginning of project don’t /are less relevant later on – and new ones emerge as necessary to track
107. Choosing appropriate incentives during surveys to compensate people for their time responding to questions.
108. Creating indicators in areas that are new to use (such as gender)
109. Difficult to measure ‘ripple effect’
110. Difficult to measure increased participation of persons with disabilities in our programming.
111. Difficulty with baseline data.
112. Field logistics
113. Follow up training evaluations 30-60-90 days or a year afterwards. Difficulty accessing trainees afterward.
114. Following up with the patient after the initial visit – phone numbers changes, addresses moved, etc
115. Indicators
116. Lack of access to statistics from sponsoring agencies that might serve as indicators of principles being applied.
117. Lack of baseline data (data was flawed).
118. Lack of relevant baseline data
119. Long –term process – difficult to capture in interview methodology because it went beyond the timeframe of the defined project on both sides
120. Poor language translation – did not always ask the exact questions we wanted
121. Quality and validity of data collected
122. Tendency to emphasize outputs (number of people trained) over outcomes (seizure)
123. Developing a simple but effective system to capture outcomes data for a number of years
124. How to decide what key performance evaluation indicators to report on (How to measure impact without standards)
125. Measuring impact when a program contract has closed (Also categorized under MANAGE)
126. Not enough time/funding to measure impact. (Also categorized under MANAGE)
127. On a long program that has gone through several CoPs and program changes, it can be difficult to collect standardized baseline data in all communities.
128. Scope – challenges in data collection
UNDERSTANDING CAUSES
One of the features which distinguishes impact evaluation from other types of evaluation is its focus on understanding the cause-and-effect relationship between the intervention and the outcomes and impacts. Tasks include:
- Check results match the program theory/logic model
- Compare results to counterfactual
- Investigate alternative explanations
- Investigate complementary explanations
Worked well:
129. Even with little information there was evidence of impact
130. Good evaluation results - captured the relationship building that led to replication and intersectoral multifaceted commitment to project ends. (Also classified as part of DEFINE)
131. Truly randomized assignment of communities between treatment and control arms with a sufficient number in each group largely due to good partnerships with a research University. (Also classified as part of MANAGE)
132. Use of control group vs. experimental group to determine impact on beneficiaries
133. Working with a comparison group instead of a true control group if it’s not possible get a true control group. Adds more rigor than no comparison at all.
Challenges:
134. Adapting message on “rigor” and RCTs to our work – which measures change in whole communities. So we had an entire community to be an intervention or a control and somehow we have to finesse the message to communities that are randomly assigned to be controls that “you just have to wait 3 years” until their turn to get the intervention.
135. Capturing and reporting on qualitative data in a meaningful way that helps to show impacts
136. Designing Impact Evaluations that do not compromise the integrity/purpose of our programming. Balancing the “research” with the delivery of work in the field.
137. Discerning attribution, particularly how to estimate the share of attribution across different partner agencies. How do we give credit for instance to the donor? What share of attribution do they get credit for?
138. Dynamic of success and collective commitment made it difficult for any one organization to clearly claim credit for results against budget investments
139. It was a qualitative evaluation with a relatively small sample size, so attribution was a concern.
140. Presenting rigorous quantitative story about impact
SYNTHESIZE
There are three different types of synthesis that might be needed:
- Synthesis of evidence about a single intervention
- Synthesis of evidence of different components of an intervention (e.g. synthesis of projects in a larger program)
- Synthesis of evidence from different evaluations
Worked well:
141. Compared several interventions and their comparative impact to learn which factors have the strongest influence on outcomes. Found that some of our interventions were not building blocks of change or as influential as we thought.
Challenges:
142. Professional/expert judgment outweighs data – bias of SME technical evaluators
143. Translating numbers and data into real impact statements about quality,
REPORT & SUPPORT USE
In addition to managing and framing strategies to support use (such as identifying and engaging intended users and clearly articulating purposes), there are specific tasks involved at the reporting stage of an evaluation and afterwards:
- Choose relevant report format
- Ensure report is accessible
- Support use
Worked well:
144. Communicating success based on success case stories
145. Evidence for policy
146. Identify particular ways to implement the intervention and achieve effect – Bangladesh Stepmothers
147. Using evaluation results with various audiences
Challenges:
148. Deliver results with high staff turnover rates
149. Culture of evaluation (cut of funds, lack of will to be accountable for one’s activities, to show there are mistakes)
150. Finding a balance between what the results actually turned out to be and what has been promised to the donor, and communicating a truthful and transparent set of outcomes that will satisfy the donor. Issues to consider include: (1) original indicators of success could have been tailored to donor needs, (2) donors are not as interested in “learning” anymore, but are heavily focused on “impact”, (3) donors expect to see impact in too-short a period of time – not feasible. (Also classified under FRAME)
151. Getting the “last mile” – full disseminations, verification by participants in the process done well and fully – i.e. according to HAP principles/
152. Knowledge transfer is time consuming
153. Presenting data collected from an impact evaluation in a way that is most useful to implementing partners so that they can use data/analysis not just to look back at past projects but also to plan for future/current projects
Are there other important areas of an impact evaluation that have worked well - or had ongoing challenges?