For many of us, this is a time for reflecting back on the year and exchanging gifts. So for this final post in the 52 weeks series for 2013, we wanted to share some evaluation gifts you can keep for yourself or share with your friends and colleagues.
Thank you to everyone who has contributed to BetterEvaluation in 2013 - page authors, stewards, guest bloggers, funders, advocates, partners, Facebook friends, site redesign panel members, members who have added comments, feedback and suggestions for new resources, and the core team who work across the world with ILAC, ODI, Pact and RMIT University. And a special thank to Skyla Kyeema Woodhill (6 years old), for the rainbow Christmas treee picture. Skyla was a baby when the idea of BetterEvaluation was first discussed at CIAT in Cali, Colombia.
2014 is shaping up to be an exciting year. We will be sponsoring the methodological innovation strand in the African Evaluation Association conference, in Cameroon in March. We will continue our 52 weeks of BetterEvaluation series, featuring guest blogs from Michael Quinn Patton on trends in qualitative evaluation and others. And we will have a particular focus on supporting program managers and others who commission and manage external evaluations.
Until then, enjoy these free gifts from the BetterEvaluation team.
Planning an evaluation
Making Evaluations Matter
Develop evaluation capacity
12 Components Monitoring & Evaluation System Assessment, UNAIDS
A wide range of stakeholders are brought together to identify strengths and challenges in national M&E systems and plan collaboratively for system-strengthening.
This document includes the M&E capacity framework (12 components) and practical guidance for preparing and conducting the assessment and following up afterwards.
PDF version of the assessment tool.
Excel workbook version of the assessment tool including dashboards for displaying the assessment results and the actions planned.
I am recommending these guidance documents and tools as they use a comprehensive and collaborative approach for the assessment but also support planning for action. It was originally developed for HIV/AIDS-related national M&E systems and has been adapted and widely used in a range of other sectors.
Theories of change
The behaviour change wheel, created by Susan Michie, Maartje M van Stralen and Robert West is a new conceptual framework for characterising and designing behaviour change interventions. Based on a systematic review of research literature on behaviour change interventions, it proposes three main sources of behaviour (motivation, capability and opportunity), different intervention functions (education, persusasion, incentivisation, coercion, training, enablement, modelling, environmental restructuring, and restrictions) and different policy categories (service provision, legislation, communication/marketing, environmental/social planning, guidelines, fiscal measures and regulation).
I find the three-part overview (motivation, capacity and opportunity) a useful mental model to use whenever someone is talking about theories of change relating to behaviour change. Very often their theory of change only looks at one or two of these. For example, a program to improve organisational performance might look at strategies to improve staff capacity (eg training to improve skills) but not look at their opportunity to use these skills (eg organisational procedures which are incompatible) .
This paper explores the use of a 'web' diagnostic and operational framework for the institutionalisation of a gender perspective in development policy, planning and practice.
All of my work involves trying to change an organision or the behaviour of its staff or a collaboration. These changes are often embedding new ways of working, a new norm or concept. For example, making gender a genuinely cross-cutting consideration or making forest management more participatory or embedding Theory of Change thinking or power analysis in an organisation. So how we think that such change processes will happen is pretty important for our choice of strategy and how we then assess success. The default responses by organisations resort to are 'let's do some training' or 'let's write up and share some good examples'. Clearly, this is not enough. I have found Caren Levy's paper on 'The Web of Institutionalisation' (1996) (http://discovery.ucl.ac.uk/34/1/wp74.pdf) very useful to think more broadly than these default options. I recently used it with three organisations keen to pursue more theory of change-based research planning. After identifying fears and aspirations related to this endeavour, participants identified ideas for action. Mapping them onto Levy's 'web', showed where the comfort zone was and where they needed to consider additional activities. It also gave them an initial 'road map' against which to assess their process of embedding ToC-thinking into the three organisations.
How to climb the R learning curve without falling off the cliff
R is a free statistical software package - it is a program language and is very powerful in terms of data visualization and seamless integration into reports and documents. For many people it's a better alternative than SPSS or SAS. These resources can help you learn:
Recommendation from Joris Vandelotte, Pact:
I decided to take the a Coursera course on programming in R to learn the program. But I was really disappointed - I couldn't even follow the lecturer. I felt stupid.
A few weeks later, I am at the AEA conference where Tony Fujs, Will Fenn, and Ann K. Emery are presenting a panel called "How to climb the R learning curve without falling off the cliff –" -
I attend and it turns out that one of the presenters also took the Coursera course and failed to follow as well - I felt much better about my intelligence ... That particular course wasn't the right one to start off with.
The session then went on and explained in detailed how to get going with R - providing guidance on installation and learning - and they had this really cool one-pager with a lot of resources.