In today's world, we are facing a series of interconnected global challenges—sometimes referred to as a ‘polycrisis’. For young people (and many not-so-young people), it often feels like issues such as climate change, social injustice, and inequality have always been a reality.
Alice Macfarlan
Alice Macfarlan is the Knowledge Platform Manager of BetterEvaluation, the knowledge platform of the Global Evaluation Initiative (GEI). She works with the global M&E community to curate and co-create practical resources that help strengthen evaluation systems and practices. Alice is dedicated to helping people make informed decisions about which evaluation methods are appropriate for different contexts.
Alice is passionate about supporting Young and Emerging Evaluators (YEEs) and is a strong advocate for embedding environmental and equity concerns into evaluations. She’s a member of the Footprint Evaluation Initiative, which focuses on embedding sustainability-inclusive evaluation into the field.
With a background in linguistics and knowledge management, Alice has a special interest in translating complex information and concepts into clear and useful knowledge products, helping ensure knowledge about how to do M&E well is accessible to practitioners worldwide.
Contributed by this member
Blog
- A recent essay competition for young and emerging evaluators (YEEs), jointly hosted by the Independent Evaluation Group (IEG), the Global Evaluation Initiative (GEI), and EvalYouth, invited emerging evaluators to propose solutions for closing the gap between global and local knowledge.
- Understanding the 'place' - be it the site of a project, program, strategy or policy - is critical when addressing environmental sustainability as part of an evaluation.
- How can donors and grantees work together to create effective monitoring, evaluation, and learning (MEL) practices that drive field-wide transformation?
- Happy Holidays! As we head into 2022 we thought we'd share a list of resources for you to peruse in the new year.
- We’re currently going through a global period of rapid change and adaption, due in large part to the effects of the COVID-19 pandemic on our lives and work.
- While we’re happy to wish the year 2020 farewell, many of the challenges and difficulties that arose over the past 12 months are still with us, as is the sadness over the many different forms of loss we’ve all experienced.
- We’re continuing our series, sharing ideas and resources on ways of ensuring that evaluation adequately responds to the new challenges during the pandemic.
- Given the numerous interconnected environmental crises the world faces, there is an urgent need to include consideration of environmental impacts into all evaluations.
- Evaluation needs to respond to the changes brought about by the Covid-19 pandemic. As well as direct implications for the logistics of collecting data and managing evaluation processes, the pandemic has led to rapid changes
- We’re excited to be involved in the 2020 IPDET Hackathon – a week-long event in which hundreds of people from around the world bring together their skills,
- The Covid-19 pandemic has led to rapid changes in the activities and goals of many organisations, whether these relate to addressing direct health impacts, the consequential economic and social impacts or to the need to change the way thing
- Organisations around the world are quickly having to adapt their programme and project activities to respond to the COVID-19 pandemic and its consequences. We’re starting a new blog series to help support these efforts.
- Meetings and gatherings are vital components of evaluation. Often these are done face-to-face, however sometimes necessity or practicality makes meeting online the best option.
- Often referred to as 'visual note-taking', graphic recording is a method that merges data collection and reporting to create a visual record of a discussion.
- Strengthening evaluation capacity of individuals, organisations and systems continues to be a major focus for BetterEvaluation so in this blog we present 7 ways that might be useful when planning how to do evaluation better in 2020.
- In this first blog of 2019, Patricia Rogers, Greet Peersman and Alice Macfarlan examine how New Year's resolutions are similar to many evaluation practices.
- Evaluation reporting is important.
- In this blog, I wanted to share three examples of communication plan templates that address this and allow for more detail and thinking through of the communication and dissemination process.
- If you’re like me, you think you’ve got a pretty good handle on data visualisation – you know how to make basic customisations to graphs in Excel, you know you should probably think carefully about whether or not to put that large tabl
- We've now completed the first component of our user research - the user survey.
- The material from BetterEvaluation comes from a combination of curating existing material and co-creating new material. This blog is part of an ongoing series about material that we have co-created with BetterEvaluation users.
- The material from BetterEvaluation comes from a combination of curating existing material and co-creating new material. This blog is part of an ongoing series about material that we have co-created with BetterEvaluation users.
- We've got our head in realism this week, partly because early-bird registrations for the 2017 International Realist Conference close soon, and partly because we've been shown Chris Lysy's realist cartoon series (commissioned by the Rameses
- We often get email enquiries asking for advice about preparing the documents used to invite evaluators to assemble proposals for an evaluation.
- Adaptive management is usually understood to refer to an iterative process of reviewing and making changes to programmes and projects throughout implementation.
- The wonderful thing about BetterEvaluation is that it is, at its core, a platform to co-create and share knowledge about how to better conduct, use and manage evaluations.
- In this edition of the BE FAQ blog, we address a question that comes up quite often: How do you go about analysing data that has been collected from respondents via a questionnaire?
- Big data is emerging as a new world currency.
- In May we blogged about ways of framing the difference between research and evaluation. We had terrific feedback on this issue from the international BetterEvaluation community and this update shares the results.
Resource
- The resource, developed by the Health Policy Project, is a self-assessment tool designed to align with an organization's mission concerning health policy, though the tool is useful more broadly outside the health sector.
- This guidance from the Footprint Evaluation Initiative aims to support those doing or overseeing evaluations to include environmental sustainability in feasible and useful ways.
- This resource, from the Footprint Evaluation Initiative, discusses how the six evaluation criteria of the OECD DAC (Organization for Economic Co-Operation and Development – Development Assistance Committee) can be used to get environmental
- This recorded webinar featuring Scott Chaplow, Juha Uitto, Eddah Kanini, and Lennise Baptiste provides a useful discussion of evaluation's role in mainstreaming environmental concerns, as well as practical guidance and resources for evaluators on how to
- In part one of this three-part webinar series, Andy Rowe and Patricia Rogers discuss what was learnt during the
- In the last part of this three-part webinar series, Andy Rowe and Patricia Rogers introduce a typology being developed that will assist a wide range of evaluations in assessing the effect of interventions on natural systems and sustainabili
- In part two of this three-part webinar series, Jane Davidson and Patricia Rogers discuss several ways to get sustainability on the evaluation agenda, even for projects that have no explicit environmental objectives and where there is no men
- This Footprint Evaluation Initiative report describes four 'thought experiments' undertaken as part of this project.
- This two-part series from CLEAR-Anglophone Africa and MERL Tech covers both theoretical and practical aspects of responsible data governance in M&E.
- This guide presents a number of options for creating visual press releases, which is especially important when your press release is for distribution on social media.
- This blog post by Lou Dubois writing for Inc. outlines some key elements of a traditional media release and gives guidance on how to adapt these to social media. It also touches on how to measure engagement.
- This webinar to launch the book Randomized Control Trials in the Field of Development: A Critical Perspective brings together five representatives of the book's editors and authors for a discussion around some of the key
- This paper is the first in the BetterEvaluation Monitoring and Evaluation for Adaptive Management working paper series.
- This is the second paper in the BetterEvaluation Working Paper Series, Monitoring and Evaluation for Adaptive Management.
- This paper explores the nexus of data science and evaluation, probing the issues and challenges of incorporating big data into evaluation practice. This resource and the following information was contributed by Alice Macfarlan.
- In this webinar on evaluative thinking, Thomas Schwandt explores two perspectives on the topic.
- In this AEA365 'Rad Resource' blog post, Sara Vaca discusses the importance and challenges of introducing a Gender perspective into evaluations - even when this is not included in the evaluation ToRs.
- This free, one-hour online course by USAID gives an introduction to the Collaboration, Learning, Adapting (CLA) approach.
- The Gender Spectrum Collection is a stock photo library featuring images of trans and non-binary models.
- Blue Marble Evaluation is a global initiative focused on training the next generation of evaluators to Think Globally, Act Globally and Evaluate Globally.
- This resource, produced by the South African Monitoring and Evaluation Association (SAMEA) brings together a range of content about monitoring and evaluation practice.
- This online tool is designed to assist with choosing accessible colour combinations for text on the web. This resource and the following information was contributed by Alice Macfarlan.
- This briefing paper from IIED argues that if the world is going to make significant progress towards achieving the Sustainable Development Goals (SDGs) by 2030, development actors will need to think and work in new ways, including in e
- This blog series shares advice for young and emerging evaluators from a range of experienced evaluation practitioners. The tips range from methodological advice to personal tips on building resilience and relationships.
- This template aims to help evaluators think through and document some of the issues around reporting needs during an evaluation. This resource and the following information was contributed by Alice Macfarlan.
- The report from UN Women, with support from UN Global Pulse, outlines the value of big data for monitoring the Sustainable Development Goals (SDGs) in relation to women.
- In this webinar, three evaluators (Kylie Hutchinson, Benoit Gauthier, and E. Jane Davidson) share their real-life evaluation blunders and ways to incorporate their lessons learned into the classroom or personal practice.
- Designing Quality Survey Questions addresses challenges such as language preferences for standard demographic questions (e.g.
- This four-part series by Elizabeth Grim is a good introduction for those new to Twitter.
- This site contains a great roundup of blog posts about using social media to encourage research uptake.
- A decision making matrix can be useful to summarise decision makers and types of decisions in a matrix which can be referred to when developing and implementing evaluation processes.
- This book by Kylie Hutchinson presents a number of innovative ways of reporting, including different methods for presentations, narrative summaries, presenting findings visually and making use of digital outputs.
- Qualitative Chart Chooser 3.0 is an updated version of Stephanie Evergreen and Jennifer Lyons attempt to organise different ways to show qualitative data.
- This suite of tools is useful for creating highly interactive, beautiful representations of data.
- This worksheet by the Action Evaluation Collective gives a steps by step run down of how to use storyboards to engage people in telling their stories. It's focus is on working with and engaging young people in a participatory process.
- In this example, the Independent Evaluation Group (IEG - part of the World Bank Group) have used a network analysis to gain a better understanding of the role of the World Bank Group's policy interventions in the health sector in Liber
- In this document, Richard Krueger outlines some key ideas around using observation effectively in evaluation.
- This report by DFID-ESRC Growth Research Programme (DEGRP) describes in detail how the researchers turned findings into recommendations, and how the various stages of stakeholder consultation influenced different elements of the projec
- Dylomo is a free, web-based tool that can be used to create interactive, online logic models.
- This checklist by Kelly N. Robertson and Lori Wingate provides suggestions for the content and organization of long-form evaluation reports that are concise, easy to understand, and easy to navigate.
- This is a step-by-step guide for creating a dashboard with Google Sheets (a free, online spreadsheet application).
- This tool, developed by USAID/Rwanda and shared as part of USAID's Collaboration, Learning and Adapting (CLA) Toolkit, is one way of graphically depicting relationships between key stakeholders.
- USAID's annual Collaborating, Learning and Adapting (CLA) Case Competition captures case studies of USAID staff and implementing partners using a CLA approach for organizational learning and better development outcomes.
- The Health Policy Project's series of Capacity Development Resource Guides are focused on enhancing the abilities required to impact health policy through design, implementation, and monitoring and evaluation, though many of the areas cover
- This blog by Stephanie Evergreen curates a number of examples of engaging executive summaries and evaluation reports. The focus is on highlighting effective visual design of these reporting formats.
- This blog post in AEA365: A Tip-a-Day by and for Evaluators by Artineh Samkian and Joelle Greene discusses the use of Graphic Timelines as a way of presenting data to stakeholders that was both acc
- This list compiled by Alex Ivanovs brings together a range of books aimed at both beginners and those who are looking to fill in gaps in their knowledge.
- This report on a community-based survey about Jamaican youth risk and resiliency behaviour conducted in 2006 on behalf of USAID contains an example of a data use calendar as Appendix 9 (page 158).
- Stephanie Evergreen (Evergreen Data) and Ann K.
- This online guide by Ann K. Emery walks user through the full process of creating a data visualisation, from understanding your audience, through to sharing your dataviz.
- A powerpoint put together by Steve Montague.
- This blog post by Ann K. Emery and Johanna Morariu outlines their strategies for setting up and running a webinar. This resource and the following information was contributed by Alice Macfarlan.
- This chapter section by Eben A. Weitzman guides readers through some of the important considerations when choosing appropriate qualitative analysis software.
- This document, created by the UK Data Archive, gives researchers an overview of the best practices involved in data management for long-term, accessible data storage, protection and archive.
- Written by Darren Cambridge and Vicki Suter for the EDUCAUSE Learning Initiative (ELI), this guide provides practical steps and advice for the creation of Communities of Practice (CoPs).
- A free and easy to use platform available online and as an iPad app for quickly making simple, well laid-out presentations.
- This NVivo 10 help file walks readers through the process of creating and using framework matrices in NVivo for the analysis of qualitative data.
- ICPSR is an international consortium made up of over 700 academic institutions and research organizations.
- This guide addresses the issue of ensuring that evaluation findings are used by stakeholders.
- Produced for the Early Childhood Nutrition and Anaemia Prevention Project, Fred Hollows Foundation, this Summary Report presents a large amount of non-numeric data effectively through the use of a timeline of the project.
- This short guide by the Technical and Operational Performance Support (TOPS) Program and USAID focuses on identifying appropriate formats and approaches for presentations, group discussions and problem solving sessions to enhance
- This report presents the results of a performance audit conducted by the Office of the Auditor General of Canada.
- Phil Rabinowitz's chapter in CHINA.org gives a detailed overview of GIS mapping.
- This free, online course run by the Open Learning Initiative takes students through the basics of statistics and probability.
- In this blog post for the NVivo blog, Meg Callanan discusses her experiences using the NVivo framework matrices function with both small and large data sets.
- Adobe CC (formerly Adobe Kuler) is useful for generating colour schemes from screen shots of, say, a logo or photograph.
- Michael Woolcock's paper addresses the fact that rising standards required for assessing the impact of development projects have not been met with equally rigorous procedures for guiding decisions about external validity - "whether and
- The GESIS - Leibniz Institute for the Social Science's Data Archive focuses primarily on the processing and archiving of German and international comparative surveys in the fields of social and political science.
- Part of the Australian Bureau of Statistic's Statistic Language website, this guide to Correlation and Causation is easy to follow and explains the major features of correlation and its difference from causation.
- This article by Anthony Morgan and Peter Homel of the Australian Institute of Criminology discusses the importance of evaluation in crime prevention.
- This presentation at the AEA Evaluation 2013 Conference by Artineh Samkian, the University of Southern California, and Joel Greene, Harder+Company, gives an overview of the steps involved in creating a graphic timeline that is acc
- This resource is a friendly, practical and fun guide to giving presentations, aimed at those who are not confident about presenting. This resource and the following information was contributed by Alice Macfarlan
- This article explores the issues of developing standards for an evaluation when these have not previously been agreed upon, in a rural development program in Argentina.
- This blog post by Stephanie Evergreen from Evergreen Data uses the designs of a number of business cards that she had collected over the years to highlight a number of important principles o
- This resource is an invited presentation to the European Parliament committee on regional development.
- This book by Sarah Earl, Fred Carden and Terry Smutylo takes an original approach to assessing development impacts by focusing on the way in which people relate to each other and to their environment rather than simply evaluating the p
- This manual, written by Karina Kielmann, Fabian Cataldo and Janet Seeley, aims to give readers of a non-scientific background an introduction to key theoretical concepts and methodologies in qualitative research.
- This guide by the Center for Research on Environmental Decisions, while focused on communicating research on climate change, will be useful for anyone interested in the theory behind communication and behaviour change and those who nee
- This chapter Jon A. Krosnick and Stanley Presser presents a number of recommendations about survey design based on conventional wisdom and a review of the methodological literature.
- This guide to the Most Significant Change approach, by the Overseas Development Institute (ODI), provides an overview, a detailed description of the process, and an example of the technique in action.
- In this journal article, Jane Davidson advocates for better evaluation practice by picking apart four common practices in academic and social research that evaluators might inappropriately apply when working in the field of evaluation.
- Although focused on viewing critical reflection through the scope of management learning, this article discusses a range of critical reflection tools that have been drawn from other professional areas, such as storytelling, metap
- This report by Burt Perrin offers evidence that countries are moving away from evaluating the performance of government on activities, inputs and outputs, and focusing instead on a results-driven approach.
- This paper in the Nurse Researcher journal by Tracey Williamson and Andrew Long walks readers through the benefits of using a number of different types of data displays when pres
- Burt Perrin's 2002 report to the OECD outlines common challenges in performance measurement and results-based management, and identifies potential solutions to these.
- In this 1998 paper, Steve Montague addresses the limitation of logic models in that they have a tendency to direct their attention to causal chains without making reference to who and where the action was.
- This 1998 paper by J.
- This document, produced by the United Nations World Food Programme's Office of Evaluation, lays out some general guidelines on the management of an evaluation and how to disseminate the evaluation's findings effectively to key stakeholders.
- This interactive graphic created by Kristoffer Magnusson graphically illustrates the principles of
- The Treasury Board of Canada Secretariat's Centre of Excellence in Evaluation (CEE) provided advice and guidance on evaluation practices across the federal government.
- Jason Davies' website contains a large number of data viz examples, particular around geo data. It also contains some links to to tools, including a word cloud and tree map.
Theme
- Footprint evaluation aims to embed consideration of environmental sustainability in all evaluations and monitoring systems, not only those with explicit environmental objectives.
Approach
- Qualitative Comparative Analysis (QCA) is an evaluation approach that supports causal reasoning by examining how different conditions contribute to an outcome.
- An impact evaluation approach based on identifying and investigating the most successful cases and seeing if their results can justify the cost of the intervention (such as a training course).
Method
- Evaluation journals play an important role in documenting, developing, and sharing theory and practice. They are an important component in strengthening evaluation capacity.
- Reviewing documents produced as part of the implementation of the evaluand can provide useful background information and be beneficial in understanding the alignment between planned and actual implementation.
- Lessons learnt can take the form of describing what should or should not be done, or describing the outcome of different processes.
- An internet questionnaire allows the collection of data through an electronic set of questions that are posted on the web.
- A validation workshop is a meeting that brings together evaluators and key stakeholders to review an evaluation's findings.
- Correlation is a statistical measure ranging from +1.0 to -1.0, represented by 'r', that indicates how strongly two or more variables are related and whether that relationship is positive or negative.
- Key informant interviews involve interviewing people who have particularly informed perspectives on an aspect of the program being evaluated.
- Analytical generalisation involves making projections about the likely transferability of findings from an evaluation, based on a theoretical analysis of the factors producing outcomes and the effect of context.
- Measures of dispersion provide information about how much variation there is in the data, including the range, inter-quartile range and the standard deviation.
- An important aspect of data quality is to ensure data is collected consistently across different sites and different data collectors.
- Effective data transfer involves processes to move data between systems, including between software packages, to avoid the need to rekey data.
- Putting systems in place to store de-identified data so that they can be accessed for verification purposes or for further analysis and research in the future, researchers can extend the range of the data collection efforts and en
- A community of practice allows a group of people with a common interest or concern to share and learn through a series of interactions, thus reflecting the social nature of human learning.
- There are a number of ways that documents can be made more accessible to people who are blind or have low vision.
- A key to creating effective and accessible reporting documents is using effective techniques to emphasise important information.
- Questionnaires and surveys can be conducted through mobile phones which are able to connect to the internet.
- Multiple lines and levels of evidence (MLLE) is a systematic approach to causal inference that involves bringing together different types of evidence (lines of evidence) and considering the strength of the evidence in terms of different ind
- An in-depth interview is a type of interview with an individual that aims to collect detailed information beyond initial and surface-level answers.
- Graphic recording is the translation of conversations into images and text on large sheets of paper during meetings and events.
- Non-participant Observation involves observing participants without actively participating.
- Process tracing is a case-based and theory-driven method for causal inference that applies specific types of tests to assess the strength of evidence for concluding that an intervention has contributed to changes that have been observed or
- Face-to-face questionnaires are conducted by an interviewer asking questions of a respondent in person.
- Email Questionnaires are surveys or questionnaires that are distributed online via email.
- Mentoring is a process where people are able to share their professional and personal experiences in order to support their development and growth in all spheres of life.
- Interactive mapping involves using maps that allow zooming in and out, panning around, identifying specific features, querying underlying data such as by topic or a specific indicator (e.g., socioeconomic status), generating reports and oth
- Data cleaning involves the detection and removal (or correction) of errors and inconsistencies in a data set or database due to data corruption or inaccurate entry.
- Measures of Central Tendency provide a summary measure that attempts to describe a whole set of data with a single value that represents the middle or centre of its distribution.
- Big data refers to data that are so large and complex that traditional methods of collection and analysis are not possible.
- Data backup refers to onsite and offsite, automatic and manual processes to guard against the risk of data being lost or corrupted.
- Processes to protect electronic and hard copy data in all forms, including questionnaires, interview tapes and electronic files from being accessed without authority or damaged.
- Timelines and time-ordered matrices are useful ways of displaying and analysing time-related data.
- A framework matrix is a way of summarizing and analyzing qualitative data in a table of rows and columns.
- A data use calendar is produced to guide the collection of data and reporting requirements, as well as ensuring that analysis and evaluation data is actively used.
- Questionnaires can be mailed out to a sample of the population, enabling the researcher to connect with a wide range of people.