Measuring results and impact in the age of big data

This paper explores the nexus of data science and evaluation, probing the issues and challenges of incorporating big data into evaluation practice.

This resource and the following information was contributed by Alice Macfarlan.

Authors and their affiliation

Authors - York, P. and Bamberger, M.

Affiliation - The Rockefeller Foundation

Key features

(The following is abridged and edited from the executive summary)

Chapter 1 of the paper looks at how data science vastly broadens the range of data and analytical tools, including predictive models, that evaluators can draw upon. 

Chapter 2 shows how big data is increasingly used in all aspects of our personal and professional lives, and throughout the commercial and public sectors. In defining big data, it discusses the significant differences between big data and the kinds of conventional data currently used in most evaluations.

Chapter 3 discusses the increasing use of big data in social and economic programs in both industrialized and developing countries, noting that Agriculture, health, and education are among the sectors in which big data has had the most impact. It also looks at the slow progress of integrating big data into evaluation practice, and examines why this is and the consequences of this slow uptake for the field of evaluation.

Chapter 4 introduces some of the big data tools that can help strengthen evaluations by reducing the time, cost, and effort required for data collection, sample design, and data analysis.

Chapter 5 discusses some of the areas of disagreement or misunderstanding between evaluators and data scientists, and categorizes them into issues relating to theory, data quality and validity, and sampling and sample selection bias. 

Chapter 6 draws on the previous two chapters to illustrate how data science tools and techniques can be applied to strengthen evaluations, focusing on some of the challenge areas faced by conventional program evaluation: evaluation design, data collection, sample design, and data analysis and dissemination of findings.

Chapter 7 uses the example of gender differences in the impacts of big data and other new information technologies to illustrate the importance of understanding how different groups – based on gender, age, income, or geographical location – have access to, use, and are affected by the new technologies. It begins by illustrating how women and men have different experiences with the new information technology. 

The concluding Chapter 8 summarizes lessons about the potential benefits and challenges at the nexus of data science and evaluation practice, and builds on identifying a set of recommendations on ways to move forward to create a conducive environment for integration. 

How have you used or intend on using this resource?

I found this a useful resource for thinking about the use of big data in evaluation in a way that goes beyond techniques and methodologies. I found the focus on the challenges of integrating the two fields of data science and evaluation, and the consequences of not doing so for both fields, particularly helpful for looking at the wider picture of big data and evaluation.  

Why would you recommend it to other people?

This is an important read for those working in evaluation who are considering or starting to integrate big data into their evaluation practice, as well as for those who are trying to advocate for more use, or more thoughtful use, of big data in evaluation generally.

Sources

York,P. & Bamberger, M. (2020). Measuring results and impact in the age of big data: The nexus of evaluation, analytics, and digital technology. The Rockefeller Foundation. Accessed on 18th May 2021 from https://www.rockefellerfoundation.org/wp-content/uploads/Measuring-results-and-impact-in-the-age-of-big-data-by-York-and-Bamberger-March-2020.pdf