Big Data

Big data refers to data that are so large and complex that traditional methods of collection and analysis are not possible. The amount and variety of big data has increased exponentially over the past decade. 'Data exhaust' is one source, which relates to data produced passively as a byproduct of user interactions with a system, such use of a mobile phone service or Internet banking. Online information is another, giving an indication of human intent, emotions, and wishes by collecting the information contact in web content such as news and social media interactions (e.g., Facebook, blogs, twitter), and online search history. Data can also come from physical sensors such as satellite images and traffic information (UNDP 2013).

One trend in big data is the promotion of open data (or open knowledge). This entails the free-release of de-identified data sets to the public. This allows underused data sets (for example, government data sets) that contain valuable data to be accessed by people who can turn this raw data into something useable and useful. For example, VicRoads has released a number of data sets about crashes and road networks, which will be useful for people who are wanting to evaluate traffic interventions. The Open Knowledge website has more information about this trend.


The UN Global Pulse runs a number of projects that utilise social media data to monitor social and environmental issues. For example, one project analyses social media conversations to understand public perceptions of sanitation, providing a baseline of how public discourse of sanitation changed over time so as to allow monitoring of the effectiveness and reach future health campaigns. Similarly, other Global Pulse projects use Twitter to measure global engagement on climate change and to evaluate the impact of the Every Women Every Child movement since its 2010 launch, by monitoring changes in related public health dialogue in Twitter conversations.


BetterEvaluation Blog Posts

Big Data and Evaluation - Use and Implications - Alice Macfarlan takes a look at some of the ways that Big Data is being used in evaluation, and raises some questions about the implications of this emerging form of information.


  • Global Pulse - Global Pulse is an innovation initiative launched by the Executive Office of the United Nations Secretary-General, which conceives and co-ordinates research on big data for development through a network of innovation labs.

  • Open Knowledge Foundation Australia - A site that shares knowledge about open data sets.



  • Introduction to Big Data - by Hilary Mason, Chief Data Scientist at Bitly

  • Big Data Intro by Global Pulse - Andreas Weigend, former Chief Data Scientist for and Peter Hirshberg, former head of Enterprise Marketing for Apple, Inc., offer a range of examples and visualisations of how Real-Time, or big data, can augment and improve decision making in the 21st century



  • Big Data, Big Ruse - This article from Stephen Few of Perceptual Edge provides a critique of many of the arguments used to promote big data - in particular concerns that without adequate analysis and visualisation, big data will just increase the problems of monitoring.


UNDP (2013) Discussion Paper: Innovations in Monitoring and Evaluation. Retrieved from 

UN Global Pulse (2012). Big Data for Development: Challenges & Opportunities [White Paper]. Retrieved from

Updated: 9th January 2018 - 12:28pm
This Option is useful for:
A special thanks to this page's contributors
BetterEvaluation Website and Engagement Coordinator, BetterEvaluation and ANZSOG.
Melbourne, Australia.


There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.