Big data refers to data that are so large and complex that traditional methods of collection and analysis are not possible. The amount and variety of big data has increased exponentially over the past decade. 'Data exhaust' is one source, which relates to data produced passively as a byproduct of user interactions with a system, such use of a mobile phone service or Internet banking. Online information is another, giving an indication of human intent, emotions, and wishes by collecting the information contact in web content such as news and social media interactions (e.g., Facebook, blogs, twitter), and online search history. Data can also come from physical sensors such as satellite images and traffic information (UNDP 2013).
One trend in big data is the promotion of open data (or open knowledge). This entails the free-release of de-identified data sets to the public. This allows underused data sets (for example, government data sets) that contain valuable data to be accessed by people who can turn this raw data into something useable and useful. For example, VicRoads has released a number of data sets about crashes and road networks, which will be useful for people who are wanting to evaluate traffic interventions. The Open Knowledge website has more information about this trend.
The UN Global Pulse runs a number of projects that utilise social media data to monitor social and environmental issues. For example, one project analyses social media conversations to understand public perceptions of sanitation, providing a baseline of how public discourse of sanitation changed over time so as to allow monitoring of the effectiveness and reach future health campaigns. Similarly, other Global Pulse projects use Twitter to measure global engagement on climate change and to evaluate the impact of the Every Women Every Child movement since its 2010 launch, by monitoring changes in related public health dialogue in Twitter conversations.
BetterEvaluation Blog Posts
Big Data and Evaluation - Use and Implications - Alice Macfarlan takes a look at some of the ways that Big Data is being used in evaluation, and raises some questions about the implications of this emerging form of information.
Global Pulse - Global Pulse is an innovation initiative launched by the Executive Office of the United Nations Secretary-General, which conceives and co-ordinates research on big data for development through a network of innovation labs.
- Open Knowledge Foundation Australia - A site that shares knowledge about open data sets.
Discussion Paper: Innovations in Monitoring and Evaluation - This discussion paper produced by the United Nations Development Programme discusses various innovations that are occurring in M&E, and the advantages and disadvantages of these methods
- Big Data for Development: Challenges & Opportunities - This white paper by UN Global Pulse examines the use of Big Data in development contexts.
Introduction to Big Data - by Hilary Mason, Chief Data Scientist at Bitly
Big Data Intro by Global Pulse - Andreas Weigend, former Chief Data Scientist for Amazon.com and Peter Hirshberg, former head of Enterprise Marketing for Apple, Inc., offer a range of examples and visualisations of how Real-Time, or big data, can augment and improve decision making in the 21st century
- Mining Indonesian Tweets to Understand Food Price Crises - This research project identified that analysis of tweets could provide early warning and impact monitoring of food price rises.
- Big Data, Big Ruse - This article from Stephen Few of Perceptual Edge provides a critique of many of the arguments used to promote big data - in particular concerns that without adequate analysis and visualisation, big data will just increase the problems of monitoring.
UNDP (2013) Discussion Paper: Innovations in Monitoring and Evaluation. Retrieved from http://www.outcomemapping.ca/download/UNDP%20Discussion%20Paper%20Innovations%20in%20Monitoring%20and%20Evaluation.pdf
UN Global Pulse (2012). Big Data for Development: Challenges & Opportunities [White Paper]. Retrieved from http://www.unglobalpulse.org/sites/default/files/BigDataforDevelopment-UNGlobalPulseJune2012.pdf