Big data refers to data that are so large and complex that traditional methods of collection and analysis are not possible.
The amount and variety of big data has increased exponentially over the past decade. 'Data exhaust' is one source, which relates to data produced passively as a byproduct of user interactions with a system, such use of mobile phone services or internet banking. Online information is another, giving an indication of human intent, emotions, and wishes by collecting the information contact in web content such as news and social media interactions (e.g., Facebook, blogs, Twitter), and online search history. Data can also come from physical sensors such as satellite images and traffic information (UNDP 2013).
One trend in big data is the promotion of open data (or open knowledge). This entails the free release of de-identified data sets to the public. This allows underused data sets (for example, government data sets) that contain valuable data to be accessed by people who can turn this raw data into something useable and useful. For example, VicRoads has released a number of data sets about crashes and road networks, which will be useful for people who are wanting to evaluate traffic interventions. The Open Knowledge website has more information about this trend.
The UN Global Pulse runs a number of projects that utilise social media data to monitor social and environmental issues. For example, one project analyses social media conversations to understand public perceptions of sanitation, providing a baseline of how public discourse of sanitation changed over time so as to allow monitoring of the effectiveness and reach future health campaigns. Similarly, other Global Pulse projects use Twitter to measure global engagement on climate change and to evaluate the impact of the Every Women Every Child movement since its 2010 launch, by monitoring changes in related public health dialogue in Twitter conversations.
BetterEvaluation Blog Post
- Global Pulse
Global Pulse is an innovation initiative launched by the Executive Office of the United Nations Secretary-General, which conceives and co-ordinates research on big data for development through a network of innovation labs.
- Open Knowledge Foundation Australia
A site that shares knowledge about open data sets.
- Introduction to Big Data
by Hilary Mason, Chief Data Scientist at Bitly
- Big Data Intro by Global Pulse
Andreas Weigend, former Chief Data Scientist for Amazon.com and Peter Hirshberg, former head of Enterprise Marketing for Apple, Inc., offer a range of examples and visualisations of how Real-Time, or big data, can augment and improve decision making in the 21st century
- Mining Indonesian Tweets to Understand Food Price Crises
This research project identified that analysis of tweets could provide early warning and impact monitoring of food price rises.
UNDP (2013) Discussion Paper: Innovations in Monitoring and Evaluation. Retrieved from http://www.outcomemapping.ca/download/UNDP%20Discussion%20Paper%20Innovations%20in%20Monitoring%20and%20Evaluation.pdf
UN Global Pulse (2012). Big Data for Development: Challenges & Opportunities [White Paper]. Retrieved from http://www.unglobalpulse.org/sites/default/files/BigDataforDevelopment-UNGlobalPulseJune2012.pdf
'Big data' is referenced in:
- Rainbow Framework :