Seven evaluation methods to add to the toolbox?

Patricia Rogers's picture 3rd August 2012 by Patricia Rogers

In a recent workshop on 'Designs for Performance Evaluation', which Patricia Rogers conducted with program officers from USAID, we looked at seven methods and strategies which might be usefully added to the repertoire  for collecting, analyzing or synthesizing data.

Which of these might be useful to add  to the methods that you use  - or that you encourage evaluators to use in evaluations that you manage?

1. Mobile Phone Data Logging

Mobile phones (cell phones) can be a great method for collecting data - and they don't have to be smart phones.  Phones can use SMS, Java or Android to log data. Advantages of using this method include making decentralized data collection easier and providing immediate reporting and analysis.

For an example, check out how Water for People is using a mobile phone-based technology called FLOW – Field Level Operations Watch – to maintain accountability and allow communities to monitor their own water and sanitation.

NOMAD  has developed a useful review of the current state of the art in mobile data collection systems

Mariel Lazcano using FLOW to interview a community member. Photo: Water for People

Read more:  Mobile Data Collection page

2. Community mapping of an area’s resources and problems

Community mapping can be used to aggregate local citizen reports, community media and other relevant news and information.

The Map Kibera project has gathered data about the location of services to produce the first map of Kibera, Nairobi (one of the largest slums in Africa).

Map Kibera http://mapkibera.org/

3. Interactive mapping of crowdsourced data

Interactive maps can be built from data contributed by text messages, email, twitter, news and web-forms. Like mobile phone data logging this method allows for decentralized data collection and rapid reporting and analysis.

Ushahidi  is a non-profit tech company that specializes in developing free and open source software for information collectionvisualization and interactive mapping.

Stop Stockouts uses interactive mapping to share information and advocate against stockouts (when a pharmacy has no medicine on the shelf).

Stop Stockouts map of pharmacies out of stock of first line malaria drugs

Read more: Interactive Mapping page

4. Photovoice

This method involves community members in taking photos, selecting the most significant photos, telling stories about what they mean, and identifying key themes.

PhotoVoice builds skills within disadvantaged and marginalised communities using participatory photography and digital storytelling methods to represent themselves to achieve positive social change.

Read more: PhotoVoice page

5. Structured observation

Instead of having vague and unsystematic observation during field visits, this method uses trained and experienced observers who look for particular aspects of implementation and impact, record what they observe where possible, record details about the context, and check the meaning of their observations.

Useful guidance on observation can be found in  Tips on Observation from the USAID Center for Development Information and Evaluation.

Read more:

  • Field Trips: organizing trips where participants visit physical sites.
  • Non-participant Observation: observing participants without actively participating.
  • Participant Observation: identifying the attitudes and operation of a community by living within its environs.
  • Photography/video: discerning changes that have taken place in the environment or activities of a community through the use of images taken over a period of time.
  • Transect: gathering spatial data on an area by observing people, surroundings and resources while walking around the area or community.

6. Keypad technology

This technology allows data to be collected anonymously from individuals in a group setting using special clickers or texts from cellphones. Uses can select from multiple choice options and the results can be displayed immediately. The results can form the basis for further discussion or be used as voting.

The National Coalition for Dialogue & Deliberation has developed information on keypad technology .

Read more: Keypad technology page

7. Evaluation rubric

Rubrics are an attempt to be systematic and transparent about evaluative judgements by developing ratings from evaluative inference.

presentation by Jane Davidson, Nan Wehipeihana and Kate McKegg describes evaluation rubrics and two examples of their application.

Read more: 

Rubrics page

Case study by Judy Oakden on the application of rubrics in an education program

Update:  We updated this post on 18 September 2014 to add links to resources on the BetterEvaluation site that have been added since the post was first published.

 

 

 

A special thanks to this page's contributors
Author
Director of BetterEvaluation/ Professor of Public Sector Evaluation, Australia and New Zealand School of Government.
Melbourne.