How many methods do you usually see in evaluation reports as having been used to collect data? Chances are you’ll see project document review, key information interviews, surveys of some kind, and perhaps group interviews with intended beneficiaries. These methods are all useful to help describe what has happened, the outcomes and the context in which change occurred.
It is not difficult to go beyond these usual suspects. Many other methods exist to collect data that can enrich the analysis and improve the quality of the evaluation.
For example, polling booth and keypad technology are more appropriate than a group interview for gathering information about sensitive issues. Methods such as PhotoVoice and PhotoLanguage can help people articulate what they value in ways that go beyond what they might be able or willing to share in an interview. Participatory methods, such as dotmocracy and murals, are ways to support discussions by a group of the different perspectives among the members. Or what about a reputational monitoring dashboard to automatically track what is being said about an organization or a project in social media?
The Describe cluster guides you on the options for collecting or retrieving data that are available, when to choose which ones and how to use them well.
But data collection is only one part of the Describe cluster. It also includes options for critical tasks such as how to sample people, sites or time periods, how to combine qualitative and quantitative data, and how to use existing measures and indicators. Once you have the data, you can turn to this cluster for ideas about how to manage the data, clean it, and analyze data, including a section on data visualization to facilitate analysis.
In the recent AEA CoffeeBreak webinar series, I provided an overview of these tasks and some of the available options. You can watch it here or on YouTube [20 minutes].
Are there other methods you use to collect data to describe what has happened? Do you have some good examples of using innovative methods to document implementation or record opinions and perspectives, or tips for managing data and analysing multiple sets of data? Please share these ideas with us here or simply add a comment below. We can then share it with the world.
Questions from the webinar
1) Does this cluster of tasks refer to planning for describing the data collection plan and presenting findings?
The Describe cluster covers planning and implementing data collection and facilitating analysis, how to choose options well and use them well. The cluster on Report and Support Use has information about different options for presenting findings.
2) What software package do you use for analysing qualitative data?
There are a number of different packages available. The CAQDAS site (Computer Assisted Qualitative Data Analysis Software) provides an overview of different packages, including NVivo, HyperResearch, and Atlas TI.
3) You mentioned 'combining qualitative and quantitative data'. Please explain what you mean by 'combining'.
It can help to start by considering why you want to combine qualitative and quantitative data. Is it intended to triangulate data, to check interim findings from different perspectives and using different methods with complementary strengths? Is it primarily intended to elaborate or explain findings? Do you need to elaborate on what lies behind a simple quantitative change or do you want quantify a described change?
Once you are clearer on the purpose, there are different ways of combining quantitative and qualitative data.
Sometimes they are combined sequentially. For example, an evaluation might start with a few open-ended interviews and reviewing project documentation to identify core issues, and then using this to develop a structured questionnaire to collect quantitative data. Or it might start by analyzing existing quantitative data on service delivery, and then explore issues through interviews that generate qualitative data.
Sometimes the two types of data are collected simultaneously with one method. For example, a questionnaire might have quantitative as well as qualitative data, and might be combined with quantitative and qualitative data gathered through observation.
You can find more information and examples about combining qualitative and quantitative data at http://betterevaluation.org/plan/describe/combining_qualitative_and_quantitative_data.
This blog post is part of a series of eight posts covering the BetterEvaluation Framework and presenting the recordings of eight corresponding webinars hosted by the American Evaluation Association. The full series of posts is below.
|1. Using the Rainbow Framework, Irene Guijt||5. Understanding causes, Jane Davidson|
|2. Defining what needs to be evaluated, Simon Hearn||6. Weighing the data for an overall evaluative judgment, Patricia Rogers|
|3. Framing the evaluation, Patricia Rogers||7. How can evaluation make a difference?, Simon Hearn|
|4. Choosing methods to describe activities, results and context, Irene Guijt||8. Manage an evaluation or evaluation system, Kerry Bruce|