C4D Hub: Specify the key R,MandE questions

What is it?

R,M&E Questions are the small number of broad questions that R,M&E are intended to answer, not the many specific questions that might be on a questionnaire or an interview schedule.  Deciding which questions should be answered is one of the most important and often the most difficult part of designing M&E (Catley et al. 2008: 12). The approach advocated here is a questions-driven approach, where key users first agree on what they need to know and use that as the basis for selecting methods and indicators. The evaluation part of R,M&E systems, by definition, should answer truly evaluative questions: it must ask not only ‘What were the results?’ (a descriptive question) but also ‘How good were the results?’ (an evaluative question). Depending on the type of M&E, causal questions also need to be addressed (to what extent were the results due to the intervention?).

General Information 

The approach advocated on this page draws heavily from the 'Develop agreed key evaluation questions' and the 'Consider important aspects of the evaluation' pages in the Steps in the Commissioning Process tool. This particular resource brings clarity to the task of articulating questions and understanding the type of question being asked (descriptive, causal, evaluative and action), and therefore the kinds of methods that can be used to answer them. Other key generalist advice includes:

  • Limit the number to 5-7 high level questions
  • Understand the kinds of questions asked at different points in the program cycle (click here for more)
  • Use the purpose to guide the selection of questions 

The Specify the Key Evaluation Questions page on the BetterEvaluation webpage similarly offers generalist advice, and presents the task in a slightly different way. It includes some good links to guide on engaging with stakeholders to develop evaluation questions, which is useful for taking a participatory approach to specifying questions. These pages are recommended background reading before considering options to apply to C4D. 

Specifying Key R,M&E Question in C4D

Why it is useful to analyse the types of questions within key questions for C4D

Observations as part of the Evaluating C4D project with UNICEF have revealed two problems:

  • People often think that indicators come first, and questions are developed based on these
  • Relatedly, R,M&E work tends to deal mostly focus on describing situations, and it is assumed that from there it is easy to infer contribution and causation, and decide on actions.

In particular, questions about contribution and attribution in C4D are common themes in discussions about needs, but causal questions and methods are rarely incorporated in C4D R,M&E designs. If questions about C4D contributions are important for stakeholders, it is vital that causal questions are reflected in the key questions (most likely as sub-questions) selected.

When deciding on key questions for C4D R,M&E the following steps are recommended:

  1. Use the Program Theory or Logic Model  
  2. Consider the types of key questions
  3. Analyse each Key Question  

Applying the C4D principles

Participatory 

The C4D Evaluation Framework advocates for a participatory approach. In the context of specifying key questions, a participatory approach would mean engaging (at least) with primary intended users and other stakeholders to decide on key questions.  

Holistic 

C4D initiatives usually respond to problems strongly connected with different social, cultural, economic, political, geographic and structural contexts. This means that in C4D R,M&E it is important to ask questions about underlying causes and social, cultural economic, political, geographic and structural contexts - from the situation analysis right through to the monitoring and evaluation. 

Critical 

In C4D it is important that questions are framed in such a way that allows for multiple and diverse voices to contribute answers. This is important for descriptive questions, causal questions and evaluative questions. 

Realistic 

In C4D the questions should be written in a way that calls for need for various methods and tools that will capture people's voices.

Complex

It it likely that there will be differing views that need to be taken into account about what the key R,M&E questions should be. In addition, the boundaries may need adjusting as situations change, particularly with the emergence of new understandings, stakeholders and ideas.

 

Resources

Monitoring and Evaluation of Participatory Theatre for Change (PTC) - This guide sets out six key evaluation questions around the concept of 'Relevance' (where monitoring questions are structured according to 'Reach, Resonance, Response'). These questions directly relate to the Theory of Change, though are broad and forward looking. Although it is written with reference to Participatory Theatre, the resource can be easily adapted to a range of C4D approaches, especially participatory C4D approaches. Click here to read a summary and review of this resource. This resource is consistent with the C4D Evaluation Framework in relation to this task in the following ways:

  • complex: the strong use of a theory of change to guide the selection of evaluation questions
  • realisticthe six questions are specific. There are not too many questions, but there are no major gaps. 
  • Learning-basedthe evaluation questions will not just check what happened, but seek out new insights and practices that can be used to inform future programs.  

Examples

Retrospective Analysis study of Open Defecation in Nadia District, India: 

The Retrospective Analysis of ODF in Nadia District, India is an example of a study that was framed by questions about underlying causes and contexts. It is consistent with the C4D Evaluation Framework in relation to this task in the following ways:

  • realistic:  the questions bind the focus to the needs of the stakeholders, based on gaps in the knowledge.

  • participatory: the first phase of the study engaged with key stakeholder to find out what their key questions are.

More information about how this study exemplifies the approaches advocated in the C4D Evaluation Framework will be available soon.

Comments

There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.