C4D Hub: Create a Questions-Led M&E Framework

What is it?

A questions-led M&E Framework starts with thinking about the information needs (questions) of the primary intended users, and builds a plan for answering those questions. This is a good option for C4D and is consistent with the C4D Evaluation Framework in the following ways:

  • Participatory: The potential uses that stakeholders, especially the primary intended users, have are the focus of the M&E. These stakeholders and users should be involved in deciding on the purpose and questions, and selecting options for answering questions.
  • HolisticThe key M&E questions drive the direction of the framework. These questions should go beyond 'what happened' and also question the causes, how good programs and results are, and what to do next. 
  • CriticalA questions-led M&E Framework encourages mixed methods to build a rich understanding of what is working, and what is not working, for different groups. 
  • Realistic: A questions-led M&E Framework prioritises efforts around the questions that matter most to users. It does not try to measure everything. If primary intended users want to know about impact of C4D initiatives, that implies certain types of strategies, and should be planned for as part of the M&E Framework. If there are lots of uncertainties about what might work, an M&E Framework can be built to allow for trialling and comparison of different strategies that are investigated through smaller studies and inform an emergent approach.   
  • Learning-basedA questions-led M&E Framework takes learning from RM&E seriously, beyond a list of recommendations at the end. If key users priorities understanding how to make improvements during implementation, this implies certain strategies. Further, learning structures, events and processes (such as committees, annual reviews etc.) can be built into the M&E Framework.
  • AccountableA questions-led M&E Framework supports a true accountability, beyond compliance-oriented reporting against indicators, through building a rigourous, mixed-methods M&E Framework that can be designed to answer questions about effectiveness, impact, relevant and other quality standard criteria.   
  • ComplexA questions-led M&E Framework is much easier to design around complicated and complex types of C4D initiatives and problems. Depending on the framing of key questions, a Questions-Led M&E Framework can be designed to support emergent and responsive implementation using methods and strategies suited to understanding uncertainty. The focus on questions means it remain realistic, rather than trying to measure every single thing that might possibly be measured.

Steps:

Step 1. Recommended preparation tasks: a checklist

The M&E Framework should be informed by several other important decisions and tasks. The C4D Evaluation framework approach would suggest consideration of the following aspects as preparation for undertaking this task:

Step 2: Specify the key questions, and analyse them by type.

Different types of questions require different types of methods and strategies to get answers. The four main types are:

  • Descriptive 
  • Causal
  • Evaluative
  • Predictive and Action

(click here for more information on the Specify Key Questions task, and on analysing each Key Evaluation Question).  

Step 3: Click to download matrix template to fill in as you make decisions:

Matrix Template.docx

Step 4: Sort Questions by Type

  1. Start by sorting all the smaller questions by their type. This means making a new list of all the descriptive questions, all the causal questions, all the evaluative questions, and all the action/predictive questions (it is helpful to keep the numbers, i.e. 1.1, 1.2 etc. for resorting according to the Key Question later).
  2. Identify any questions that are the same or similar, and if possible adjust the wording of very similar questions slightly to avoid unnecessary duplication, making sure not to lose the essence of any questions.
  3. Paste the list of questions under each of the headings (Descriptive, Causal, Evaluative, Action/Predictive) in Matrix Template document the space provided. (see NationalProgramforChildProtectionCommunicationMEPlan.docx starting at page 17)

Step 5: Decide how to answer descriptive questions and compile a matrix

In your matrix template add all the descriptive questions to the first column:

Descriptive Question (DQ)

What will be described

Existing data

Additional data collection/ retrieval

Sampling/ disaggregation (equity)

Analysis

Timing

DQ x.x

 

 

 

 

 

 

DQ x.x

   

 

 

 

 

DQ x.x

   

 

 

 

 

             

In the second column make a clear statement about what will be described (e.g. types of/number of communication activities undertaken, or levels of knowledge on a specific topic). A theory of change can be very helpful here. (see here for more on Develop a Theory of Change).

In the third column list any existing or accessible data that could be used to answer that question, and assess their quality and relevance (see Determine and secure resources). There are often statistics available that can be used for C4D indicators. Other existing data that might be useful can come from previous research and evaluation studies, official records and publicly available statistics.

Finally, make selections for additional data collection/retrieval, sampling and analysis, and add these to the matrix. More information on options for these is below:

Sample 
Use measures, indicators or metrics 
Collect and/or retrieve data 
Manage data 
Combine qualitative and quantitative data 
Analyse data

Step 6: Decide how to answer causal questions and compile a matrix

The matrix for answering causal questions is slightly different. Often a matrix to answer causal questions will refer to descriptive data and will use analysis strategies that investigate causal relationships between variables.

Causal relationship

Comments

Strategy 1: Scope for a credible counterfactual?

Strategy 2: Scope for checking consistency of evidence?

Strategy 3: Scope for ruling out other alternative explanations?

Variable 1

Variable 2

       

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

First we need to identify the variables. Looking at each of your causal questions try to identify what the variables are. A very simple example might be:

Variable 1: Exposure to communication materials

Variable 2: Level of understanding of a specific topic

In a question about bottleneck and barriers, Variable 1 might be 'the presence of a barrier' and variable 2 the intermediate outcome. Your theory of change can be useful for clarifying variables (see Develop program theory or logic model). 

Use the comments column to note any important information e.g. the treatment of groups of variables, or use of answers from descriptive questions. 

There are three main strategies for answering questions about the causal relationships between variables. 

Compare results to a counterfactual (strategy 1) 
Check the results support causal attribution (strategy 2)
Investigate possible alternative explanations (strategy 3) 

Review these strategies, note whether or not a credible counterfactual will be feasible; and the list selected strategies for checking the consistency of evidence and for ruling out alternative explanations. It is recommended that you include multiple strategies of different kinds.

Examples:

Resources

Step 7: Decide how to answer evaluative questions and compile a matrix

The matrix for answering evaluative questions needs to show the processes you will use to select and apply criteria, standards and weighting. Each evaluative question in your list might need its own processes, or a group of evaluative questions might be answered using the same processes. 

What will be evaluated

Criteria

Standards

Synthesis/Weighting

Process for developing agreed standards, criteria and synthesis

 

 

 

 

 

Begin by making a statement about what will be evaluated (that is, what will be judged or valued). This might be particular activities, particular C4D approaches, particular sites, or particular outcomes. 

To judge and value something we can apply criteria, standards and then we would synthesise and weight those to come to conclusions.

Determine what 'success' looks like 

Synthesise data from a single evaluation 

Once you have made your selections, add these to the matrix and describe the processes to be used.

Step 8: Decide how to answer Action/ Predictive Questions and compile a matrix

Answering action questions in a credible way often requires a process of identifying and assessing options for action.  It is often useful to have a wider group of people involved in this process than simply an external evaluation team.

Action/Predictive Questions (AQ)

Process and participants for answering Action/Predictive Questions

AQ x.x

 

AQ x.x

 

AQ x.x

 

Begin by listing the Action/Predictive questions in the first column. To read about options for answering action/predictive questions.see:

Generalise findings (decide on actions) 

Once you have made your selections add these to the second column. Note: you may use the same process to answer all questions and in these cases you may simplify the matrix to indicate this.

Step 9: Develop a summary evaluation matrix with all planned data collection and analysis, including use of existing data 

The next step is to compile a matrix that summarises how you will answer each of the Key Questions and associated smaller questions. This is intended as a summary table; in most cases the more detailed matrixes for answering descriptive, causal, evaluative and action questions will remain in the final document. 

KQ

Data source / method / analysis 1

Data source / method / analysis 2

Data source / method / analysis 3

Data source / method / analysis 4

1 [add KQs]

1.1 [add sub questions]

 

 

 

 

1.2

 

 

 

 

1.3

 

 

 

 

1.4

 

 

 

 

2.

2.1

 

 

 

 

2.2

 

 

 

 

3.

3.1

 

 

 

 

3.2

 

 

 

 

3.3

 

 

 

 

4.

4.1

 

 

 

 

4.2

 

 

 

 

Add the Key Questions into the shaded rows, the associated sub or smaller questions underneath. You may need to add or remove rows. Add short descriptions of the data source or method in the corresponding boxes. Where possible, make note of timing, (i.e. baseline + every six months; baseline, midline, endline; ad hoc or as triggered etc.)

You can either rename the column headings (e.g. Existing data; Data Collection and Analysis methods; Causal Analysis methods; Stakeholder workshops), which makes it easy to see all the additional data collection in one column; or you could leave the headings as listed and fill in from left to right in the corresponding rows. This makes sense where there are a high number of different methods being used.

Step 10. List required tasks, studies, events, processes 

The final step is to extract a list of the tasks, studies, events and processes that are outlined in the matrix, and the associated methods (e.g. baseline studies, bottleneck analysis studies, evaluations, workshops, expert analysis or review processes etc). This list will later be used as the basis of a cost estimate and a workplan. If not already decided, see the task relating to Decide who will conduct the evaluation

Example

The Vietnam CO and RMIT University researchers followed these steps with counterparts to co-develop an M&E Framework and Plan for the VAC campaign. See the how they used these matrices to document their decisions here NationalProgramforChildProtectionCommunicationMEPlan.docx

Comments

There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.