What are some methods and processes to help stakeholders articulate how they think a program works? (AES17 co-creation challenge #1)

Co-creation%20challenge.png

The material from BetterEvaluation comes from a combination of curating existing material and co-creating new material.  This blog is part of an ongoing series about material that we have co-created with BetterEvaluation users.

It shares material that was jointly developed through a challenge process at the 2017 Australasian Evaluation Society conference in Canberra in September.

Thank you to everyone who contributed to this co-creation challenge!

It is often helpful to use a program theory (or theory of change) to guide an evaluation.  On the BetterEvaluation site, we list 8 processes you can use (ideally in combination) to develop or revise a program theory:
  • Articulating mental models: talking individually or in groups with key informants (including programme planners, service implementors and clients) about how they understand an intervention works.
  • Backcasting: working backward from a desirable future, to the present in order to determine the feasibility of the idea or project.
  • Five Whys: asking questions in order to examine the cause-and-effect relationships that create underlying problems.
  • Generic action theories
  • Generic change theories - common theories about how change comes about, that can be used in various interventions (for example, deterrence)
  • Group model building: building a logic model in a group, often using sticky notes.
  • Previous research and evaluation: using the findings from evaluation and research studies that were previously conducted on the same or closely related areas.
  • SWOT Analysis: reflecting on and assessing the Strengths, Weaknesses,Opportunities and Threats of a particular strategy in order to discover how it can best be implemented.

 

One of the co-creation challenges we conducted explored ways of articulating participants’ mental models. Sometimes direct questions can be useful, and sometimes more creative processes, involving drawing and expressions of feeling, might help people work through these discussions. Here are eight suggestions for techniques that came out of our co-creation challenge can be used with key informants, either individually or in a group:

  1. Facilitate reflection - Support participants to engage in reflective practice and honest self-reflection
  2. Open up a space for honest conversations about the program
  3. Look at an existing logic model and ask stakeholders to comment on what is wrong with it
  4. Ask for a 'peak experience description', a succinct and coherent description of a program, project or policy when it is operating at its best. This can then be used to develop a logic model (program theory) and to investigate how often it operates like this and how this can happen more often.
  5. Refer to original program documents or other existing data and ask if this is what happened
  6. Develop Storyboards - see this newly added resource for some guidance
  7. Draw Mudmaps (accompanied by a narrative) - people use the term 'mudmap' in two different ways: 
    1. The first typically means a simple mind map, as discussed briefly in Les Robinson's CoCreate: A Facilitator's Guide to Collaborative Planning (page 164) - you can try using the 'Shower Curtain' technique to help with this
    2. The second refers to actual maps, quickly sketched to illustrate an area and key facts (which Rosemary Cairn discusses using for this very purpose in a BetterEvaluation blog)
  8. Use projective techniques: including role play (see our projective techniques and our theatre option page for more information), where you might invite stakeholders to pretend they're explaining as if to a journalist or doing a (short) hand over to incoming project staff

How else can you do this? What's worked well for you in the past? What hasn't worked so well?

'What are some methods and processes to help stakeholders articulate how they think a program works? (AES17 co-creation challenge #1)' is referenced in: