Asking Key Informants to Attribute Causality

The program theory describes the way the program is supposed to function. It includes causal reasoning: ‘If we do this….we will get that outcome’. One way of testing this causal reasoning is to ask key informants - participants in a program and others who were involved in some way, or whose insights are likely to be relevant and credible.

Key questions to ask key informants include:

  • whether,
  • to what extent, and
  • why they attribute the outcomes to the program

Example

An evaluation of a leadership development programme  asked ‘Could recent promotions and career advances (e.g. appointments to chief executive (CE) and other senior positions) be attributed to the leadership development programme?’

The evaluator asked other key informants (previous programme participants and the participant Managers) abut the leadership development programme.

The evaluator asked alums:

  • How much (if any) they would give the Advanced Leadership Programme (ALP) for their successful appointment to a CE position at the time it happened? Why?
  • Where they think they’d be now if they hadn’t enrolled in the ALP?
  • Would they have headed for and been appointed to a CE position anyway?
  • Would it have taken longer to get there without the ALP, or about the same?
  •  Or, would they have taken a different career path?

The evaluator asked CEs/Managers about improvements in the Public Service’s and/or State Services’ pool of senior leaders. Questions included:

  • What senior-level selection decisions (Tier 2 or large agency Tier 3) have they been involved in over the past three years?
  • Have they noticed any changes in the quality, quantity and/or diversity of the applicants for each position?
  • Had any of the applicants for these senior positions participated in the Advanced Leadership Programme, the Executive Fellows Programme, or the Executive Master of Public Administration? Did these strengthen their applications in the eyes of the selection panel? Why [not]?
  • What credit (if any) would they give the programme for strengthening the public sector’s senior leadership pool? Why? Based on what? (Davidson, 2010)

Advice

Do you have any advice for CHOOSING this option?

Click on Contribute Content or Contact Us to share your experience of choosing this option.

Advice for USING this option (tips and traps)

  • Provide the evaluator with the original program theory - if there was one - or relevant documents to develop a retrospective program theory.
  • Assist the evaluator to identify key informants.
  • Develop a list of possible alternatives, based on evidence and information gathered throughout the evaluation process. Test the program theory and counterfactual with key informants.
  • Liaise with the evaluators to check the program theory is correct and the validity of possible alternative explanations and the counter-factual.

Resources

Guides

Sources

Updated: 14th January 2014 - 1:51am

Comments

There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.