A method for testing causal reasoning by asking key informants.
The program theory describes the way the program is supposed to function. It includes causal reasoning: ‘If we do this….we will get that outcome’. One way of testing this causal reasoning is to ask key informants - participants in a program and others who were involved in some way, or whose insights are likely to be relevant and credible.
Key questions to ask key informants include:
- to what extent, and
- why they attribute the outcomes to the program
An evaluation of a leadership development programme asked ‘Could recent promotions and career advances (e.g. appointments to chief executive (CE) and other senior positions) be attributed to the leadership development programme?’
The evaluator asked other key informants (previous programme participants and the participant Managers) about the leadership development programme.
The evaluator asked alums:
- How much (if any) they would give the Advanced Leadership Programme (ALP) for their successful appointment to a CE position at the time it happened? Why?
- Where do they think they’d be now if they hadn’t enrolled in the ALP?
- Would they have headed for and been appointed to a CE position anyway?
- Would it have taken longer to get there without the ALP, or about the same?
- Or, would they have taken a different career path?
The evaluator asked CEs/Managers about improvements in the Public Service’s and/or State Services’ pool of senior leaders. Questions included:
- What senior-level selection decisions (Tier 2 or large agency Tier 3) have they been involved in over the past three years?
- Have they noticed any changes in the quality, quantity and/or diversity of the applicants for each position?
- Had any of the applicants for these senior positions participated in the Advanced Leadership Programme, the Executive Fellows Programme, or the Executive Master of Public Administration? Did these strengthen their applications in the eyes of the selection panel? Why [not]?
- What credit (if any) would they give the programme for strengthening the public sector’s senior leadership pool? Why? Based on what? (Davidson, 2010)
Advice for using this method
- Provide the evaluator with the original program theory - if there was one - or relevant documents to develop a retrospective program theory
- Assist the evaluator to identify key informants
- Develop a list of possible alternatives, based on evidence and information gathered throughout the evaluation process. Test the program theory and counterfactual with key informants
- Liaise with the evaluators to check the program theory is correct and the validity of possible alternative explanations and the counter-factual
Davidson, J. (2010, May). Outcomes, impacts & causal attribution. Presented at ANZEA regional symposium, Auckland, New Zealand. Retrieved from http://realevaluation.com/pres/Causation-anzea2010.pdf
'Asking key informants to attribute causality' is referenced in:
- Rainbow Framework :