Critical system heuristics
Critical System Heuristics (CSH) provides a framework of questions about a program including what is (and what ought to be) its purpose and its source of legitimacy and who are (and who ought to be) its intended beneficiaries.
CSH, as developed by Werner Ulrich and later elaborated upon in collaboration with Martin Reynolds, is an approach used to surface, elaborate, and critically consider boundary judgments, that is, the ways in which people/groups decide what is relevant to the system of interest (any situation of concern).
CSH is concerned not only with purposive evaluation, where the system or project has a predefined goal and the focus lies in evaluating the means of reaching it, but also more broadly with purposeful evaluation, where both the means and the ends become subjects of inquiry. CSH rests on the foundations of systems thinking and practical philosophy, both of which emphasize the 'infinite richness' of the real world. In this view, understandings of any situation are inherently incomplete and therefore based on the selective application of knowledge. By systematically questioning the sources of motivation, control, expertise, and legitimation in the system of interest, CSH allows users to make their boundary judgments explicit and defensible. The immediate goal of a CSH evaluation is to elaborate multiple perspectives on a given situation, but the broader aim is to share these perspectives and thereby cut down on actors 'talking past' each other by promoting mutual understanding.
Bases and background
Systems thinking is a broad field that employs 'systems' as conceptual tools which help to acknowledge and deal with the complexity of the real world (see: Churchman, 1971; Churchman, 1979). Evaluators are constantly confronted with overlapping networks of people, technology, geography, institutions, funding, and ideology which generate complex realities never entirely visible to any single person or from any single viewpoint. Each actor must develop their stances and actions by interpreting the flows of information available to them. Because no two people or viewpoints are exactly the same, this process will inevitably lead to a diversity of conclusions regarding the most appropriate appraisals and the best pathways forwards. Rather than focus on the stances and actions themselves, therefore, CSH zooms in on the process of interpretation. Rather than asking 'what do you see in this situation?' as in most evaluation approaches, CSH inquires more deeply, 'what are the underlying assumptions - values, power structures, knowledge bases, and moral stances - which affect how this situation is perceived by different concerned parties?'. Asking the twelve boundary questions (see: Table 1) fulfills the analytic purpose of CSH, which is to reveal how decisions about what is relevant to this interpretive process (boundary judgments) are being made.
The several strands of practical philosophy which underpin CSH emphasize (a) an orientation towards practical (rather than theoretical) goals, and (b) the role of the researcher as an agent of change, not simply a passive observer. Overlapping with the analytic purpose (of providing more information), therefore, CSH also intends to generate discussion, reflection, and critique. The boundary questions in Table 1 provide a common rubric designed to enable mutual understanding and constructive dialogue. Engaging debate about the issues exposed during the evaluation fulfills a practical purpose by opening up new ways to improve communication and hence the overall wellbeing of those involved.
Finally, by intentionally setting all actors' perspectives on equal footing regardless of their expertise or technical competence (sometimes referred to as a 'Socratic' professional stance), CSH can have an emancipatory purpose, amplifying otherwise unheard voices.
Conversation and learning from an evaluation can be severely hindered by debates that stem from differing boundary judgments and situational framings, but many approaches lack a means of openly and explicitly discussing the underlying sources of these judgments. CSH opens doorways to profound investigation of the different worldviews involved in a project.
The CSH toolbox is composed of twelve 'boundary questions' designed to outline and provoke thought about boundary judgments that determine situational framings. Each question is considered in two modes: an ideal mode (what 'should' be), and a descriptive mode (what 'is'), making twenty-four questions in total.
Checklist of critically heuristic boundary questions
For systematic boundary critique, each question needs to be answered both in the `is’ and in the `ought’ mode. Differences between `is’ and `ought’ answers point to unresolved boundary issues. There are no definitive answers, in that boundary judgements may always be reconsidered. By means of systematic alteration of boundary judgements, it is possible to unfold the partiality (selectivity) of an assumed system of concern from multiple perspectives, so that both its empirical content (assumptions of fact) and its normative content (value assumptions) can be identified and can be evaluated without any illusion of objectivity.
Sources of Motivation
- Who is (ought to be) the client? That is, whose interests are (should be) served?
- What is (ought to be) the purpose? That is, what are (should be) the consequences?
- What is (ought to be) the measure of improvement? That is, how can (should) we determine that the consequences, taken together, constitute an improvement?
Sources of Power
- Who is (ought to be) the decision-maker? That is, who is (should be) in a position to change the measure of improvement?
- What resources are (ought to be) controlled by the decision-maker? That is, what conditions of success can (should) those involved control?
- What conditions are (ought to be) part of the decision environment? That is, what conditions can (should) the decision-maker not control (e.g. from the viewpoint of those not involved)?
Sources of Knowledge
- Who is (ought to be) considered a professional? That is, who is (should be) involved as an expert, e.g. as a researcher, planner or consultant?
- What expertise is (ought to be) consulted? That is, what counts (should count) as relevant knowledge?
- What or who is (ought to be) assumed to be the guarantor of success? That is, where do (should) those involved seek some guarantee that improvement will be achieved - for example, consensus among experts, the involvement of stakeholders, the experience and intuition of those involved, political support?
Sources of Legitimation
- Who is (ought to be) witness to the interests of those affected but not involved? That is, who is (should be) treated as a legitimate stakeholder, and who argues (should argue) the case of those stakeholders who cannot speak for themselves, including future generations and non-human nature?
- What secures (ought to secure) the emancipation of those affected from the premises and promises of those involved? That is, where does (should) legitimacy lie?
- What worldview is (ought to be) determining? That is, what different visions of `improvement’ are (ought to be) considered, and how are they (should they be) reconciled?
(Ulrich, W., 2000)
CSH is not a 'prescribed methodology' with step-by-step directions of standard best practices, but instead a framework to encourage reflection. The technique of asking the boundary questions, their order, wording, and the manner in which the answers to them are consolidated will vary. Interested parties can refer to Reynolds (2007) for practitioner guidelines to application based on experience (p. 6-8), and Ulrich and Reynolds (2010) for a 'standard sequence' for beginners (p. 258-259), but the final form will remain eternally open to critique and modification.
Advice for USING this Approach (tips and traps)
CSH offers a unique approach with powerful tools, but its analytic, practical, and emancipatory power relies heavily on the specifics of its application. Discovering and critiquing boundary judgments can be a delicate process, depending on the context. It is important to ensure that participants in the process fully comprehend the end goal of the exercise and thus understand the level of questioning and the types of responses that may emerge. A key factor in developing this understanding is the language and phrasing used to explain CSH concepts. CSH has a significant corpus of jargon and vocabulary which require reformulation and rephrasing in each circumstance. An open, flexible professional culture with high levels of ex-ante 'buy-in' will greatly ease application, though these circumstances may not be where CSH has the most impact or affects the greatest change. Finally, the most significant 'results' of a CSH evaluation (greater awareness of selectivity of claims, enhanced communication, increased tolerance) are generally intangible and inaccessible to direct measurement. Disseminating these results to audiences outside of the project is an exercise of more than factual explanation which includes tactful and persuasive communication.
Impact of CSH approach on the conduct of evaluation
Churchman, C. W. (1971)The Design of Inquiring Systems: Basic Concepts of Systems and Organizations. New York: Delta/Dell Publishing.
Churchman, C. W. (1979) The Systems Approach and its Enemies. New York: Basic Books.
Jeppesen, Sara Lise. (2011) “Exploring an Explicit Use of the Concept of Sustainability in Transport Planning.” Systemic Practice and Action Research 24 (August 26): 133-146. https://link.springer.com/article/10.1007/s11213-010-9180-6 (accessed April 25, 2012).
Reynolds, M (2007). “Evaluation based on critical systems heuristics.” In Using Systems Concepts in Evaluation: An Expert Anthology, edited by Bob Williams and Iraj Imam, 101-122. Point Reyes, CA: EdgePress. Available online at: http://oro.open.ac.uk/3464/
Romero, Roman Vega (2002). “The Relevance of Ulrich’s Critical Systems Heuristic for Social Planning Evaluation.” Cuadernos de Administracion 15, no. 24: 41-71.
Ulrich W (2005). A brief introduction to critical systems heuristics (CSH). ECOSENSUS project web site, Open University, Milton Keynes, UK, 14 October 2005. Available online at http://kmi.open.ac.uk/projects/ecosensus/publications/ulrich_csh_intro.pdf and http://wulrich.com/downloads/ulrich_2005f.pdf
Ulrich, W (1983) Critical Heuristics of Social Planning: A New Approach to Practical Philosophy. Bern, Switzerland and Stuttgart, Germany: Haupt. Paperback reprint edition, 1994, Chichester, UK, and New York: Wiley (same pagination)
Ulrich, W (1987). Critical heuristics of social systems design. European Journal of Operational Research 31, no. 3: 276-283.
Ulrich, W. (2000). Reflective practice in the civil society: the contribution of critically systemic thinking. Reflective Practice 1, no. 2: 247-268.
Ulrich, Werner, and Reynolds M, (2010) “Critical Systems Heuristics.” In Systems Approaches to Managing Change: A Practical Guide. Edited by Martin Reynolds and Sue Holwell:.243-292. London: Springer. https://www.researchgate.net/publication/43642939_Critical_Systems_Heuristics (accessed March 4, 2012).
'Critical system heuristics' is referenced in: