Critical system heuristics

Available languages
Contributing author
William Faulkner

Critical System Heuristics (CSH) provides a framework of questions about a program including what is (and what ought to be) its purpose and its source of legitimacy and who are (and who ought to be) its intended beneficiaries.

CSH, as developed by Werner Ulrich and later elaborated upon in collaboration with Martin Reynolds, is an approach used to surface, elaborate, and critically consider boundary judgments, that is, the ways in which people/groups decide what is relevant to the system of interest (any situation of concern). 

CSH is concerned not only with purposive evaluation, where the system or project has a predefined goal and the focus lies in evaluating the means of reaching it, but also more broadly with purposeful evaluation, where both the means and the ends become subjects of inquiry. CSH rests on the foundations of systems thinking and practical philosophy, both of which emphasize the 'infinite richness' of the real world. In this view, understandings of any situation are inherently incomplete and therefore based on the selective application of knowledge. By systematically questioning the sources of motivation, control, expertise, and legitimation in the system of interest, CSH allows users to make their boundary judgments explicit and defensible. The immediate goal of a CSH evaluation is to elaborate multiple perspectives on a given situation, but the broader aim is to share these perspectives and thereby cut down on actors 'talking past' each other by promoting mutual understanding.

Bases and background

Systems thinking is a broad field that employs 'systems' as conceptual tools which help to acknowledge and deal with the complexity of the real world (see: Churchman, 1971; Churchman, 1979). Evaluators are constantly confronted with overlapping networks of people, technology, geography, institutions, funding, and ideology which generate complex realities never entirely visible to any single person or from any single viewpoint. Each actor must develop their stances and actions by interpreting the flows of information available to them. Because no two people or viewpoints are exactly the same, this process will inevitably lead to a diversity of conclusions regarding the most appropriate appraisals and the best pathways forwards. Rather than focus on the stances and actions themselves, therefore, CSH zooms in on the process of interpretation. Rather than asking 'what do you see in this situation?' as in most evaluation approaches, CSH inquires more deeply, 'what are the underlying assumptions - values, power structures, knowledge bases, and moral stances - which affect how this situation is perceived by different concerned parties?'. Asking the twelve boundary questions (see: Table 1) fulfills the analytic purpose of CSH, which is to reveal how decisions about what is relevant to this interpretive process (boundary judgments) are being made.

The several strands of practical philosophy which underpin CSH emphasize (a) an orientation towards practical (rather than theoretical) goals, and (b) the role of the researcher as an agent of change, not simply a passive observer. Overlapping with the analytic purpose (of providing more information), therefore, CSH also intends to generate discussion, reflection, and critique. The boundary questions in Table 1 provide a common rubric designed to enable mutual understanding and constructive dialogue. Engaging debate about the issues exposed during the evaluation fulfills a practical purpose by opening up new ways to improve communication and hence the overall wellbeing of those involved.

Finally, by intentionally setting all actors' perspectives on equal footing regardless of their expertise or technical competence (sometimes referred to as a 'Socratic' professional stance), CSH can have an emancipatory purpose, amplifying otherwise unheard voices.

Conversation and learning from an evaluation can be severely hindered by debates that stem from differing boundary judgments and situational framings, but many approaches lack a means of openly and explicitly discussing the underlying sources of these judgments. CSH opens doorways to profound investigation of the different worldviews involved in a project.

Application

The CSH toolbox is composed of twelve 'boundary questions' designed to outline and provoke thought about boundary judgments that determine situational framings. Each question is considered in two modes: an ideal mode (what 'should' be), and a descriptive mode (what 'is'), making twenty-four questions in total.

Checklist of critically heuristic boundary questions

For systematic boundary critique, each question needs to be answered both in the `is’ and in the `ought’ mode. Differences between `is’ and `ought’ answers point to unresolved boundary issues. There are no definitive answers, in that boundary judgements may always be reconsidered. By means of systematic alteration of boundary judgements, it is possible to unfold the partiality (selectivity) of an assumed system of concern from multiple perspectives, so that both its empirical content (assumptions of fact) and its normative content (value assumptions) can be identified and can be evaluated without any illusion of objectivity.

Sources of Motivation

  1. Who is (ought to be) the client? That is, whose interests are (should be) served?
  2. What is (ought to be) the purpose? That is, what are (should be) the consequences?
  3. What is (ought to be) the measure of improvement? That is, how can (should) we determine that the consequences, taken together, constitute an improvement?

Sources of Power

  1. Who is (ought to be) the decision-maker? That is, who is (should be) in a position to change the measure of improvement?
  2. What resources are (ought to be) controlled by the decision-maker? That is, what conditions of success can (should) those involved control?
  3. What conditions are (ought to be) part of the decision environment? That is, what conditions can (should) the decision-maker not control (e.g. from the viewpoint of those not involved)?

Sources of Knowledge

  1. Who is (ought to be) considered a professional? That is, who is (should be) involved as an expert, e.g. as a researcher, planner or consultant?
  2. What expertise is (ought to be) consulted? That is, what counts (should count) as relevant knowledge?
  3. What or who is (ought to be) assumed to be the guarantor of success? That is, where do (should) those involved seek some guarantee that improvement will be achieved - for example, consensus among experts, the involvement of stakeholders, the experience and intuition of those involved, political support?

Sources of Legitimation

  1. Who is (ought to be) witness to the interests of those affected but not involved? That is, who is (should be) treated as a legitimate stakeholder, and who argues (should argue) the case of those stakeholders who cannot speak for themselves, including future generations and non-human nature?
  2. What secures (ought to secure) the emancipation of those affected from the premises and promises of those involved? That is, where does (should) legitimacy lie?
  3. What worldview is (ought to be) determining? That is, what different visions of `improvement’ are (ought to be) considered, and how are they (should they be) reconciled?

(Ulrich, W., 2000)

CSH is not a 'prescribed methodology' with step-by-step directions of standard best practices, but instead a framework to encourage reflection. The technique of asking the boundary questions, their order, wording, and the manner in which the answers to them are consolidated will vary. Interested parties can refer to Reynolds (2007) for practitioner guidelines to application based on experience (p. 6-8), and Ulrich and Reynolds (2010) for a 'standard sequence' for beginners (p. 258-259), but the final form will remain eternally open to critique and modification. 

Advice

Advice for USING this Approach (tips and traps)

CSH offers a unique approach with powerful tools, but its analytic, practical, and emancipatory power relies heavily on the specifics of its application. Discovering and critiquing boundary judgments can be a delicate process, depending on the context. It is important to ensure that participants in the process fully comprehend the end goal of the exercise and thus understand the level of questioning and the types of responses that may emerge. A key factor in developing this understanding is the language and phrasing used to explain CSH concepts. CSH has a significant corpus of jargon and vocabulary which require reformulation and rephrasing in each circumstance. An open, flexible professional culture with high levels of ex-ante 'buy-in' will greatly ease application, though these circumstances may not be where CSH has the most impact or affects the greatest change. Finally, the most significant 'results' of a CSH evaluation (greater awareness of selectivity of claims, enhanced communication, increased tolerance) are generally intangible and inaccessible to direct measurement. Disseminating these results to audiences outside of the project is an exercise of more than factual explanation which includes tactful and persuasive communication. 

Impact of CSH approach on the conduct of evaluation

  Component
and Tasks
Approach Response
& Methods
  MANAGE Beyond simply deciding who controls/conducts the evaluation and who the stakeholders are, CSH encourages those involved in the process to consider how and why such decisions are made. An actual CSH project will of course require the allocation of resources (money, time) before the process can even begin, and these processes will likely come before the methodology is chosen. Those who choose CSH, however, can expect to profoundly reexamine these choices and their underpinnings after-the-fact.
  DEFINE In principle, CSH does not define any precise process to engage stakeholders, guide data collection, or test causal linkages. Instead, CSH remains one step removed from these questions, preferring instead to focus on the underlying factors which define the ways in which they are asked and answered by different actors within the situation of concern. Examining the case studies in literature (some of which are cited in the "More to Explore" section) provides information about specific instances.
 

ENGAGE &
FRAME

The primary focus of CSH is on how different actors in a project / intervention engage and frame the problems, strategies, solutions, and outcomes. CSH aims to provide a common scaffolding which allows different perspectives / framings to engage meaningfully with one another.
  DESCRIBE Data collection for CSH has, until now, remained qualitative, including different forms of interviews, surveys, and focus groups. As previously mentioned, CSH views the researcher(s) not just as an observer(s) or collector(s) of data, but also as an actor(s) within the system whose choices will influence outcomes. A CSH evaluator should reflexively consider decisions regarding both the collection and analysis of data.
  UNDERSTAND
CAUSES
Attribution is not a central feature of CSH, and the approach makes no claim to be able to definitively establish causal linkages. CSH does, however, stress the importance of surfacing potential sources of influence, which allows the investigation of how different framings and viewpoints might influence the processes and decisions surrounding attribution, a valuable aspect left out of many other methodologies.
  SYNTHESISE
AND VALUE
The boundary questions systematically probe boundary judgements, permitting both the analytic (revealing more information) and synthetic (weaving multiple perspectives together) processes to progress together. CSH makes no claim to provide a complete or holistic view of the system of interest but instead seeks to reveal the inevitable selectivity of all conceptions of this situation and facilitate dialogue between dissimilar views. Similar to data collection and description, the synthesis of information by the researcher(s) should be accomplished through a reflective and reflexive process, with full awareness of the subjectivities inherently involved.
  REPORT AND
SUPPORT USE
In some ways, the targeted 'users' of a CSH evaluation are not those reading the report(s) or publication(s) which come later, but instead those involved in the process itself. 'Finding' are less central than the essential act of asking and answering the boundary questions with a number of different actors involved in the situation of concern.

 

Resources

Overview

Guides

Examples

Websites

Churchman, C. W. (1971)The Design of Inquiring Systems: Basic Concepts of Systems and Organizations. New York: Delta/Dell Publishing.

Churchman, C. W. (1979) The Systems Approach and its Enemies. New York: Basic Books.

Jeppesen, Sara Lise. (2011) “Exploring an Explicit Use of the Concept of Sustainability in Transport Planning.” Systemic Practice and Action Research 24 (August 26): 133-146. https://link.springer.com/article/10.1007/s11213-010-9180-6 (accessed April 25, 2012).

Reynolds, M (2007). “Evaluation based on critical systems heuristics.” In Using Systems Concepts in Evaluation: An Expert Anthology, edited by Bob Williams and Iraj Imam, 101-122. Point Reyes, CA: EdgePress. Available online at: http://oro.open.ac.uk/3464/

Romero, Roman Vega (2002). “The Relevance of Ulrich’s Critical Systems Heuristic for Social Planning Evaluation.” Cuadernos de Administracion 15, no. 24: 41-71.

Ulrich W (2005). A brief introduction to critical systems heuristics (CSH). ECOSENSUS project web site, Open University, Milton Keynes, UK, 14 October 2005. Available online at http://kmi.open.ac.uk/projects/ecosensus/publications/ulrich_csh_intro.pdf  and  http://wulrich.com/downloads/ulrich_2005f.pdf

Ulrich, W (1983) Critical Heuristics of Social Planning: A New Approach to Practical Philosophy. Bern, Switzerland and Stuttgart, Germany: Haupt. Paperback reprint edition, 1994, Chichester, UK, and New York: Wiley (same pagination)

Ulrich, W (1987). Critical heuristics of social systems design. European Journal of Operational Research 31, no. 3: 276-283.

Ulrich, W. (2000). Reflective practice in the civil society: the contribution of critically systemic thinking. Reflective Practice 1, no. 2: 247-268.

Ulrich, Werner, and Reynolds M, (2010) “Critical Systems Heuristics.” In Systems Approaches to Managing Change: A Practical Guide. Edited by Martin Reynolds and Sue Holwell:.243-292. London: Springer. https://www.researchgate.net/publication/43642939_Critical_Systems_Heuristics (accessed March 4, 2012).

Last updated:

Expand to view all resources related to 'Critical system heuristics'

'Critical system heuristics' is referenced in: