Evaluating capacity development
Unlike programs supporting health, livelihoods, and other impact areas, capacity development does not have stand-alone outcomes. Instead, capacity development supports a diverse set of goals in different sectors, at different levels, through different activities. Capacity development consequently presents multiple evaluation opportunities and challenges for practitioners.
Some important issues surrounding capacity development and evaluation are:
- Differing interpretations of what ‘capacity’ means - capacity of whom to do what and whether it is ‘adaptive capacity,’ which allows local organizations to remain resilient to change.
- Defining long-term outcomes of CD rather than just short-term outputs. For example, what kinds of outcomes can we achieve by helping an organization build a strategic plan?
- Establishing causal links between particular CD activities, improvements in organizational performance, and the target development impact.
- Articulating a clear and logical theory of change, accompanied by realistic, need-driven targets rather than donor-driven activities and programs.
- Choosing appropriate evaluation tools on a case-by-case basis.
In addition to the specific information on evaluating capacity development which is presented below, the following guidance can be useful for evaluators and for those managing an evaluation and overseeing the development of a monitoring and evaluation system or an evaluation plan:
The Manager's Guide to Evaluation is an interactive guide goes through the steps of planning and managing an evaluation, including overseeing design and choice of methods and processes. It also includes the GeneraTOR - which supports development of a terms of reference for an evaluation.
The Rainbow Framework provides a brief descriptions of methods, processes and approaches that can be used in evaluation, and organises them according to the different tasks involved in an evaluation or a monitoring system.
Definitions of Capacity Development
The definition of Capacity Development, also referred to as Capacity Building, differs by source. Here are just a few examples:
- The United Nations Development Program (UNDP) defines CD as “the process through which individuals, organisations, and societies obtain, strengthen, and maintain the capabilities to set and achieve their own development objectives over time.”
- The Organisation for Economic Coordination and Development (OECD) defines CD as “the process whereby people, organisations and society as a whole unleash, strengthen, create, adapt and maintain capacity over time.”
- Global Affairs Canada (formerly Canadian International Development Agency, CIDA) defines CD as “the activities, approaches, strategies, and methodologies which help organizations, groups and individuals to improve their performance, generate development benefits and achieve their objectives.”
- PACT defines CD as “a continuous process that fosters the abilities and agency of individuals, organizations, and communities to overcome challenges and contribute towards positive social change. Though often developed in response to an immediate and specific issue, capacities are adaptable to future opportunities and challenges.”
A capacity development framework
The following is based on Pact's approach to Capacity Development, a comprehensive CD framework.
Pact’s CD framework breaks capacity down into three parts, which together form the universe of capacity development interventions.
1. Whose capacity is being developed?
The first part of the capacity development framework describes the range of recipients for capacity development support. This includes individuals and organizations, networks and systems, and complex ecosystems of diverse actors engaged in development processes in multiple ways and with different perspectives on social change.
Traditionally, capacity development efforts have focused at the individual and organizational levels. Recently, however, capacity development practitioners are increasingly recognizing the importance of working at the system and network levels in order to bring multiple competencies to work on complex challenges.
2. How is capacity being developed?
The second part of the capacity development framework describes the range of methodologies for capacity development interventions. Capacity development interventions vary from expert-driven consultancy services and trainings to participant-driven peer-to-peer exchanges.
The best capacity development programs employ a wide range of intervention types. The interventions are chosen based on a deep understanding of an issue’s underlying causes and tailored to the local context. Traditionally, capacity development interventions have over-relied on big ticket events such as trainings and workshops.
3. What capacities are being developed?
The third part of Pact’s capacity development framework describes the range of capacities that we seek to develop. These include:
- Technical capacities related to the impact area of any given intervention.
- Operational capacities needed to accomplish individual tasks.
- Systemic capacities to ensure that key functions are performed continuously over time.
- Adaptive capacities to respond to changes in their operating environment.
- Influencing capacities enabling an entity to bring about change within its environment.
Any or all of these capacities may be necessary within a given program or country context.
Other frameworks for evaluating capacity development
There are many other ways and frameworks to look at capacity development from various angles, and the choice of the framework heavily depends on the scope and type of the intervention:
- By capacity, such as program implementation, management, or organizational structure and policies.
- In the paper 'The Design and Use of Capacity Development Indicators', Peter Morgan from CIDA breaks capacity development indicators into the following areas:
- The ‘product,’ meaning capacities actually developed
- ‘Performance,’ or substantive program outcomes
- ‘Permanence,’ meaning sustainability of the capacities produced
- UNDP’s framework of capacity development results offers three levels of measurement:
- Impact, or change in people’s well-being
- Outcome, or change in institutional performance, stability, and adaptability
- Output, or the CD service provided
- The World Bank Institute’s Capacity Development and Results Framework (CDRF) connects capacity development results to broader development goals via the following framework: Activities → Learning outcomes → Change process driven by change agents → Local ownership, effectiveness and efficiency of resource use → Capacity for achieving a development goal. The Australian Department of Foreign Affairs and Trade Development Program (DFAT) applies Kirkpatrick’s model for evaluating training to evaluating capacity development outcomes (PDF, 94KB) on the following continuum:
- Reaction to inputs → Learning → Behaviour Change → Organizational change at the individual, entity, network and environmental levels.
- In “Monitoring and Evaluating Capacity Building: Is it really that difficult?,” The International NGO Training and Research Centre (INTRAC) distinguishes between the M&E of capacity and the M&E of capacity building. The former is concerned with assessing the changing capacity of an organisation (or individual, or society). The latter is concerned both with the quality and relevance of capacity-building efforts, and with the immediate changes occurring.
- The elements of the International Development Research Center (IDRC)/Universalia’s Organizational Assessment Framework are organizational performance, organizational environment, organizational motivation, and organizational capacity.
- The European Centre for Development Policy Management (ECDPM)’s research entitled “Capacity, Change and Performance” divides capacity into the following capabilities:
- The capability to act and self-organize (Vision, Volition, Strategy, Agency)
- The capability to generate development results (Programmatic Outcomes, Achievement of Mission)
- The capability to relate (Networking, Collaboration, Advocacy Mobilizing Resources, Relevance)
- The capability to adapt and self-renew (Learning, Change Management)
- The capability to achieve coherence (Innovation, Flexibility, Resilience)
- The MEASURE CD Framework for Health Sector looks at health sector-specific capacity outcome levels: system, organization, health personnel, and individual/community. Each of these levels is broken down by inputs, processes, outputs, and outcomes.
The above list is not exhaustive and many more approaches to capacity development exist.
Describe capacity development
Due to some of the challenges outlined in the “Definition” section above, most capacity development measurements today still rely on anecdotal evidence of change and assess effectiveness through outputs like numbers of people trained or strategic plans developed. Many international, regional, and national institutions have designed an Organizational Capacity Assessment tool to measure capacity development to address this issue.
However, such tools are typically limited to short-term results of concrete activities (for example, setting up a new M&E system). The tools also rarely take into account the influence of the external environment, i.e. change in political, economic, legislative, cultural, and social spheres, on the entity whose capacity is being developed. Such assessments cannot demonstrate capacity strengthening outcomes: changes in how the organization behaves and functions, and consequently how capacity development impacts the lives of its targeted beneficiaries.
In order to understand the longer-term influence of capacity development on an entity, practitioners need to be able to see whether the entity has improved its performance over time. Pact’s theory of change (see graphic) connects organizational change at the output level (change in the systems, skills, and policies of entities) to changes at the impact level (influence at the community level) through measuring growth in organizational performance. Pact has developed the Organizational Performance Index (OPI) to measure this growth in each individual partner entity, and to analyze trends by country, region, and around the world.
A good capacity development evaluation literature review can be found in the materials of the European Commission’s Directorate General for Development and Cooperation.
Examples of new approaches in CD evaluation
There are a number of other organizations exploring new approaches to measuring and analyzing the outcomes of capacity development interventions:
'Evaluating capacity development' is referenced in: