Week 12: Evaluation innovation in transparency and accountability

IGuijt's picture 21st March 2014 by IGuijt

Innovation is a relative concept. It is about new practice … for the topic and person or group in question. The context-specific nature of what constitutes ‘an innovation’ became clear during a recent event around global transparency and accountability efforts.

Last week the T/A Initiative (see Box)  hosted the second gathering of its community of practice. Over 70 activists, researchers, international NGOs, and donors converged in Jakarta to share ideas and challenges about how to evaluate and learn about ‘Transparency and Accountability’.

The Transparency and Accountability Initiative

The Transparency and Accountability Initiative is a donor collaborative working to expand the impact and scale of transparency and accountability interventions. Current members include the Hewlett Foundation, the Ford Foundation, the Open Society Foundations, Omidyar Network, DFID, HIVOS, the International Budget Partnership, and Revenue Watch.

T/A initiatives leverage information to empower citizens to hold the state or private sector to account. But they differ greatly in strategies, scale, sector and stakeholders. Examples include working with local communities and citizen groups to promote budget transparency, supporting anti-corruption efforts through research and information, and using evidence on health system performance for social mobilization and advocacy. These initiatives make information available about expected and actual resource use and service performance and support its active use by citizens.

The inherently political nature of T/A work means that evaluation challenges are similar to those of other complex change processes (Rogers 2008). Many different players are involved in the many transitions from access to information, to how people perceive it, their ability to use it, the willingness of receivers to change and finally better performing services, such as schools or hospitals. These lengthy causal pathways are embedded in strong power inequalities. Changing such relations of power is difficult, dangerous and has no guarantees. Formal and informal channels need to be used to build trust with those who want change and those who need to be changed. Change is slow and subject to possible backsliding (Fölscher 2012).

Over the past ten years, the surge of donor interest in the aims of the T/A sector has been accompanied by a growing interest to see ‘proof’ of effectiveness. Efforts were ramped up, for example by the International Budget Partnership to document a series of impact studies and share insights on planning for impact.  Case studies have documented the positive changes in budget institutions, policies, allocations and the quality of expenditure mainly in middle-income countries, with encouraging signs in low- income and challenging environments.

During the Jakarta event, I was struck by three themes in relation to evaluation of the T/A sector.

First, many participants expressed the need for more investment in and capacity for refining their ‘theories of change’. Participants shared several cases of disappointment about unmet expectations and initial strategies that proved inadequate. The consensus seemed to be that prior – or as part of – setting out how to monitor and evaluate T/A efforts, it is critical to be clear about the assumptions between more transparency and more accountability. For example, many discussions recognized that transparency of information is just one strategy to improve social accountability. Participants queried whether perhaps it was better to start with social accountability and then figure out which strategy was best for that context, which might – or might not –information transparency. The T/A sector can learn much from ToC work elsewhere, such as by Hivos, from which to draw inspiration.

Second, more refined theories of change links directly to the challenge of defining success. Is it only when legislation has changed or when hospital patients have better quality care that success can be claimed? Or do the intermediate steps also count, such as encouraging youth to consider pushing their local councilors to publish budgets? Participants were clear about the need to document intermediate steps as progress towards success (and to understand the way change was evolving) while also recognizing the need to be humble. Particularly important was the observation that success depends on where you stand. One example was given of an international NGO who claimed success by gaining press headlines for local action and being kicked out of the region, while staff members of the local organisation were left with empty hands, socially ostracized and receiving threats. 

Third, the definition of success is shaped by often-difficult tensions between what participants called the ‘projectising’ of political change and the long term nature of T/A efforts. Some presenters described how their work was at risk of being viewed as bounded, technocratic projects, when activities had many different facets and scales and were aiming to shift power relations. Donor requirements play an oftentimes unhelpful and unhealthy pressure on T/A efforts to bound the work in terms of time and result guarantees, with limited flexibility for grabbing opportunities as they arise.  

These three themes were discussed intensely by participants at the Jakarta event. While new for the T/A sector, they echo questions many have pondered for the past decades in working with local organisations to develop their M&E systems and learning approaches. How do we improve our theories of change and then use this to develop our M&E systems? What best methods are there for understanding impact? How can more narrative approaches, such as outcome mapping, process tracing and citizen ethnography help us? How do we learn to learn? What is the link between M&E, research and learning? What cases of strong organisations can we learn from?

My takeaway from the TALearn event was that the challenge of innovation for the T/A sector is one of efficient learning about monitoring, evaluation, impact and learning that builds on advances in different sectors, adapting them to the specific questions and processes of social accountability.

Read more

Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions

This article by Patricia J. Rogers highlights the way program theory can be used in order to evaluate complicated or complex elements of programs.

Read more

Develop Programme Theory/Logic Model

A logic model demonstrates how an intervention (a project, a program, a policy, a strategy) is understood to contribute to possible or actual impacts. It can include positive impacts (which are beneficial) and negative impacts (which are detrimental).

Read more

Complexity

An issue of increasing interest in evaluation, especially development evaluation, is whether and how we might apply ideas and methods from complexity science to evaluation.

Read more

Community Scorecard Approach for Performance Assessment - ProNet North's Experience

This WaterAid Ghana Briefing Paper, outlines ProNet North’s experience of using the Community Scorecard Approach to assess performance.

Read more

Protocol - Under what circumstances does enhancing community accountability and empowerment improve education outcomes, particularly for the poor?

This paper, written by Dr Gill Westhorp, Bill Walker and Professor Patricia Rogers for University of London EPPI-Centre, provides a protocol for a realist synthesis on community accountability and empowerment interventions in education in middle and low income countries and the circumstances in which these improve outcomes, particularly for the poor.

Read more

Image: "Focus group discussion in South Sulawesi", by AgFor South Sulawesi team/ICRAF

52 Weeks of BetterEvaluation. Click here to view past features

A special thanks to this page's contributors
Author
Learning by Design.
Australia.

Comments

There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.