Week 15: Fitting reporting methods to evaluation findings – and audiences
This week we're sharing some ideas from Rakesh Mohan on ways of making evaluation reports more interesting.
Rakesh is Director at the Office of Performance Evaluations, Idaho State Legislature. He discusses how his team presented the findings of different evaluations which were intended for both policy-makers and public audiences.
As part of this evaluation, we had a difficult task of understanding and then explaining to policy-makers the movement of appropriated funds from various funding sources within Idaho’s largest state agency. The complexities of agency organization and processes and the sheer number of financial transactions made it difficult for even many members of the legislative budget committee to see a clear picture.
As part of our evaluation approach, we decided to use many large, colored charts to explain the management of appropriated funds and the movement of those funds across various divisions and programs. The use of software traditionally used to depict the flow of energy was very helpful in creating Sankey charts that depicted the movements of funds (pages 11 and 23). We also used Visio to prepare two other charts (pages 17 and 29). The charts were so effective in clarifying a complex system that department officials asked us to provide training to their staff on how to create the same charts for their own reporting purposes.
To help policy-makers and stakeholders better understand how to assess business tax policies, we developed an interactive web-based tool that allowed its users to evaluate the link between their tax policy proposal and tax policy goals. This tool also got the attention of the media. Our work was highlighted on Idaho Reports, a show of Idaho Public Television.
To make this report useful to policy-makers, we developed interactive data visualizations that we put on our website and used in our presentations. We used Tableau to develop these charts.
In terms of the evaluation approach and the report message, we wanted to make sure we did not duplicate other concurrent efforts to address compensation issues: we didn’t want to produce a report that had the same old message noted in previous studies, (i.e., state employees are paid about 18% below the market rate). Our message focused on the state’s compensation policy and how it was being implemented.
We conducted this study during a time when Idaho was going through a very contentious debate on public education policies, including workforce issues involving public school teachers. We issued our report on the heels of 2012 elections when the public voted to repeal three pieces of controversial public education legislation.
We believed that our report would not be seen as credible work if we did not include stakeholder perspectives in our report, so we surveyed teachers, principals and school district superintendents to get their perspectives on workforce issues affecting them. As expected, our report was widely read and used primarily because of the stakeholder perspectives brought to light by the survey. However, we also had to face some unexpected consequences as reflected in this media story which was front page of the Idaho Statesman.
Example - One teacher articulated his or her perception of why teachers may leave:
“I can understand why no teacher would want to teach here or stay because every year for the past five years we have taken a pay cut, had more responsibilities, and been given less support and appreciation.”
This evaluation dealt with a politically sensitive issue that had been discussed and debated for almost three decades. The four institutions of higher education in Idaho argued that the legislature was not providing equitable funding to them.
To answer policy-makers’ questions, we used charts that plotted the distribution of per student funding (represented by the Gini coefficient) over time. In our report and in presentations, we used charts that clearly illustrated to policy-makers and stakeholders that the funding mechanism expected to close gaps in funding levels did not affect the distribution in any significant way.