52 weeks of BetterEvaluation: Week 11: Using rubrics

Patricia Rogers's picture 12th March 2013 by Patricia Rogers

The term "rubric" is often used in education to refer to a systematic way of setting out the expectations for students in terms of what would constitute poor, good and excellent performance.

In recent years rubrics have begun to be used explicitly in evaluation to address the challenge of systematically and transparently synthesising diverse evidence into an overall evaluative judgement. Sometimes they are called rubrics, and sometimes Global Assessment Scales, as they provide an overall rating of performance, based on detailed descriptions on a scale.

The main point of a rubric is that it is designed to support a holistic judgement about performance. It is not a scale that involves awarding points for this and points for that and then adding them up and reading off the scale.

There are now examples of rubrics being used in many sectors, including agriculture (rating community groups involved in a dryland salinity project), employment (rating the success of a program for seasonal immigrant workers from the point of view of both the workers and the employers), and Indigenous education (12 separate rubrics developed for different aspects of the Measurable Gains Framework in the Maori Education Strategy).

Rubrics can be used to ensure comparability of ratings across sites and raters and to effectively synthesise diverse evidence. They can also have important impacts through the process of developing and using them - negotiating agreement about criteria standards, and appropriate evidence of performance, and focusing attention on what is highly valued.

On the BetterEvaluation site, we have links to some great resources for developing rubrics, including guides and examples, which you can access through the options page on rubrics. But many questions remain.

That is why we're delighted that this week, when we launch our series of evaluation cases (developed through a writeshop process led by Irene Guijt) the first case will focus on an evaluation where rubrics were used.  

Judy Oakden's case focuses on the use of rubrics in the evaluation of a leadership development program for principals - discussing how the rubric was developed, what it looked like and how it was used. In addition, her case sets the use of rubrics within the context of the whole evaluation - how it was managed, the framing of the evaluation in terms of its purpose and key questions, the theory of change underpinning the program, how data were collected and analysed, and what was done to report findings and support their use.

Watch for her guest blog post, the evaluation case and the chance to ask questions about developing and using rubrics in evaluation.

 

Image source (top): New Zealand Qualifications Authority (2009) Policy and Guidelines for the Conduct of External Evaluation and Review

52 Weeks of BetterEvaluation. Click here to view past features

A special thanks to this page's contributors
Author
Director of BetterEvaluation/ Professor of Public Sector Evaluation, Australia and New Zealand School of Government.
Melbourne.

Comments

rickjdavies's picture
rick davies

Hi Patricia

1. How are rubrics different to protocols?

2. (They may already be staring me in the face, but..) Could you provide some simple examples?

regards, rick

Patricia Rogers's picture
Patricia Rogers

Hi Rick,

Not sure what you mean by protocol as the term is used for very different things.  Can you please give an example?

There are a number of examples of rubrics we've recently added to the site:

Judy Oakden's new case study about developing and using rubrics in leadership development http://betterevaluation.org/resource/example/rubrics-oakden

three examples embedded in the blog post above (but perhaps not obviously enough - click on the blue words):

rating community groups involved in a dryland salinity project - Cam Nicholson's example of using Global Attainment Scaling in evaluating the strength of community groups - on our rubrics page http://betterevaluation.org/evaluation-options/rubrics

employment (rating the success of a program for seasonal immigrant workers from the point of view of both the workers and the employers)

Indigenous education (12 separate rubrics developed for different aspects of the Measurable Gains Framework in the Maori Education Strategy).

rickjdavies's picture
rick davies

Hi Patricia

Thanks for the list of links to examples

Re protocols, I was thinking of protocols as expected procedures for doing something. I suppose a rubric of the kind you have described is a particular kind of protocol, in that it gives guidance on how to apply a given rating scale value. But the name (rubric) suggests something somewhat more optional, like guidance. Is that what is intended?

regards, rick

Patricia Rogers's picture
Patricia Rogers

Thanks for the clarification, Rick.

I've seen the term 'protocol' used to refer to all sorts of specified procedures, including how to do a systematic review, how to collect data, and how to sterilise equipment.  So I don't think it's specific enough  to use to refer to specified procedures for synthesising data/evidence.

As to whether or not a rubric is more optional than guidance, my understanding from the examples I've seen where they have been developed and used systematically (eg AES presentation by Jane Davidson, Nan Wehipeihana and Kate McKegg http://kinnect.co.nz/wp-content/uploads/2011/09/AES-2011-Rubric-Revoluti...) and Judy Oakden's case ( http://betterevaluation.org/resource/example/rubrics-oakden) is that they were not used like algorithms, just following rules for adding up points, nor were they vague suggestions, but processes which should be done carefully and only revised through consultation if inadequacies arose.

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.