This report describes a holistic approach and assessment framework for evaluating 'research' that goes beyond the traditional deliberative means (e.g., peer review) and often used analytics (e.g., bibliometrics).
The approach underscores that scientific merit is a necessary but insufficient condition for judging research quality and acknowledges that determining the effectiveness of research (in terms of its relevance, use and impact) should not only involve researchers, research managers and funders but multiple stakeholders and potential users. Hence, the RQ+ Assessment Framework consists of three components:
(1) key influences (enabling or constraining factors) either within the research endeavor or in the external environment;
(2) research quality dimensions (integrity, legitimacy, importance, positioning for use) and subdimensions which are closely inter-related; and,
(3) customizable assessment rubrics (also called 'evaluative rubrics') that make use of both qualitative and quantitative measures.
The work was conducted with input from staff and support from the International Development Research Centre (IDRC) in Canada which primarily funds and facilitates global South-based research for development.
The information provided was supplied by Robert McLean, IDRC, and Andréanne Martel, IDRC.
Authors and their affiliation
Zenda Ofir - Independent Evaluator & Honorary Professor, University of Stellenbosch, South Africa
Thomas Schwandt - Professor, University Of Illinois, Urbana-Champaign, USA
Colleen Duggan - Senior Program Specialist, Policy and Evaluation Division, International Development Research Centre, Canada
Robert McLean - Senior Program Specialist, Policy and Evaluation Division, International Development Research Centre, Canada & Lead Evaluator, Canadian Institutes of Health Research, Canada
Year of publication
Type of resource
Interest and debate about the quality and use of research, as well as the relationship of research undertakings to social goals and values, have been particularly intense in the field of internationally funded research on the development priorities of low-to-middle-income countries. It is in this arena -research for development (R4D)- that the approach to evaluating research quality discussed in this report was designed.
This report describes, in brief, the rationale behind the development of this new approach to evaluating research quality. It then discusses in some detail the assessment instrument itself, followed by a section on lessons learned from the first effort to implement it in the External Reviews of the IDRC work. In conclusion, the report reflects on potential uses of the instrument as well as ideas for its further refinement.
Who is this resource useful for?
- Commissioners/managers of evaluation;
- Those involved in evaluation capacity strengthening;
- Funders of research;
- Researchers engaged in evaluating research quality.
How have you used or intend on using this resource?
What is discussed here is the inaugural version of the RQ+ Assessment Framework. It is currently undergoing revision and adaptation by IDRC for its own specific circumstances and uses. Readers are encouraged to treat this Framework as a dynamic, evolving tool that they can adapt and modify for their specific purposes.
The authors expect that this approach to evaluating research quality will be improved by the practitioners who use the Framework and through further examination by scholars concerned with issues of research quality and use.
Why would you recommend it to other people?
This RQ+ approach holds potential as an evaluation method, but also as a method for enhancing research management. As an evaluation tool it can help one to systematically evaluate research quality. As a research management tool, RQ+ can support individual research project, but might also serve as a tool for multi-project research programming.
RQ+ may be most useful for “policy and practice-oriented research”. Particularly, research that encourages a wide variety of actors in the knowledge production process, conducted by researchers concerned with the potential social benefit of their research.
Furthermore, unlike other tools to evaluate research quality, RQ+ is adapted to the complexity and specific nature of research in the field of international development.
Part 1: Rationale
Part 2: The RQ+ Assessment Framework - this is the inaugural version of the framework. IDRC is currently making adaptations for its own circumstances and uses but also encourages others to treat the framework as a dynamic, evolving tool to modify to their specific purposes.
Part 3: Lessons from the implementation of the RQ+ Assessment Framework by IDRC
Part 4: The Potential of the RQ+ Assessment Framework
Ofir, Z., T. Schwandt, D. Colleen, and R. McLean (2016). RQ+ Research Quality Plus. A Holistic Approach to Evaluating Research. Ottawa: International Development Research Centre (IDRC).