BetterEvaluation contributors at AEA2014 Denver

BE Contributors at the AEA 2014 Conference in Denver

Overview - Contributors with sessions

Presentation Details

Beverly Parsons

Opening Session: Visionary Evaluation for a Sustainable, Equitable Future


Session Number: PS1
Track: Plenary Sessions
Session Type: General Session
Presenter 1: Lovely Dhillon [ORS Impact]
Presenter 2: Matt Keene [US Environmental Protection Agency]
Presenter 3: Xiuhtezcatl Martinez [Earth Guardians]
Presenter 4: Beverly Parsons [InSites]
Time: Oct 15, 2014 (03:15 PM - 04:15 PM)
Room: Centennial D-H


Level: None


Session Abstract: 

Visionary evaluation encourages us to focuses on what we deeply want the world to look like whether or not we can envision it as possible or accessible from where we currently are. Xiuhtezcatl Martinez, 14-year-old youth director of Earth Guardians, joins Beverly Parsons (2014 AEA President), and Matt Keene and Lovely Dhillon (2014 AEA conference program chair and co-chair) to challenge your thinking about a sustainable, equitable future. They will engage you in creating your own version of visionary evaluation that contributes to a future that matters to you, those you serve through your evaluation work, and future generations.

A Sustainability Manifesto for Evaluation


Session Number: 10001
Track: Presidential Strand
Session Type: Panel
Session Chair: Matt Keene [US Environmental Protection Agency]
Presenter 1: Eleanor Chelimsky
Presenter 2: Juha Uitto [Global Environment Facility]
Presenter 3: Beverly Parsons [InSites]
Presenter 4: Andy Rowe [ARCeconomics]
Presenter 5: Alejandro Ortega-Argueta [El Colegio de la Frontera Sur]
Presenter 6: Claudia Romero
Time: Oct 15, 2014 (04:30 PM - 06:00 PM)
Room: Capitol 4


Level: None


Session Abstract: 

In this session we will speak with colleagues who are looking at sustainability through an evaluation lens. Two kinds of sustainability come into focus: the sustainability of the intervention and the sustainability of human and natural systems related to the intervention. Panelists will discuss why sustainability turns our attention toward program origins and public interest values; the dynamics of human and natural systems and the tradeoffs that emerge at their nexus; strong versus weak sustainability; and the micro - macro paradox where project level successes are heralded but enveloped by large scale failures. Panelists will present ways for evaluators to think about sustainability. They will begin to clarify the practical implications of the sustainability concept for evaluation practice and research. Panelists will build upon the concepts and discussions introduced in this session to their perspectives on a sustainability manifesto to guide the conduct of evaluation.

American Evaluation Association Annual Business Meeting


Session Number: AEA
Track: General
Session Type: General Session
Presenter 1: Beverly Parsons [InSites]
Presenter 2: Susan Tucker [Evaluation and Development Associates LLC]
Presenter 3: Jody Fitzpatrick [University of Colorado Denver]
Presenter 4: Stewart Donaldson [Claremont Graduate University]
Presenter 5: Denise Roosendaal [American Evaluation Association]
Time: Oct 16, 2014 (11:55 AM - 12:45 PM)
Room: Capitol 4


Level: None


Session Abstract: 

Join your AEA Leadership to discuss the state of the association, our plans for the future, and our role in evaluation practice.

From Visionary Evaluation to Exemplary Evaluations in a Multicultural World: Learning from Evaluation’s Successes Around the Globe Part 1


Session Number: 10012
Track: Presidential Strand
Session Type: Panel
Session Chair: Michael Quinn Patton [Utilization-Focused Evaluation]
Discussant: Stafford Hood [University of Illinois at Urbana-Champaign]
Presenter 1: Beverly Parsons [InSites]
Presenter 2: Stewart Donaldson [Claremont Graduate University]
Presenter 4: Marco Segone [UN Women]
Presenter 5: Claudine Voyadzis [European Evaluation Society]
Time: Oct 17, 2014 (03:30 PM - 04:15 PM)
Room: Capitol 4


Level: None


Session Abstract: 

President Beverly Parsons summarizes some of the key findings and insights that have been gleaned about visionary evaluation during 2014. President-elect Stewart Donaldson will introduce AEA’s 2015 conference theme, linking it to both Visionary Evaluation and the International Year of Evaluation. The 2015 theme, Exemplary Evaluations in a Multicultural World, aspires to promote sharing and learning from one another about exemplary evaluations and evaluation approaches across diverse cultures and national boundaries. A special emphasis will be placed on learning from extraordinary examples of influential evaluations and enabling environments for evaluation in both developed and developing countries across the globe.

Closing Session: AEA’s Role in A Sustainable, Equitable Future: A Community Conversation


Session Number: PS2
Track: Plenary Sessions
Session Type: General Session
Presenter 1: John A Gargani [Gargani + Company Inc]
Presenter 2: Claudine Voyadzis [European Evaluation Society]
Presenter 3: Michael Scriven [Claremont Graduate University]
Presenter 4: Stewart Donaldson [Claremont Graduate University]
Presenter 5: Rodney K Hopson [George Mason University]
Presenter 6: Beverly Parsons [InSites]
Time: Oct 18, 2014 (11:30 AM - 12:30 PM)
Room: Centennial D-H


Level: None


Session Abstract: 

Join your conference colleagues for table discussions, chats, and tweets about Now What? What is the role of AEA in supporting a sustainable, equitable future for all? How can AEA support your plans for visionary evaluation? How should AEA contribute to the global evaluation community in a way that, as stated in the AEA Guiding Principles, takes into account “the diversity of general and public interests and values… going beyond analysis of particular stakeholder interests to consider the welfare of society as a whole.”

Following table conversations, past, present, and future AEA presidents (Michael Scriven, Rodney Hopson, Beverly Parsons, Stewart Donaldson, and John Gargani) and the European Evaluation Society president (Claudine Voyadzis) engage in a conversation about AEA’s role in a sustainable, equitable future.

The session ends with our 2014 AEA poem written and read by Susan Robbins based on the conference theme. 

Bob Williams

Realizing the Vision: What Helps and Hinders the Use of Systems Ideas in Evaluation… Voices from the Global Field


Session Number: 10017
Track: Presidential Strand
Session Type: Think Tank
Session Chair: Bob Williams [All over the place]
Presenter 1: Richard Hummelbrunner [OEAR Regionalberatung]
Time: Oct 15, 2014 (04:30 PM - 06:00 PM)
Room: Centennial C


Level: None


Session Abstract: 

This session will bring together in an engaging and practical way, the experiences of those who have used and attempted to use ideas drawn from the systems field and address the question; what has helped and what has hindered the adoption and application of systems ideas in the evaluation field? The process itself will alternate between description of experiences and reflection on those experiences using highly participative processes such as World Café, Circular Dialogue, Open Space and/or Fishbowl. It will definitely not be a talking heads session. This process a unique opportunity to be part of an international initiative with a similar session proposed for the 2014 European Evaluation Society, a special edition of the journal Evaluation in 2015 and other activities such as Webinars. The result of this suite of processes will be practical ideas to help develop truly visionary evaluation.

Applying a Critical Systems approach to Developmental Evaluation


Session Number: 671
Track: Systems in Evaluation
Session Type: Skill-Building Workshop
Presenter 1: Bob Williams [All over the place]
Time: Oct 18, 2014 (09:45 AM - 11:15 AM)
Room: 705


Level: None


Session Abstract: 

This demonstration session will show how to use methods from the Critical Systems tradition to identify and critique rapidly core boundary choices that underpin Developmental Evaluation. Systems concepts embody three main features; an awareness of inter-relationships, an acknowledgement of multiple perspectives and an exploration of boundary choices. For evaluators the key systems concept is boundaries. Boundaries delineate what is "worthwhile" and "valued" from what is not. Thus identifying and critiquing boundaries is at the heart of the evaluation enterprise. And the best place is during the adaptive phases of an intervention; the home of Developmental Evaluation. This is a more focused and developed version of a demonstration session I ran last year. I've had a number of requests to repeat it. This is a revised version that is even more focused on the core evaluation task of "valuing".

Patricia Rogers

The Evaluator’s Role in Building Sustainable, Equitable Communities 


Session Number: 10002
Track: Presidential Strand
Session Type: Panel
Session Chair: Patricia Rogers, PhD [Royal Melb Institute of Technology]
Presenter 1: Abby Ruskey [E3 Washington]
Presenter 2: Chris Y. Lovato [University of British Columbia]
Time: Oct 15, 2014 (06:15 PM - 07:00 PM)
Room: Capitol 4


Level: None


Session Abstract: 

This session illustrates how evaluators can help communities become sustainable and equitable. One presenter describes a statewide visioning process that helped define what is meant by Sustainable Communities and the role of education. It includes the establishment of a set of outcomes that position a community to create positive environmental, social, and economic value. Another presenter illustrates how an evaluation is shifting as a community moved rapidly forward in economic and social change stimulated by health care issues. The session illustrates how evaluators can play important roles in assisting social systems to transform their norms, policies, infrastructures and practices to support such communities.

Evaluating Prizes, Challenges and Competitions


Session Number: 942
Track: Nonprofit and Foundations
Session Type: Panel
Session Chair: Nancy MacPherson [The Rockefeller Foundation]
Discussant: Patricia Rogers, PhD [Royal Melb Institute of Technology]
Presenter 1: Jon Sotsky [The Knight Foundation]
Presenter 2: Jared Raynor [TCC Group]
Time: Oct 17, 2014 (03:30 PM - 04:15 PM)
Room: 703


Level: None
Abstract 1 Title: An Overview of the Field of Prizes, Challenges and Competitions
Presentation Abstract 1: Jon will provide an overview of the field of prizes, challenges and competitions and why many funders choose the approach. He will ground the discussion in practical examples, through sharing the work of the Knight Foundation and representing the work of others in the field.
Abstract 2 Title: The Framework for Evaluating Prizes, Challenges and Competitions
Presentation Abstract 2: Jared will discuss the framework for evaluating prizes, challenges and competitions, including a discussion of relevant measures and indicator categories and examples.


Session Abstract: 

Use of prizes, challenges and competitions has become an important tool in philanthropy and in the public and private sectors. However, relatively little has been done to strategically think about evaluation implications for dealing with these kinds of programs. This session will provide participants with a greater understanding of the current state of the field of evaluating prizes, challenges and competitions, including sharing practical examples, discussing a framework of measures and indicators, and exploring methodological implications for evaluators.

Learning about better evaluation across silos and oceans


Session Number: 967
Track: International and Cross Cultural Evaluation
Session Type: Demonstration
Presenter 1: Patricia Rogers, PhD [Royal Melb Institute of Technology]
Presenter 2: Dugan Fraser [Consultant]
Time: Oct 17, 2014 (04:30 PM - 05:15 PM)
Room: Capitol 6


Level: None


Session Abstract: 

BetterEvaluation, an international collaboration, provides a central, freely accessible, public good access point for expertise in evaluation from across the world. It links to existing resources and knowledge communities and also generates original material and decision support. The website is complemented by events and activities to support evaluators, evaluation managers, and those evaluating their own programs to choose the appropriate methods and processes and to use them effectively. The session will demonstrate the knowledge generation processes used to identify, document and analyze good practice and the knowledge translation activities used to support users to apply material appropriately to their particular situation, adapting it to suit a new political, geographic, or social setting. It will also demonstrate the decision support provided for choosing methods and processes to build knowledge about when to use particular methods, and which contextual features need to be considered when making choices of evaluation methods and processes.

Rakesh Mohan

Effective and Sustainable Practices for Government Evaluation Contracting


Session Number: 1693
Track: Government Evaluation
Session Type: Panel
Session Chair: David J Bernstein, Ph.D. [Westat]
Discussant: Kathryn Newcomer [George Washington University]
Presenter 1: William H Scarbrough [PIRE]
Presenter 2: Rakesh Mohan [Idaho State Legislature]
Presenter 3: Jennifer Hamilton [WESTAT]
Presenter 4: Herbert Mark Baum [HMB Consulting Services]
Presenter 5: Connie K Della-Piana [National Science Foundation]
Presenter 6: David J Bernstein, Ph.D. [Westat]
Time: Oct 17, 2014 (08:00 AM - 09:30 AM)
Room: 708

Abstract 2 Title: A State Government Agency Perspective on Contracting for Evaluation
Presentation Abstract 2: Rakesh Mohan has been the director of the Idaho Legislature's Office of Performance Evaluations (OPE) since 2002. Under his direction, OPE has received many awards, including the 2011 Alva and Gunnar Myrdal Government Evaluation Award from AEA. Rakesh is currently a member of AEA's Evaluation Policy Task Force and the editorial advisory board of the American Journal of Evaluation. In the past, he served on the US Comptroller General's Advisory Council on Government Auditing Standards, the AEA Board of Directors, and the editorial advisory board of New Directions for Evaluation. Mr. Mohan will present contracting practices that have been effective for his award winning state performance evaluation office.

Building New Partnerships to Promote the Use of Evaluation among Public Policymakers


Session Number: 10011
Track: Presidential Strand
Session Type: Panel
Session Chair: Rakesh Mohan [Idaho State Legislature]
Discussant: William Trochim [Cornell University]
Presenter 1: Bill Pound [National Conference of State Legislatures]
Presenter 2: Michael Caudell-Feagan [The Pew Charitable Trusts]
Presenter 3: Rich Wills [Idaho Legislature]
Time: Oct 17, 2014 (01:45 PM - 03:15 PM)
Room: Capitol 4


Level: None


Session Abstract: 

The conference theme asks us to “envision evaluation as fundamental to a sustainable, equitable future” and to “unleash the power of evaluation…to impact the world in positive ways.” To unleash that power, we will need to build new partnerships with those who have access to policymakers and understand the nuances of the public policy process. These partnerships will help us better understand policymakers’ information needs and allow us to educate them about the value of evaluation in making informed policy and program decisions that would help create a sustainable and equitable future. This highly interactive session brings together experts from three organizations that have worked with policymakers for decades: the National Conference of State Legislatures, the Council of State Governments, and the Pew Charitable Trust. The audience will hear from a senior legislator who is using the work of these organizations to improve criminal justice policies in his state.

Irene Guijt

Challenges of evaluation use and evaluation influence in government


Session Number: 764
Track: Evaluation Use
Session Type: Multipaper
Session Chair: Janet May Clinton [University of Melbourne]
Presenter 2: Irene Guijt [Learning by Design]
Presenter 3: Kerrie Ikin [Drummoyne Services in Education]
Presenter 4: Andrew Lejeune
Time: Oct 16, 2014 (03:00 PM - 04:30 PM)
Room: Capitol 1

Abstract 2 Title: Cultivating Curiosity and Capacity about Impact Evaluation in the Department of Foreign Affairs and Trade, Australia
Presentation Abstract 2: This paper describes the journey in 2012-2014 of embedding willingness, processes, and relationships in the Department of Foreign Affairs and Trade (DFAT) headquarters and in selected projects. In 2012, DFAT (then AusAID) commissioned the Overseas Development Institute to help it do more impact evaluation. An explicit focus was included on learning how to evaluate 'hard-to-measure' types of development interventions, and collaborative learning via project cases. Developing ownership of an initiative spanning departments, regions and themes proved a significant challenge. The diversity of cases, in timeframe, modality and scale, led to unique processes in each case. Dealing with this diversity required balancing shared processes to ensure quality and flexibility. The paper describes conditions that helped and hindered the embedding of impact evaluation in the agency, and takes stock of uptake to date.

Just Means, Just Ends: Documenting Evaluations' Contributions to Social Justice


Session Number: 876
Track: Theories of Evaluation
Session Type: Panel
Session Chair: Jennifer C Greene [University of Illinois at Urbana-Champaign]
Discussant: Jennifer C Greene [University of Illinois at Urbana-Champaign]
Presenter 1: Rebecca Campbell [Michigan State University]
Presenter 2: Irene Guijt [Learning by Design]
Time: Oct 17, 2014 (01:45 PM - 03:15 PM)
Room: 103

Abstract 2 Title: Social Justice and Poverty Reduction: Participatory Impact Assessment and Learning
Presentation Abstract 2: Dr. Guijt will draw on two cases for the international development arena that illustrate the tensions between ensuring that the voices of those living with intended poverty reduction changes deeply inform findings while the evaluators also strive to meet narrowly defined demands for rigour. One case involves a recent impact evaluation of an IFAD (International Fund for Agricultural Development) project in Vietnam seeking to make markets work for the poor, in which village level participatory sense making was piloted where villagers are rarely asked their views. A second example involves GirlHub in Ethiopia and Rwanda, that uses innovative social media to shift societal views to value girls and focuses on ensuring girls' voices are central to evaluation processes.

Julia Coffman

New Frontiers in Policy Advocacy Evaluation and Strategy


Session Number: 1914
Track: Advocacy and Policy Change
Session Type: Panel
Session Chair: Barbara Masters [Barbara Masters]
Discussant: Julia Coffman [Center for Evaluation Innovation]
Presenter 1: Barbara Masters [Barbara Masters]
Presenter 2: Gigi Barsoum [Barsoum Policy Consulting]
Presenter 3: Sandra Martinez [County of San Diego]
Time: Oct 17, 2014 (01:45 PM - 03:15 PM)
Room: Centennial A


Level: None

Session Abstract: 

This session will highlight three new frameworks for assessing foundation-funded public policy and advocacy efforts. The frameworks are useful in helping evaluators to assess and visually present: 1) an issue's "ripeness" and advocacy field's "readiness" for policy change, 2) a foundation's contribution to policy change, and 3) a comprehensive spectrum of approaches that foundations can use in their public policy grantmaking. These tools emerged from a retrospective evaluation of the last ten years of one foundation's public policy grantmaking. The frameworks developed can be used for retrospective assessment as well as prospective strategic learning and planning. Presenters will describe the evaluation design and how the frameworks were developed and used. A discussant will provide commentary on how the frameworks contribute to the continued building of the policy advocacy evaluation field.

Micheal Quinn Patton

(03) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use


Session Number: 03
Track: Professional Development Workshops
Session Type: Professional Development Workshops
Workshop Lead Presenter: Michael Quinn Patton [Utilization-Focused Evaluation]
Time: Oct 13, 2014 (09:00 AM - 04:00 PM)
Room: Mineral F/G


Workshop Speaker Bios:

Michael Quinn Patton is an independent consultant and professor at the Union Institute. An internationally known expert on Utilization-focused Evaluation, this workshop is based on his seminal book, "Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use" (Guilford, 2010).


Learning Outcomes: • The specific niche for which developmental evaluation is appropriate and useful;
• Five types of DE and the implications of those types;
• The dimensions of complexity that affect how DE is;
• Practical frameworks and innovative methods for use in DE.
Prerequisites: None
Level: Beginner


Session Abstract: 

Developmental evaluation (DE) is especially appropriate for innovative initiatives or organizations in dynamic and complex environments where participants, conditions, interventions and context are turbulent, pathways for achieving desired outcomes are uncertain, and conflicts about what to do are high. DE supports reality-testing, innovation and adaptation in complex dynamic systems where relationships among critical elements are nonlinear and emergent. Evaluation use in such environments focuses on continuous and ongoing adaptation, intensive reflective practice and rapid, real-time feedback. The purpose of DE is to help develop and adapt the intervention (different from improving a model). This evaluation approach involves partnering relationships between social innovators and evaluators in which the evaluator’s role focuses on helping innovators embed evaluative thinking into their decision-making processes as part of their ongoing design and implementation initiatives.

Through lecture, discussion and small-group practice exercises, this workshop will position DE as an important option for evaluation, in contrast to formative and summative evaluations as well as other approaches to evaluation.

Principles as Evaluand: A Principles-focused Developmental Evaluation Approach to Visionary Evaluation for Principles-focused Theories of Change


Session Number: 565
Track: Theories of Evaluation
Session Type: Demonstration
Presenter 1: Michael Quinn Patton [Utilization-Focused Evaluation]
Time: Oct 16, 2014 (08:00 AM - 09:30 AM)
Room: 102


Level: None


Session Abstract:  Historically, evaluation has focused on projects and programs as the evaluand. Theories and methods of evaluation are primarily program-focused. Visionary Evaluation for a Sustainable, Equitable Future, however, will have to address new evaluands, e.g., evaluating strategies, visions, and principles. Principles-focused interventions constitute an approach to change (a type of theory of change) that is distinct from model-focused interventions. For example, a group of diverse programs, all adhering to the same principles but each adapting those principles to its own particular target population within its own context, presents a different kind of evaluation challenge than a group of programs all implementing the same model. Distinct from summative evaluation and meta-evaluation, a principles-focused synthesis approach grounded in Developmental Evaluation offers an approach to and theory of evaluation for generating evidence-based principles. This session will demonstrate principles-focused developmental evaluation with actual examples, integrating both theory and practice.

Qualitative Inquiry in the Practice of Evaluation: Perspectives on Theory and Practice


Session Number: 440
Track: Qualitative Methods
Session Type: Panel
Session Chair: Leslie Goodyear [Education Development Center (EDC)]
Discussant: Michael Quinn Patton [Utilization-Focused Evaluation]
Presenter 1: Leslie Goodyear [Education Development Center (EDC)]
Presenter 2: Janet Usinger [University of Nevada, Reno]
Presenter 3: Eric Barela [Partners in School Innovation]
Presenter 4: Jennifer Jewiss [University of Vermont]
Time: Oct 16, 2014 (08:00 AM - 09:30 AM)
Room: 705

Session Abstract:  In this dynamic, multi-faceted panel, the editors of the forthcoming book, Qualitative Inquiry in the Practice of Evaluation (Jossey-Bass, forthcoming), will bring together the chapter authors to share their perspectives on the intersections between qualitative inquiry and evaluation. Brief presentations by the editors and chapter authors will be followed by in-depth discussion among the panelists and attendees. This session will offer insights gleaned from the panelists' reflections on the state of practice and theory in the realm of qualitative evaluation. The session is designed for all evaluators who are interested in incorporating qualitative inquiry into their evaluation practice, particularly those who are new to qualitative inquiry or seasoned qualitative evaluators seeking an opportunity to reflect on their practice. The panel will explore a wide range of topics such as the history of qualitative evaluation, ethical issues, key competencies, evaluator role and identity, and other lessons learned from practice.

Corporate Sustainability and Evaluation: (Plenary Session Follow Up)


Session Number: 10004
Track: Presidential Strand
Session Type: Panel
Session Chair: Cameron Norman [CENSE Research + Design]
Presenter 1: Bob Willard [Sustainability Advantage]
Presenter 2: Michael Quinn Patton [Utilization-Focused Evaluation]
Time: Oct 16, 2014 (11:00 AM - 11:45 AM)
Room: Capitol 4


Level: None


Session Abstract: 

This session will provide an opportunity for questions and reflections related to the plenary session A Vision of Corporations Fit for the Future: Implications for Evaluation in which Bob Willard talks about how businesses are addressing economic, social, and environmental sustainability and the role of standards and benchmarks. He will be joined in this discussion by Michael Patton to look more in depth at various links between evaluation and how businesses are addressing sustainability.

Puzzles, problems and messes: insights and reflections on generating evidence based principles from developmental and adaptive evaluation practice in innovative and complex settings


Session Number: 1024
Track: Organizational Learning & Evaluation Capacity Building
Session Type: Multipaper
Session Chair: Michael Quinn Patton [Utilization-Focused Evaluation]
Presenter 1: Kate McKegg [The Kinnect Group]
Presenter 2: Nora Murphy [TerraLuna Collaborative]
Presenter 3: Nan Wehipeihana, Independent Evaluator [Kinnect Group]
Time: Oct 16, 2014 (03:00 PM - 04:30 PM)
Room: Capitol 5

Session Abstract: 

The lure of a simple approach to monitoring and evaluation that can demonstrate a straightforward relationship between inputs, outputs and outcomes is understandable when managers of programs are under pressure to deliver results for less. However, the lure often turns out to be a mirage in the messy, real world where social and environmental change of some kind is the driving purpose for people; and where there is a real need for new ideas, new ways of working to solve entrenched, wicked problems. It turns out that the relationships between inputs and outcomes is complex, involving systems of interaction in which there is a great deal of in-built uncertainty. The challenge for evaluation amid this messy real world, is to still generate credible, evidence based learning that can be used beyond a particular single intervention, adapted, rather than adopted, tailored and modified for context, without losing the kernel of what has been found to lead to success somewhere else. Drawing on several case studies where developmental evaluation has been applied in complex and innovative social settings, this session will provide insights through the use of case examples, of the generation, adaption and use of evidence based principles.

Lack-of-vision evaluation ideas that should be retired to realize Visionary Evaluation for a Sustainable, Equitable Future


Session Number: 559
Track: Qualitative Methods
Session Type: Think Tank
Session Chair: Michael Quinn Patton [Utilization-Focused Evaluation]
Time: Oct 16, 2014 (04:45 PM - 06:15 PM)
Room: 708


Level: None


Session Abstract:  Over the last 40 years the evaluation profession has generated many innovative approaches that support Visionary Evaluation for a Sustainable, Equitable Future, but we also have accumulated baggage in the form of outdated and counterproductive ideas and approaches that need to be abandoned. In revising my Qualitative Evaluation Methods textbook this past year, I identified 10 such outdated, counterproductive ideas ready for retirement. I'll begin the session with my own Top 10 List of lack-of-vision evaluation ideas ready for retirement, then invite additional nominations from those in attendance. My list of lack-of-vision ideas focus on concepts, terminology, and approaches that create barriers to the full, appropriate, and credible use of qualitative methods in evaluation.

Comparing and Contrasting Alternative Approaches to Addressing Uncertainty from a Systems Perspective


Session Number: 536
Track: Systems in Evaluation
Session Type: Panel
Session Chair: Michael Quinn Patton [Utilization-Focused Evaluation]
Presenter 1: Jonathan Morell [Fulcrum corporation]
Presenter 2: Glenda Eoyang [HSD Institute]
Time: Oct 17, 2014 (08:00 AM - 09:30 AM)
Room: 111

Session Abstract: 

This session emerged from an evaluation book group's frustration. The group had spent the summer carefully reading three books: (1) Morell's Evaluation in the Face of Uncertainty: Anticipating Surprise and Responding to the Inevitable; (2) Eoyang and Holladay's Adaptive Action: Leveraging Uncertainty in Your Organization; and (3) Patton's Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Each of the books clearly applied a form of systems thinking to suggest how to respond to uncertainty during evaluations, but, upon comparison, the exact details of what to do when confronting surprise proved challenging to extract. This panel presentation will make these details clear by explicating the three perspectives and then comparing and contrasting them to determine which details (if any) they share and which features are unique to a given approach. The key outcome will be to discuss how evaluators can better learn to anticipate and cope with surprise.

From Visionary Evaluation to Exemplary Evaluations in a Multicultural World: Learning from Evaluation’s Successes Around the Globe Part 1


Session Number: 10012
Track: Presidential Strand
Session Type: Panel
Session Chair: Michael Quinn Patton [Utilization-Focused Evaluation]
Discussant: Stafford Hood [University of Illinois at Urbana-Champaign]
Presenter 1: Beverly Parsons [InSites]
Presenter 2: Stewart Donaldson [Claremont Graduate University]
Presenter 4: Marco Segone [UN Women]
Presenter 5: Claudine Voyadzis [European Evaluation Society]
Time: Oct 17, 2014 (03:30 PM - 04:15 PM)
Room: Capitol 4


Level: None


Session Abstract: 

President Beverly Parsons summarizes some of the key findings and insights that have been gleaned about visionary evaluation during 2014. President-elect Stewart Donaldson will introduce AEA’s 2015 conference theme, linking it to both Visionary Evaluation and the International Year of Evaluation. The 2015 theme, Exemplary Evaluations in a Multicultural World, aspires to promote sharing and learning from one another about exemplary evaluations and evaluation approaches across diverse cultures and national boundaries. A special emphasis will be placed on learning from extraordinary examples of influential evaluations and enabling environments for evaluation in both developed and developing countries across the globe.

APPLICATIONS OF DEVELOPMENTAL EVALUATION: LESSONS FROM COMMUNITY CARE, DANCE, HEALTH AND ADDICTIONS TREATMENT SYSTEM


Session Number: 1659
Track: Systems in Evaluation
Session Type: Panel
Session Chair: Sanjeev Sridharan [University of Toronto]
Discussant: Michael Quinn Patton [Utilization-Focused Evaluation]
Presenter 1: Brit Kewin [St. Michael's Hospital]
Presenter 2: April Nakaima
Presenter 3: Joanna Maplazi [St. Michael's Hospital]
Presenter 4: Sanjeev Sridharan [University of Toronto]
Time: Oct 18, 2014 (09:45 AM - 11:15 AM)
Room: 707

Session Abstract: 

These series of papers will discuss advances in application of developmental evaluation in Canadian and other international settings. Four examples and four approaches to developmental evaluation will be discussed. The four problems will include the use of video in developing integrated care for the elderly, the use of developmental evaluation to help plan a program of dance for older adults, the utility of developmental evaluation in ensuring that research focused on health inequities achieve their goals, and the role of developmental evaluation in transforming an addictions treatment system. The utility of developmental evaluation in understudying systemic changes (in multiple senses of the term) will be explored

Kerry Bruce

Agency Level Measurements -- Can they be Used to Evaluate Organizational Impact?


Session Number: 903
Track: International and Cross Cultural Evaluation
Session Type: Panel
Session Chair: Kerry Bruce, Senior Director for Global Health and Measurement [Pact]
Discussant: Laia Grino [2025526567]
Presenter 1: Kerry Bruce, Senior Director for Global Health and Measurement [Pact]
Presenter 2: Muluemebet Chekol Hunegnaw
Presenter 3: Barbara Willett [Mercy Corps]
Presenter 4: Maliha Khan [Oxfam America]
Time: Oct 16, 2014 (04:45 PM - 06:15 PM)
Room: 104

Session Abstract:  International non-governmental organizations (INGOs) have been increasingly interested in creating organizational level metrics and results that measure and evaluate more than the sum of their individual projects. This panel will review how several INGOs have approached this process and the advantages and disadvantages of collecting data and evaluating impact in this way. Ultimately the panel will try to understand whether and how these data and organizational measurements can be used to evaluate organization outcomes at the global level. The panelists are all Directors, responsible for organizational measurement at large international NGOs.

 

Harnessing the Potential of Mobile-Based Technologies for Evaluation: Opportunities, Challenges, and Key Considerations


Session Number: 1010
Track: Integrating Technology into Evaluation
Session Type: Multipaper
Session Chair: Alison Koler [Pact]
Presenter 1: Michael Bamberger, Dr. [independent consultant]
Presenter 2: Anustubh Agnihotri
Presenter 3: Kerry Bruce, Senior Director for Global Health and Measurement [Pact]
Presenter 4: Alison Koler [Pact]
Time: Oct 18, 2014 (09:45 AM - 11:15 AM)
Room: Marble

Abstract 3 Title: Are International Non-Governmental Organizations Adopting Mobile Technology into M&E Systems at the Organizational Level?
Presentation Abstract 3: International Non-Governmental Organizations (INGOs) are increasingly adopting mobile technology for monitoring and evaluation (M&E) in their work around the world; however, adoption is not yet complete. Pulling on survey research and field experience from INGOs, this session will review the state of adoption and the factors affecting the uptake of mobile technology in the evaluation field. In 2013, survey research showed 70.6% used mobile technology during the previous year and 77.2% intended to use mobile technology in the following year. The full power of mobile technology to change the way M&E systems are built and how data are collected at a systems level has begun. Barriers include lack of access to training and technical support and a need for systems thinking in evaluation that is not yet fully developed.

Jane Davidson

(04) Actionable Evaluation


Session Number: 04
Track: Professional Development Workshops
Session Type: Professional Development Workshops
Workshop Lead Presenter: E Jane Davidson [Real Evaluation Ltd.]
Time: Oct 13, 2014 (09:00 AM - 04:00 PM)
Room: Mineral A


Workshop Speaker Bios:

Dr. Davidson runs her own successful evaluation coaching, training, and consulting practice, Real Evaluation Ltd, which offers evaluation workshops and professional development as a key part of her services. She has led successful workshops in the past for AEA, CES, AES, TEI, anzea, and UNISA, as well as numerous workshops tailored to specific organizations' needs. Prior to returning home to New Zealand, Jane led and taught the core evaluation courses in Western Michigan University’s Interdisciplinary PhD in Evaluation.


Learning Outcomes: How to write a set of big-picture overarching questions to guide the evaluation
How to use evaluative rubrics to get direct, evaluative answers to these questions, especially when there are 'intangibles' to capture
How to design interview and survey questions geared directly toward answering evaluative questions
Evaluation conceptualization and reporting tips that maximize the chances of a clear, to-the-point, and actionable evaluation
Prerequisites: Some experience with doing, using, or commissioning evaluation.
Level: Beginner


Session Abstract: 

This workshop is the antidote to evaluations that get lost in indicators, metrics, observations, and stories. If evaluation feels more like a measurement or opinion-gathering exercise; if you wonder if there's something clearer and more valuable you could be delivering for decision makers, then this workshop is for you. This workshop covers the following: • Big-picture thinking about key stakeholders, their information needs, and the evaluative questions they need answered • A hands-on introduction to evaluative rubrics as a way of directly answering those questions, especially when there are a lot of 'intangibles' to capture • Guidance for designing interview and survey questions that are easier to interpret against evaluative rubrics • A reporting structure that gets to the point, delivering direct, actionable answers that decision makers can really use (not just a laundry list of recommendations)

Rubrics methodology: What we have learned from experiences around the world


Session Number: 1492
Track: International and Cross Cultural Evaluation
Session Type: Panel
Session Chair: Thomaz K Kauark Chianca [COMEA Evaluation Ltda.]
Discussant: Michael Scriven [Claremont Graduate University]
Presenter 1: E Jane Davidson [Real Evaluation Ltd.]
Presenter 2: Kate McKegg [The Kinnect Group]
Presenter 3: Mônica Rodrigues Dias Pinto [FRM - Roberto Marinho Foundation]
Time: Oct 17, 2014 (08:00 AM - 09:30 AM)
Room: 107

Abstract 1 Title: Rubrics: What, why, and how
Presentation Abstract 1: We formulate the questions, we gather the evidence, we write it up. Why do we need rubrics at all? Jane Davidson kicks off this session by explaining the theoretical basis and logic behind rubrics methodology. Put simply, evaluation is about delivering direct and useful answers to real evaluative questions. We don't just ask "What are the outcomes?"; we have to ask "How worthwhile are the outcomes?" That's a much more challenging question. Rubrics are a tool for making explicit the basis on which we say that something is good, worthwhile, high quality, valuable -- or not. They help us interpret not just the indicators and measures, but also the rich, intangible elements that characterize complex social interventions. Jane will outline some of the key options for developing rubrics (including who to include and how to get it right), as well as her tips for fitting rubric design to purpose and context.

 

Evaluation Criteria for Judging Value: Application to Sustainability and Equity


Session Number: 10016
Track: Presidential Strand
Session Type: Multipaper
Session Chair: Jennifer C Greene [University of Illinois at Urbana-Champaign]
Discussant: E Jane Davidson [Real Evaluation Ltd.]
Presenter 1: Ghislain Arbour [Center for Program Evaluation ]
Presenter 2: Mathea Roorda, Independent evaluator [Evalue Research: www.evalueresearch.co.nz]
Presenter 3: George Julnes [University of Baltimore]
Presenter 4: Ernest Robert House [university of colorado]
Time: Oct 18, 2014 (09:45 AM - 11:15 AM)
Room: Capitol 4

Session Abstract: 

Evaluation is generally distinguished from research by the additional task of judging the value of what is being evaluated in terms of merit, worth, or importance. When applied to consumer products, this valuing is conducted in terms of identified criteria relevant to judgments of quality (e.g., safety and performance of a car). However, when evaluating social programs and policies the choice of criteria is often controversial. This lack of consensus on criteria calls into question the value judgments, and, yet, there has been insufficient discussion in the evaluation community of how criteria should be chosen. The papers in this session seek to highlight areas of needed discussion and also offer frameworks to be considered in developing more systematic approaches to selecting and applying criteria in evaluation.

Judy Oakden


If a picture paints a thousand words: the use of rich pictures in evaluation


Session Number: 1661
Track: Systems in Evaluation
Session Type: Demonstration
Presenter 1: Judy Oakden [Private Consultant]
Time: Oct 17, 2014 (11:00 AM - 11:45 AM)
Room: 105


Level: None


Session Abstract:  At times evaluators need to work with stakeholders to develop an evaluation framework that responds to the complexity of their situation and still provides robust, credible evaluation findings. Rich pictures are a Soft Systems Methodology tool that offers a quick and efficient way to work with key stakeholders to better understand their 'problematical situation' (Checkland & Poulter, 2006). In evaluation speak, rich pictures can help evaluators identify boundaries, examine and be cognisant of multiple perspectives, and understand the complexity of relationships between a range of stakeholders. We have found integrating the use of rich pictures into our evaluation practice helpful as a way of in setting the scope for the evaluation, framing the Key Evaluation Questions and in informing the development of the evaluative criteria. This presentation explores the use of rich pictures from the field, and reflects on the benefits and challenges of using them in evaluation.

William Faulkner

William Faulkner - author of Critical Systems Heuristics -
Fri, Oct 17, 2014 (08:00 AM - 09:30 AM)

     Advocacy and Policy Change
APC1:Evaluation on complex global contexts: restricted countries, multi-level advocacy, and theoretically-grounded advocacy evaluation
Fri, Oct 17, 2014 (11:00 AM - 11:45 AM)

     International and Cross Cultur
1175:Challenging the tacit: reanimating the story behind a landmark RCT from Mexico

Sonal Zaveri

Sonal Zaveri - webinar and Q and A - Evaluation and children
Thu, Oct 16, 2014 (04:45 PM - 06:15 PM)

     Feminist Issues in Evaluation
703:Gender based evaluation tools and uses
Fri, Oct 17, 2014 (01:45 PM - 03:15 PM)

     Feminist Issues in Evaluation
617:Rescuing the evaporation of gender from the international evaluation tea-pot!