EPE/EDP/KHP 520: INTRODUCTION TO EVALUATION
Instructor: Joseph J. Ferrare, Ph.D.
Spring 2019, Thursdays 11am - 1:30pm, DH 135
Office: 145C Taylor Education Building
Office Hours: Thursdays 10am - 11am & by appointment
Syllabus in PDF
Evaluation has become a ubiquitous feature of modern life. Organizations - such as schools, nonprofits, and businesses - are increasingly required to demonstrate that certain programs or policies are achieving desired outcomes in order to justify continued funding. In this context, evaluation refers to the use of empirical evidence to assess the outcome(s) associated with a program, practice, or policy. This state of affairs makes it crucial to understand the principles that inform evaluation work, including the technical details of evaluation design as well as the social and political contexts that influence how evaluations are carried out and acted upon.
This course surveys different approaches to evaluation that are widely used across the social sciences and in non-academic settings. We will begin with an overview of the different general approaches (or “paradigms”) to evaluation and then gradually work through the various steps of the evaluation process. This includes: engaging stakeholders; framing research questions; working with human participants and IRB; data collection strategies; analysis; visualization; and dissemination. Along the way, we will critically engage with a variety of examples of evaluations and explore how these projects inform programs, policies, and practices in the education sector and beyond.
Students who successfully complete the requirements of this course should expect the following:
To be able to critically analyze the elements of evaluation design from multiple perspectives (i.e., paradigms);
To understand the basic steps involved in designing an evaluation of a program or policy;
To become proficient at identifying relevant stakeholders in an evaluation context and adapting communication strategies appropriately.
The format for our class meetings will consist of interactive lectures, small group work, and individual exercises. The lectures are designed to provide an introduction to key concepts in evaluation, and to offer examples of these concepts as they apply to education and social science more generally. Please note that “interactive” lecture means that during this time I will pose questions directly to individual students with the intent of generating discussion. These questions will assume you have completed the assigned readings and have been engaged in class activities throughout the term. Finally, the small group and individual exercises will give you opportunities to practice thinking through decisions in the evaluation process within the context of practical scenarios.
REQUIREMENTS AND MODES OF EVALUATION
Your work in this course will be evaluated through multiple assignments. The specific modes of evaluation and corresponding grading weights are described below. Grades will be assigned using the following scale:
A: 90% – 100%, B: 80% – 89%, C: 70 – 79%, D: 60% – 69%, E: below 60%
When evaluating your work I will consider criteria specific to each assignment. In general, though, I consider grade ranges to meet the following generic standards:
95% – 100%: Exemplary work that exhibits mastery over the task
90% – 94%: Excellent work that approaches mastery but falls short in one key area
85% – 89%: High quality work that has ample room for improvement
80% – 84%: Work that exceeds minimum expectations but contains a number of mistakes or lacks quality in key areas
75% – 79%: Work that meets, but does not exceed, the minimum expectations
70% – 74%: Work that exhibits reasonable effort but falls short of the minimum expectations
60% – 69%: Work of poor quality that shows little effort or understanding of the task
below 60%: Work that exhibits no effort or understanding of the task
1. CRITICAL ANALYSIS OF EVALUATION DESIGN (20%): DUE FEB. 21
In this assignment, you will put your understanding of evaluation paradigms and design elements to the test. In particular, your task will be to read through an existing evaluation and to identify the paradigmatic assumptions as well as the strengths and weaknesses of the design in relation to the needs of the stakeholders involved.
To begin, select an evaluation from the four examples provided:
Your critical analysis should include the following sections (3-4 double-spaced pages if 12pt font & Times New Roman Font):
Provide a brief introduction to the evaluand(1 –2 paragraphs)
Summarize the logic model of the evaluandusing the WKKE approach (~1 page, illustration optional)
Summarize the design of the design of the evaluation with special attention to (~1 page):
Data collection strategies
Summarize the findings/outcomes of the evaluation (~0.5 page)
Critically engage with the design of the study (~1 page)
What was the primary paradigm informing the design of the evaluation? What forms of knowledge were privileged and suppressed through the evaluation?
What were the strengths of the design in relation to the questions driving the evaluation?
What were the weaknesses of the design in relation to the questions driving the evaluation?
2. DATA VISUALIZATION EXERCISE (10%): DUE MARCH 21
Being able to effectively visualize the data in an evaluation is perhaps the most important and least appreciate facet of evaluation work. During class, you will be given data and a hypothetical audience and then asked to represent the findings in way that is appropriate to that audience. Following class, your assignment is to find a data visualization in news media, an evaluation report, or an academic article (some helpful links can be found here - but you are not limited to these sources). Copy and paste a copy of the visualization into a Word document and include a source link to the image. If a link is not available, you may upload a separate file and include a citation in the Word document.
Next, in about two pages, critically analyze the data visualization using Cairo’s “five qualities of great visualizations” as a guide. In particular, answer the following questions in the Word document where you have pasted your visualization (This assignment is to be completed by students individually, i.e., this is not part of your project groups):
Who appears to be the target audience(s) for this visualization? What qualities about the visualization and its context (i.e., where it is published) support your conclusion?
What story - or stories - is the author of the visualization intending to tell? What insights were gained by visualizing the data in this way that text alone could not accomplish?
What are some alternative ways to interpret the visualization that may not have been intended by the author? In what ways, if any, is the visualization potentially misleading?
How, if at all, might the interpretation of the visualization change depending on who is reading the visualization? What factors might influence these changes (e.g., the identities of the reader, level of education, cultural familiarity)?
What else strikes you about the visualization? How might Cairo suggest that it be improved based on his framework for creating great visualizations?
3. EVALUATION DESIGN PROJECT: DUE DATES - MULTIPLE (SEE BELOW)
This project is an opportunity for you to design an evaluation of a program or policy within the realm of your professional interests. Given the multiple components, it is strongly recommended that you collaborate on this project. In addition, evaluations are often carried out by a team, and thus it is essential to gain experience with the social dynamics involved with designing an evaluation.
The project consists of multiple assignments and due dates:
PROGRAM/POLICY IDENTIFICATION & PARADIGMATIC APPROACH (15%): DUE FEB. 7
In this initial phase of the project, you will identify the specific program or policy that will serve as the specific context for your evaluation design. Your task will be to provide background information on the program and to identify and any prior research that provides relevant insight to the issue. In addition, you will highlight the specific evaluation paradigm(s) guiding the project. Be sure to include the following in about 750 - 1,250 words (3 - 5 double-spaced pages if using 12-pt. Times New Roman font and standard 1-inch margins):
Identify the specific evaluand (i.e., program, policy, idea, product, project) for your evaluation design. Provide an overview of the evaluandand give some context (e.g., history, aims/purposes, sources of funding). ~2 pages
Summarize any prior studies of the evaluand, or, if no such studies exist, point to relevant prior evaluations or research that give some indication of what you might expect to find. ~1 - 2 pages
Describe the primary evaluation paradigm that best fits the set of assumptions and approaches of your design, including a discussion of why such a paradigm is appropriate to this case. ~1 page
Identify group members (if applicable)
Upload as a Word or PDF to Canvas(only one per group is needed)
Note: Those working in groups will turn in a single assignment. However, at the end of the semester each group member will submit a summary of their contribution to each phase of the project, which will be circulated and signed off by each member of the group.
IDENTIFYING STAKEHOLDERS, QUESTIONS, AND DATA COLLECTION (20%): DUE MARCH 7
This phase involves specifying the various stakeholders in the evaluation and the particular needs being addressed through the evaluation design. Next, you will specify the questions that will guide the evaluation, as well as describing the data collection strategies most appropriate to addressing the questions. Please include the following:
Identify the range of stakeholders connected to your evaluand (~1 page)
Be as specific as possible
Discuss the different stakes
Given a general description of the type of evaluation (e.g., needs assessment, outcomes-based) and design (e.g., quasi-experimental, mixed-methods) you will use to evaluate your evaluand (~1-page)
Discuss why this type and design is appropriate given the objectives of the evaluand.
Pose a set of questions that will guide your evaluation (~1.5 - 2 pages)
Under each question, discuss why the question is important to your stakeholders (likely involves some assumptions –make those known)
Discuss the data collection strategies you will use to address each question and why they are appropriate to the question
PRESENTATION (20%): IN CLASS, APRIL 18 & 25
The presentations will take place in class during the last two weeks of the semester (not including finals week). In this assignment, you will practice the craft of communicating your design to a diverse room of stakeholders.
FINAL PROPOSAL (15%): DUE APRIL 29
The final proposal will consist of revised versions of your program/policy identification & paradigmatic approach and the identifying stakeholders, questions, and data collection assignments. In doing so, you will be asked to include a cover letter (addressed to me) that clearly specifies how you addressed my suggestions as well as any other changes you made. In addition, you will be asked to include a dissemination and action strategy to the report that specifies how you will communicate the findings of the evaluation to various stakeholders, and the specific recommendations that you expect to make depending on the findings.
The following serves as the primary text for the course:
Mertens, Donna and Amy T. Wilson. (2018). Program Evaluation Theory and Practice, Second Edition: A Comprehensive Guide. New York, NY: The Guilford Press.
Additional assigned readings will consist of journal articles and selected book chapters posted on the Course Outline and Readings page.
If any student requires specific accommodations please do not hesitate to speak with me at any point during the semester. This includes accommodations related to the curriculum, instruction, evaluations, or any other factors that would otherwise prohibit your full participation in this course. Any questions or concerns students have about this matter will be held confidential to the best of my ability. In order to receive specific accommodations in this course, you must provide me with a Letter of Accommodation from the Disability Resource Center (Room 2, Alumni Gym, 859-257-2754 for coordination of campus disability services available to students with disabilities.
All instances of academic dishonesty will be addressed according to standard UK policies on academic integrity. Please familiarize yourself with these expectations and the Code of Student Rights and Responsibilities.
STATEMENT REGARDING DISCRIMINATION
The University of Kentucky faculty are committed to supporting students and upholding the University's non-discrimination policy.
Discrimination is prohibited at UK. If you experience an incident of discrimination we encourage you to report it to Institutional Equity & Equal Opportunity (IEEO) Office, 13 Main Building, (859) 257-8927.
ACTS OF SEX- AND GENDER-BASED DISCRIMINATION OR INTERPERSONAL VIOLENCE
If you experience an incident of sex- or gender-based discrimination or interpersonal violence, we encourage you to report it. While you may talk to a faculty member or TA/RA/GA, understand that as a "Responsible Employee" of the University these individuals MUST report any acts of violence (including verbal bullying and sexual harassment) to the University's Title IX Coordinator in the IEEO Office. If you would like to speak with someone who may be able to afford you confidentiality, the Violence Intervention and Prevention (VIP) program (Frazee Hall – Lower Level), the Counseling Center (106 Frazee Hall), and the University Health Services are confidential resources on campus.
ESTIMATED COURSE OUTLINE
*Please note that I will likely supplement readings from the textbook throughout the semester. Please check the outline regularly for up-to-date reading assignments.
SECTION 1: ideas and paradigms
1-10-2019 COURSE INTRODUCTION
Read the entire syllabus
1-17-2019 The big picture of evaluation
Mertens & Wilson text, Chapters 1 & 2
1-24-2019 Evaluation Paradigms I: post-positivism & pragmatism
Mertens & Wilson text, Chapters 3 & 4
Post-positivist evaluation example (see also pp. 65-69 in Mertens & Wilson text)
Pragmatic evaluation example (see also pp. 121-124 in Mertens & Wilson text)
1-31-2019 Evaluation Paradigms II: Constructivism & social justice
NOTE: We will not meet in class this week. Please do the readings before watching the following video lecture.
Mertens & Wilson text, skim chapters 5 & 6 (By skim, I mean you should read to gain a clear understanding of the general principles of constructivist and social justice paradigms of evaluation, as well as the practical examples through which these paradigms operate in practice.)
Please watch the following video of Jeff Duncan-Andrade’s lecture, “Note to Educators: Hope Required When Growing Roses in Concrete.” I recommend the entire two-hour lecture, but play close attention to the following time-spans: 16:00 to 46:35, 104:00 to 108:00, 147:50 to 152:48
After watching the lecture and completing the readings, consider the following questions and be prepared to discuss:
In what ways, if at all, do Duncan-Andrade’s ideas about the ways schools evaluate students reflect a constructivist paradigm as discussed in ch. 5 of Mertens & Wilson?
In what ways, if at all, do Duncan-Andrade’s ideas about the ways schools evaluate students reflect a social justice paradigm as discussed in ch. 6 of Mertens & Wilson?
Does Duncan-Andrade reinforce any of the assumptions of the post-positivist paradigm? How does he challenge post-positivist paradigms of educational evaluation?
section 2: evaluation design
2-7-2019 evaluation design I: stakeholder engagement & formulating questions
Mertens & Wilson text, Chapters 7 & 8
Program identification assignment due
2-14-2019 evaluation design II: experimental & non-experimental strategies
2-21-2019 evaluation design III: data collection & working with human participants
Mertens & Wilson text, Chapters 10 & 11
Critical analysis of evaluation design due
section 3: analysis and dissemination
2-28-2019 Analytic strategies
Mertens & Wilson text, Chapter 12
3-7-2019 data visualization
Chapter 2 from The Truthful Art: Data, Charts and Maps for Communication by Alberto Cairo (2016).
Identifying stakeholders assignment due
3-14-2019 NO CLASS / SPRING BREAK
3-21-2019 dissemination strategies
Mertens & Wilson text, Chapter 13
DataViz assignment due
Section 4: putting it all together
3-28-2019 group case study 1
Evaluation of School Vouchers in Indiana
4-4-2019 NO CLASS / Project WEEK
4-11-2019 project workshop
Presenting results (Example slides)