EPE/EDP/KHP 520: INTRODUCTION TO EVALUATION
Instructor: Joseph J. Ferrare, Ph.D.
Spring 2019, Thursdays 11am - 1:30pm, DH 135
Office: 145C Taylor Education Building
Office Hours: Thursdays 10am - 11am & by appointment
Syllabus in PDF
Evaluation has become a ubiquitous feature of modern life. Organizations - such as schools, nonprofits, and businesses - are increasingly required to demonstrate that certain programs or policies are achieving desired outcomes in order to justify continued funding. In this context, evaluation refers to the use of empirical evidence to assess the outcome(s) associated with a program, practice, or policy. This state of affairs makes it crucial to understand the principles that inform evaluation work, including the technical details of evaluation design as well as the social and political contexts that influence how evaluations are carried out and acted upon.
This course surveys different approaches to evaluation that are widely used across the social sciences and in non-academic settings. We will begin with an overview of the different general approaches (or “paradigms”) to evaluation and then gradually work through the various steps of the evaluation process. This includes: engaging stakeholders; framing research questions; working with human participants and IRB; data collection strategies; analysis; visualization; and dissemination. Along the way, we will critically engage with a variety of examples of evaluations and explore how these projects inform programs, policies, and practices in the education sector and beyond.
Students who successfully complete the requirements of this course should expect the following:
To be able to critically analyze the elements of evaluation design from multiple perspectives (i.e., paradigms);
To understand the basic steps involved in designing an evaluation of a program or policy;
To become proficient at identifying relevant stakeholders in an evaluation context and adapting communication strategies appropriately.
The format for our class meetings will consist of interactive lectures, small group work, and individual exercises. The lectures are designed to provide an introduction to key concepts in evaluation, and to offer examples of these concepts as they apply to education and social science more generally. Please note that “interactive” lecture means that during this time I will pose questions directly to individual students with the intent of generating discussion. These questions will assume you have completed the assigned readings and have been engaged in class activities throughout the term. Finally, the small group and individual exercises will give you opportunities to practice thinking through decisions in the evaluation process within the context of practical scenarios.
REQUIREMENTS AND MODES OF EVALUATION
Your work in this course will be evaluated through multiple assignments. The specific modes of evaluation and corresponding grading weights are described below. Grades will be assigned using the following scale:
A: 90% – 100%, B: 80% – 89%, C: 70 – 79%, D: 60% – 69%, E: below 60%
When evaluating your work I will consider criteria specific to each assignment. In general, though, I consider grade ranges to meet the following generic standards:
95% – 100%: Exemplary work that exhibits mastery over the task
90% – 94%: Excellent work that approaches mastery but falls short in one key area
85% – 89%: High quality work that has ample room for improvement
80% – 84%: Work that exceeds minimum expectations but contains a number of mistakes or lacks quality in key areas
75% – 79%: Work that meets, but does not exceed, the minimum expectations
70% – 74%: Work that exhibits reasonable effort but falls short of the minimum expectations
60% – 69%: Work of poor quality that shows little effort or understanding of the task
below 60%: Work that exhibits no effort or understanding of the task
1. CRITICAL ANALYSIS OF EVALUATION DESIGN (20%): DUE FEB. 21
In this assignment, you will put your understanding of evaluation paradigms and design elements to the test. In particular, your task will be to read through an existing evaluation and to identify the paradigmatic assumptions as well as the strengths and weaknesses of the design in relation to the needs of the stakeholders involved.
2. DATA VISUALIZATION EXERCISE (10%): DUE MARCH 21
Being able to effectively visualize the data in an evaluation is perhaps the most important and least appreciate facet of evaluation work. In this exercise, you will be given data and a hypothetical audience and then asked to represent the findings in way that is appropriate to that audience.
3. EVALUATION DESIGN PROJECT: DUE DATES - MULTIPLE (SEE BELOW)
This project is an opportunity for you to design an evaluation of a program or policy within the realm of your professional interests. Given the multiple components, it is strongly recommended that you collaborate on this project. In addition, evaluations are often carried out by a team, and thus it is essential to gain experience with the social dynamics involved with designing an evaluation.
The project consists of multiple assignments and due dates:
PROGRAM/POLICY IDENTIFICATION & PARADIGMATIC APPROACH (15%): DUE FEB. 7
In this initial phase of the project, you will identify the specific program or policy that will serve as the specific context for your evaluation design. Your task will be to provide background information on the program and to identify and any prior research that provides relevant insight to the issue. In addition, you will highlight the specific evaluation paradigm(s) guiding the project.
IDENTIFYING STAKEHOLDERS, QUESTIONS, AND DATA COLLECTION (20%): DUE MARCH 7
This phase involves specifying the various stakeholders in the evaluation and the particular needs being addressed through the evaluation design. Next, you will specify the questions that will guide the evaluation, as well as describing the data collection strategies most appropriate to addressing the questions.
PRESENTATION (20%): IN CLASS, APRIL 18 & 25
The presentations will take place in class during the last two weeks of the semester (not including finals week). In this assignment, you will practice the craft of communicating your design to a diverse room of stakeholders.
FINAL PROPOSAL (15%): DUE APRIL 29
The final proposal will consist of revised versions of your program/policy identification & paradigmatic approach and the identifying stakeholders, questions, and data collection assignments. In doing so, you will be asked to include a cover letter (addressed to me) that clearly specifies how you addressed my suggestions as well as any other changes you made. In addition, you will be asked to include a dissemination and action strategy to the report that specifies how you will communicate the findings of the evaluation to various stakeholders, and the specific recommendations that you expect to make depending on the findings.
The following serves as the primary text for the course:
Mertens, Donna and Amy T. Wilson. (2018). Program Evaluation Theory and Practice, Second Edition: A Comprehensive Guide. New York, NY: The Guilford Press.
Additional assigned readings will consist of journal articles and selected book chapters posted on the Course Outline and Readings page.
If any student requires specific accommodations please do not hesitate to speak with me at any point during the semester. This includes accommodations related to the curriculum, instruction, evaluations, or any other factors that would otherwise prohibit your full participation in this course. Any questions or concerns students have about this matter will be held confidential to the best of my ability. In order to receive specific accommodations in this course, you must provide me with a Letter of Accommodation from the Disability Resource Center (Room 2, Alumni Gym, 859-257-2754 for coordination of campus disability services available to students with disabilities.
All instances of academic dishonesty will be addressed according to standard UK policies on academic integrity. Please familiarize yourself with these expectations and the Code of Student Rights and Responsibilities.
STATEMENT REGARDING DISCRIMINATION
The University of Kentucky faculty are committed to supporting students and upholding the University's non-discrimination policy.
Discrimination is prohibited at UK. If you experience an incident of discrimination we encourage you to report it to Institutional Equity & Equal Opportunity (IEEO) Office, 13 Main Building, (859) 257-8927.
ACTS OF SEX- AND GENDER-BASED DISCRIMINATION OR INTERPERSONAL VIOLENCE
If you experience an incident of sex- or gender-based discrimination or interpersonal violence, we encourage you to report it. While you may talk to a faculty member or TA/RA/GA, understand that as a "Responsible Employee" of the University these individuals MUST report any acts of violence (including verbal bullying and sexual harassment) to the University's Title IX Coordinator in the IEEO Office. If you would like to speak with someone who may be able to afford you confidentiality, the Violence Intervention and Prevention (VIP) program (Frazee Hall – Lower Level), the Counseling Center (106 Frazee Hall), and the University Health Services are confidential resources on campus.
ESTIMATED COURSE OUTLINE
*Please note that I will likely supplement readings from the textbook throughout the semester. Please check the outline regularly for up-to-date reading assignments.
SECTION 1: ideas and paradigms
1-10-2019 COURSE INTRODUCTION
Read the entire syllabus
1-17-2019 The big picture of evaluation
Mertens & Wilson text, Chapters 1 & 2
1-24-2019 Evaluation Paradigms I: post-positivism & pragmatism
Mertens & Wilson text, Chapters 3 & 4
1-31-2019 Evaluation Paradigms II: Constructivism & social justice
NOTE: We will not meet in class this week. I will prepare a video lecture that you will be responsible for watching outside of class.
Mertens & Wilson text, Chapters 5 & 6
section 2: evaluation design
2-7-2019 evaluation design I: stakeholder engagement & formulating questions
Mertens & Wilson text, Chapters 7 & 8
Program identification assignment due
2-14-2019 evaluation design II: experimental & non-experimental strategies
Mertens & Wilson text, Chapter 9
2-21-2019 evaluation design III: data collection & working with human participants
Mertens & Wilson text, Chapters 10 & 11
Critical analysis of evaluation design due
section 3: analysis and dissemination
2-28-2019 Analytic strategies
Mertens & Wilson text, Chapter 12
3-7-2019 data visualization
Chapter 2 from The Truthful Art: Data, Charts and Maps for Communication by Alberto Cairo (2016).
Identifying stakeholders assignment due
3-14-2019 NO CLASS / SPRING BREAK
3-21-2019 dissemination strategies
Mertens & Wilson text, Chapter 13
DataViz assignment due
Section 4: putting it all together
3-28-2019 group case study 1
4-4-2019 NO CLASS / Project WEEK
4-11-2019 Group case study 2
4-18-2019 PROJECT PRESENTATIONS
4-25-2019 PROJECT PRESENTATIONS