Technical Approaches for Conducting Impact Evaluations

Centre for Learning on Evaluation and Results at the University of Witwatersrand, Johannesburg, South Africa

The Course runs: from June 26 – July 5, 2013

Deadline for applications for scholarships:  See application form

Deadline for applications for admissions: May 10, 2013

Information: Please e-mail Thulile.Zondi@wits.ac.za

Technical Approaches for Conducting Impact Evaluations

Course to be held in Jinja, Uganda.

Print-friendly version: Download here

Course Overview:

This technical course introduces impact evaluation as a key instrument for determining project/program effectiveness, informing policy development and improving program designs. It covers commonly used econometric and statistical methods to evaluate the impacts of social and other programs in developing countries. The course includes both randomized and non-randomized methods.
Delivery of the course consists of lectures, class exercises, and group work. Many of the exercises will involve case studies using data from actual programs that are discussed in the lectures. Participants will also have an opportunity to determine options for evaluations for their own programs.
This course is highly technical. In order to benefit from the course, participants should review the prerequisites before applying. Please send any questions about the technical nature of the course to the course organizers to determine if this is a good fit for you. Though the course is technical, it will also make links to policy implications and practical applications.
Delivery of the course consists of lectures, class exercises, and group work. Many of the exercises will involve case studies using data from actual programs that are discussed in the lectures. Participants will also have an opportunity to determine options for evaluations for their own programs.
The course will be taught in English

Course Objectives:

At the end of the course participants will be able to:

• Understand what impact evaluation is and the different techniques used for impact evaluations
• Make judgments about the situations (and/or interventions) when impact evaluation is and is not appropriate to use
• Know which kind of evaluation or decision-making questions that impact evaluation is best suited to answer
• Design impact evaluation studies using appropriate methodologies
• Analyse data in accordance with the “parameters of interest” (ATE-average treatment effect, ATT-average impact of treatment on the treated, ITT-intention-to-treat, etc.)
• Describe in detail how randomized trials can estimate parameters of interest
• Employ regression methods that do not require randomized data (Difference-in- differences and within estimators, Instrumental variable methods, Regression discontinuity methods, Matching methods)
• Employ regression methods that do not require randomized data (Difference-in- differences and within estimators, Instrumental variable methods, Regression discontinuity methods, Matching methods)
• Provide advice on how to choose the sample size and attain statistical power

Course Prerequisites:

• To fully benefit from the course, participants need to be familiar with statistics, multiple regression analysis and possess some knowledge of a statistical software package (e.g., STATA, SAS, SPSS, or R). Knowledge of econometrics is highly beneficial.
• All participants applying for the course need to complete the Pre Course Quiz, which will be sent to the applicants after they send their applications.
• Priority will be given to teams (2 or more individuals) that have impact evaluations underway, recently completed and/or are considering, designing, or implementing impact evaluations. Members of the team may include technical staff, as well as policymakers, policy analysts, and program and evaluation managers who have the required technical skills to benefit from the course. Each individual will have to apply separately, but should reference the team in the application form.
• All participants must be proficient in English.
• All participants applying for the course need to fill in the Pre Course Survey and Quiz.
• Priority will be given to teams (2 or more individuals) that have impact evaluations underway, recently completed and/or are considering, designing, or implementing impact evaluations. Members of the team may include technical staff, as well as policymakers, policy analysts, and program and evaluation managers who have the required technical skillsto benefit from the course. Each individual will have to apply separately, but should reference the team in the application form.
• All participants must be proficient in English.

Application:

Applications will be considered as they are submitted. Applicants are encouraged to submit applications earlier that the deadline. Please note that submitting an application and completing the Pre Course Quiz does not mean automatic enrollment in the course. Applicants will be notified of their enrollment status within one week following the application deadline. We will acknowledge receipt of your application within two business days of your submission. If you do not receive a note from us indicating that we received your application, please contact us at thulile.zondi@wits.ac.za. The final deadline for complete applications will be May 10, 2013.
Application Forms are electronic and are available here (Applicants will receive a pre-course survey after submitting the application form.).

Course Fee:

The base price for the course fee is US$ 2500. The bulk of the fee is for (almost) two weeks of housing. Discounts will be given to participants from certain organizations, as noted below. Payment policies and procedures will be provided to selected applicants.

Fee structure:

Included in the fee:

• Tuition
• Training booklets and electronic material
• Housing at the Hotel in Jinja, Uganda from June 25th to July 6th
• Transportation to and from Jinja airport upon arrival and departure
• Breakfast, lunch and dinner each day
• Welcome reception/dinner
• Limited visa assistance (e.g., letters of invitation)
  
  The following items are NOT included in the fee:

• Transportation from your home to Uganda (airfare, public transportation, train, taxi, etc)
• Transportation outside of the course
• Per diem
• Travel insurance (for airfare, etc.)
• Visa and passport application fees
• Other miscellaneous costs (shopping, telephone charges, etc.)

Scholarships:

Only a limited number of scholarships will be available and will be assigned competitively. The selection process will be determined through the application process and will be based on the following criteria: potential benefit to you/your team in participating and applying the lessons to ongoing or planned impact evaluations, regional priorities of our scholarship funders, inability to pay the course fees. Please indicate if you are applying for a scholarship in the Application form. Applications will be reviewed and a subset of participants will be contacted about the possibility of scholarships.

Detailed Course Content:

This section of the course is targeted at technical staff engaged in evaluations as well as program and evaluation managers with sufficient degree of technical expertise in statistics and econometrics. The topics covered include:

• Impact Evaluation in the policy and program cycles: This session gives a general overview of how evidence and impact evaluations are implemented within the policy cycle, highlighting both its strengths and weaknesses. In this discussion, we explore the development of theories of change for programs. We also identify which types of evaluation questions and policy decisions best align to impact evaluation.
• The need for a counterfactual: This session provides an introduction to counterfactuals. Impact evaluations seek to answer the question: What would the situation be with or without the intervention? What is the causal impact of the program (or project or policy) on the outcomes of interest?
• Threats to Validity: This session will review impact evaluations that are very instructive regarding how evaluations should be implemented and what types of “threats” can arise regarding both internal and external validity, and how to address these threats.
• Regression Analysis: Regression analysis is the key statistical tool used in almost all impact evaluations. This session provides a refresher for regression analysis and introduces the basic commands in the software for several regression models that are often used in impact evaluation, including linear regression models, probit/logit models, and random/fixed effects models.
• Sample Size, Sample Design, and the Power Calculations: A key issue in any randomized evaluation is how large of a sample is needed, and how the sampling should be done. The session provides detailed information on how to answer these questions, and explains some trade-offs that the evaluator will face.
• Introduction to Regression Estimators: The session will present the four simplest estimators, as well as the assumptions required regarding the process by which people are selected and participate in impact evaluations to ensure that they yield unbiased estimates of impacts when program participation is not randomized. The four estimators are: cross-section, before-after, difference-in-differences, and within estimator.
• Implementing Randomized Evaluations: In theory, randomized control trials (RCTs) provide accurate estimates of program impacts. Yet, in practice there are many things that need to be considered to obtain accurate estimates from RCTs. Also, many things can go wrong with RCTs. This session provides more detailed advice on how to conduct an RCT and discusses potential solutions to problems that may arise.
• Difference-in-Differences and Within Estimators: The This session will explain the importance of the fixed effect errors structure for before-after and difference-in-difference methods, and discuss other issues relating to these two estimators (given that the assumptions needed for the cross-section estimator to be valid are unlikely to hold).
• Introduction to Matching Methods: This includes Cross-sectional matching estimators; Rosenbaum and Rubin (1983) theorem for addressing curse of dimensionality; implementation of propensity score matching estimator; nearest-neighbour methods; nonparametric methods; and determination of the overlap support region.

Certification:

Participants will receive a CLEAR Certificate.

Listing of Experts on 3ie Roster:

Those bidding for 3ie evaluations in Africa are required to include local team members in analytical positions. Participants who meet the full requirements of the course will be listed on the 3ie website as having successfully completed the course, and so may be included in study teams.