Part 3 in a multi-part series about analytical methods used in evaluations. All of the methods featured in this series appear in The Magenta Book Annex A.
We were recently inspired by The Magenta Book, the U.K. government’s comprehensive “guidance on what to consider when designing an evaluation”, to feature a series of analytical methods and how they work best in an evaluation. This week’s method, Difference-in-Difference (DiD), involves comparing trends over time between two groups.
DiD is a quasi-experimental design that typically involves two comparable groups: an intervention group, which has experienced a particular intervention, and a comparison group, which has not experienced the intervention. Change within the two groups is measured over time, both before and after the intervention. A comparatively higher rate of change within the intervention group suggests the intervention has an impact.
A graphical explanation of DiD from Columbia Public Health.
Khulisa used a modified version of DiD in a 2015-2018 study, “Impact Evaluation of the Western Cape Emergent Literacy Intervention in South Africa”. The impact evaluation studied the effectiveness over time of an early language and literacy program on Grade R (kindergarten) learners and teachers. The study found that the intervention initially increased language and literacy outcomes, but those outcomes diminished and eventually disappeared over time.
The figure above, included in the project report, shows how the intervention group (solid line) outperformed the comparison group (dotted line) during the first two waves of the evaluation. During the evaluation’s third wave, when the learners had moved on to Grade 1, the “intervention learners were at the lower end of performance compared to the comparison group learners.” Hence, this intervention did not have a sustainable impact.
Are you interested in learning more about the variety of analytical methods used in evaluations? Browse them all in The Magenta Book Annex A.