Realist evaluation, a form of theory-driven evaluation, evaluates a program by asking, “What works in which circumstances and for whom?” rather than merely, “Does it work?” Realist evaluation is effective in circumstances when an intervention has varying outcomes for different groups in different circumstances.
Khulisa recently used a realist evaluation approach for the second phase of a project for the Catholic Institute of Education in South Africa, evaluating the theory of change developed during the first phase of the project five years ago. The project evaluates child safeguarding protocols in a wide spectrum of Catholic schools all over South Africa; some of the schools are private, some are no-fee, some are on church land, some are not on church land, some are mission schools, etc.
“We knew [the theory of change] was not going to work in all schools or for all school principals or for all teachers,” said Margie Roper, Director of Khulisa’s Education Division. “So our instruments had to ask questions really covering the context, mechanisms, and outcomes. But not in a way that was like: What are the outcomes? You have to ask it in a different way, because you’re trying to get people’s experiences, thoughts, and beliefs.”
Analyzing data from a realist evaluation, which is usually qualitative, also requires a unique theoretical approach. “When analyzing [the data] and trying to document these chains of events – the context, the mechanism, the outcome – and looking for trends or patterns and trying to extract the commonalities and differences, what helped was to really draw them out – like boxes and lines,” Margie said. “What’s the context? What’s the mechanism? What was the outcome? And then actually saying that it worked for these people or didn’t work for that person. And then why did it not work for that person?”
Realist evaluation is a theoretical, not methodological – the choice of methodology depends on the specific circumstances of the evaluation. For more tips, recommendations, and specific examples, read about realist evaluation on BetterEvaluation.org.