Implemented by the Aga Khan Foundation, Schools2030 is a globally informed, locally rooted, 10-year learning improvement program, working with 1,000 schools in 10 countries: Afghanistan, Brazil, India, Kenya, Kyrgyzstan, Pakistan, Portugal, Tajikistan, Tanzania, and Uganda. Schools2030’s overarching objective is to support these countries in achieving the targets set for Sustainable Development Goal (SDG) 4: “Quality Education for All”.
In 2022, Khulisa was commissioned as the Schools2030 global evaluation partner to design an impact evaluation of the Schools2030 program and collect the baseline data for this evaluation. The baseline evaluation will be followed by a midline evaluation in 2026-2027 and an endline evaluation in 2029-2030. Khulisa has had to use a flexible approach in designing such a complex impact evaluation.
Schools2030 is working with learners in three age groups – 5-year-olds (early childhood development), 10-year-olds (primary school), and 15-year-olds (secondary school) – and includes three main components: 1) Assessment, in which teachers assess their learners on literacy, numeracy, and a number of socio-emotional learning indicators at both the beginning and end of each year; 2) Innovation, in which teachers are supported to experiment and design their own solutions (using human-centered design methodology) to the challenges their learners are facing; and 3) Showcasing, in which government officials attend showcasing events to learn about the solutions that teachers have innovated.
Schools2030 is an immensely ambitious program that seeks to transform education and influence education policy in ten countries, at a time in history when the educational needs of children are evolving at a dizzying rate. Schools2030 is an incredibly innovative project in that it implements both a bottom-up and top-down approach by working directly with teachers to innovate education solutions, while involving government officials and policymakers throughout the program to both learn from teachers and better understand children’s learning needs in their region. The Schools2030 program itself is incredibly complex, therefore the evaluation, which seeks to determine Schools2030’s level of impact over a decade, is incredibly complex as well.
“It’s a complicated design and a complex context that one’s working in, across all these different educational systems, ministers, and approaches – different schools, different cohorts, different ages, different teachers, and different countries implemented in a slightly different way,” says Margie Roper, Director of Khulisa’s Education Division. “Evaluations like this take more than one brain, more than one set of skills, and more than one kind of thinking.”
Schools2030 is unlike any evaluation that Khulisa has implemented before, and the challenges to implementing it seemed almost insurmountable at first. Fortunately Khulisa’s Education Division has implemented quite a few large, complicated, mixed-method evaluations in the past, and was able to draw upon that experience when figuring out the best approach for the Schools2030 impact evaluation.
Implementing a Complex Impact Evaluation: Baseline Phase
Khulisa’s baseline data collection for Schools2030 involves a combination of qualitative and quantitative evaluation tools.
“The teachers are the ones going through the training and receiving the Schools2030 program,” explains Jesse Webb, lead evaluator for Schools2030. “So they’re also our main denominator. We’re surveying teachers from a sample of the schools to understand their experience of the program and what’s working, what’s not working, etc. We also survey the principals of the schools where the teachers are teaching, so that we can understand the impact of the program at a contextual level. And then we are doing assessments with learners in the classes of the teachers that we’re surveying.” The data collection also includes classroom observation, as well as interviews with education stakeholders at regional and national levels. This phase of data collection will provide a baseline of classroom and school environments, student outcomes, and teacher experience (including innovative capacity, approaches and challenges), against which we can measure the impact of Schools2030 over time.
Khulisa is carrying out the data collection in each of the ten countries, in a sample of schools that are part of the Schools2030 program, as well as a corresponding sample of counterfactual schools that are not receiving the Schools2030 interventions. The sampling method for the counterfactual schools is slightly different by country, depending on the way Schools2030 is being implemented in each country.
Allowing Space for Difference
An important step in planning this impact evaluation was the realization that Schools2030 can’t be evaluated in a “traditional” way. “When approaching an evaluation from a social science perspective, you want to eliminate as much difference as possible [between the country samples]. And that’s impossible to do with Schools2030 – there is so much that’s different,” says Jesse. “There is so much that’s contextual, and that’s a strength of the program. That’s one of the things that the development world has recognized: the strength of contextualizing programs and really adapting them to the environment.”
Certain indicators, like whether learners are doing well or poorly in literacy and numeracy, can be comparable across schools and across countries. The Schools2030 baseline seeks to measure those indicators while also “allowing the evaluation to find unique, niche ingredients that are relevant for how the program has been implemented in the different countries and allowing room for the flexibility,” Jesse says.
“It’s not a linear solution to these complex global challenges, particularly with education,” adds Margie. “Education is no longer didactic – someone standing up there saying ‘This is the knowledge.’ Education is around empowering and equipping learners. Teachers’ teaching practice has had to change, and that’s where the human–centered design aspect comes in…Education is completely different to what it was even 10 years ago, and evaluations have had to adapt to these differences.”
Not Fearing Failure
Khulisa’s experience has empowered the team to innovate without being afraid that things won’t work – i.e., that the impact evaluation won’t show the results that everyone hopes for. “We’re not fearful of failure because we are coming with a very rigorous, theoretical, practical and implementable approach,” says Margie. “We’ve got the experience, we know what we’re doing. And if the program doesn’t work, there’s not going to be just one reason why we’re not seeing the evidence or impact. And we are trying to eliminate the possibility that any failure is caused by the design of the evaluation.”
Margie also credits the strong relationship between Khulisa and the Aga Khan Foundation for the success of the project so far. Designing such a complex evaluation depends on “having good, open working relationships with clients who are also very open to the innovation, the adaptability and the complexity of it, and they’re as challenged and excited and motivated by [the evaluation] as I am.”
The length and timeframe of the school years differs in the Schools2030 countries, and Khulisa plans to complete the data collection for the baseline phase of the impact evaluation around August of 2024. We look forward to providing further updates on the Schools2030 program as it evolves in the coming years.