Second in an occasional series about the processes and challenges of implementing SUN LE’s evaluation of the SUN program, a large, multisectoral nutrition program in Zambia. Read Part 1.
Since November 2018, Khulisa and its subcontractors have worked with the U.S. Agency for International Development (USAID) and the Government of the Republic of Zambia (GRZ) to implement a project called Scaling Up Nutrition Learning and Evaluation (SUN LE). This project provides learning and evaluation support to the GRZ’s Scaling Up Nutrition (SUN) program, a multi-sectoral nutrition program, as part of a drive to achieve an essential goal: reducing stunting (impaired growth and development resulting from poor nutrition) – a serious problem among children in Zambia – by 14% over seven years. SUN LE supports the SUN program to generate and disseminate evidence to facilitate learning and adaptive management, and to provide improved accountability for achieving the SUN goal of reducing childhood stunting.
The SUN LE program implements various research activities: 1) baseline, midline, and endline surveys of 7,500 households across 30 districts in Zambia; 2) a number of focused studies; and 3) multiple, large-scale Performance Assessments – also called Performance Audits (PAs) – to assess the adequacy of SUN core systems, structures, and capacities at national and subnational levels. To date, SUN LE has now completed two rounds of PAs in 2020 and 2022.
The PAs serve a distinctly different purpose than the household surveys: The PAs are about assessing the effectiveness of the SUN program’s operations, rather than the end results of the program (i.e., reducing stunting). “The PA is all about the supply of the program to the households,” explained Mary Pat Selvaggio, Director of Khulisa’s Health Division. “The household survey is all about how households have benefited from that supply.”
Designing the SUN LE Performance Audit
SUN is a large, multisectoral program, impacting millions of people and involving multiple GRZ agencies and implementing partners. Conducting a performance audit of such a complex program involved many challenges, the first of which was to define what a PA is.
A performance assessment/audit is an independent examination of a program’s functions, operations, management systems, and procedures against defined standards, criteria, benchmarks, or expectations, which determines whether the program is meeting its established performance criteria and standards.
“At the beginning, everyone we talked to had a different idea of what a performance audit was,” said Mary Pat. The SUN LE project team also found that PAs conducted in the private sector are overwhelmingly financially focused. “So we aimed to build on existing and proven PA guidance and tools that were used in public sector programs.”
For the SUN PAs, Khulisa used three internationally recognized frameworks to design its PA tool and analytical approach: the Primary Healthcare Performance Initiative Framework, the Guide to Evaluating Collective Impact, and the Scaling Up Nutrition Global Strategy Standards.
Zambia’s SUN program involves six different GRZ ministries: Health, Agriculture, Fisheries and Livestock, Water Development and Sanitation, Education, and Community Development. To determine whether this wide-ranging program was meeting its performance criteria and standards, the SUN LE team assessed performance on 31 standards across 8 domains:
- Governance and Leadership (5 standards)
- Adjustment to Population Needs (3 Standards)
- Equipment, Supplies, and Service Infrastructure (6 standards)
- Information Systems (2 standards)
- Workforce (3 standards)
- Funding (3 standards)
- Population Nutrition Management (4 standards)
- Service Organisation (5 standards)
The team then developed a rubric in which each standard was scored on a scale of 1 (worst) to 4 (best, close to ideal, or ideal). Data was collected using qualitative, in-depth interviews with key informants at the national, provincial, district, and ward levels.
For example, “Community engagement” is a standard under the Population Nutrition Management domain. For that standard, “we looked at the extent to which ministerial district offices use data for their planning and whether or not they involve communities,” said Mary Pat. “Maybe they just plan in a vacuum, and they don’t collect or receive any data [from local communities]. That would be level one. But if they consistently collect community data for planning and involve communities and local leaders, that’s level four.”
The PA tool has more than 100 questions that are asked of all ministries at different levels (national, provincial, district, and ward). Assessors answer those questions and determine the rubric score through interviews. “You can ‘slice and dice’ the results any way we want it,” said Mary Pat. “If we want to examine just the Ministry of Fisheries overall — from national to district level — we have a score. If we want to examine the Ministry of Health district levels only, we have a score. If we want to talk about the Ministry of Agriculture in one specific province, we have a score. We can get granular, and then we can roll up the data to get bigger pictures.”
Choosing the Right Assessors
In both 2020 and 2022, the SUN LE team conducted more than 500 interviews across all levels in 30 districts in Zambia. And as with most evaluations, hiring the right assessors was a key factor in the success of the SUN LE PAs. This step was particularly important for this type of project, in which assessors must do a lot of interpretation of discussions with respondents and evidence collected from those respondents. “All of [the assessors] had master’s degrees,” said Edna Berhane, SUN LE’s Quality Improvement Advisor. “That was one of the conditions. There are a lot of judgment calls that the assessors are expected to make.”
“The assessor determines the question’s rubric score based on the respondent’s answers and the evidence given. It’s not an interview in the sense that you just record what people are telling you,” said Mary Pat. “Because you’re interviewing [respondents] and making a judgement on their answers, the capacity or the skills of the assessors is the basis of the data collected.”
“Because of that, we felt that the caliber of the assessors should be much higher [than that of normal field workers],” said Edna. “So, they’re all senior people.” The team hired 33 assessors for the 2020 PA, most of whom returned for the 2022 PA.
Lessons Learned from the SUN LE Performance Audits
After the 2020 PA, the SUN LE team realized the assessors needed to insert more comments into the data collection tool while conducting the key informant interviews. “The tool is designed not to just provide a score, but to include comments justifying the score,” said Mary Pat. “The first time around, assessors didn’t add enough comments. And [comments are] really, really important when you have this kind of data collection effort where it’s not just capturing what people tell you, but you’re making a judgment…To look at their notes and say, ‘Oh, they said level three, but should it be a level two.’”
The Khulisa team is looking forward to making more use of the data collected from both the household surveys and PAs. “Next year, I want to take all four data sets – the baseline household survey, the midline household survey, and the two rounds of PAs – and I want to work the data to see if there’s a relationship,” said Mary Pat. “Are the provinces that have certain strengths in the PA achieving better results in their nutrition program, or in their household nutrition status? We can’t answer that question very well yet. We’d really like to get into the nitty gritty and figure out which aspects of the PA are most associated with improvements or deteriorations in nutritional status.”
Keep an eye on Khulisa’s social media channels for future posts about Khulisa’s SUN LE work.
