End of Programme Evaluation (EPE) for the Tuberculosis in the Mining Sector in Southern Africa (TIMS) Programme

Objective

The Wits Health Consortium (WHC) commissioned Khulisa to conduct an independent end-of-program evaluation (EPE) for its Global Fund-financed “Tuberculosis in the Mining Sector in Southern Africa (TIMS)” regional program. TIMS aimed to reduce the TB burden in the mining sector in 10 southern African countries.

The EPE’s overall purpose was to: (i) assess the TIMS grant’s achievements and effectiveness, as well as the adequacy of the program strategies toward reaching expected targets and results; (ii) identify lessons; and (iii) make recommendations to inform the next phase of the program.

The evaluation’s eight objectives were to:

1. Assess TIMS progress from January 2016 to January 2018 against key outputs/milestones in the grant;
2. Assess the effectiveness of strategies and intervention implementation and provide recommendations to further improve the project;
3. Assess the strengths and weaknesses of partnerships with implementing countries, especially key ministries, mine worker associations, development partners, country coordinating mechanisms (CCMs), etc.;
4. Assess program efficiency focusing on financial, programmatic, planning, implementation, and monitoring and reporting systems and processes;
5. Assess the strengths and weaknesses of the management of the primary recipient vis-à-vis sub-recipients;
6. Assess the strengths and weaknesses of oversight by the regional coordinating mechanism (RCM) to ensure the success of the regional program.
7. Identify the opportunities and challenges that might affect the scale and sustainability of the program.
8. Identify lessons learned and make recommendations to inform the next phase of the program.

Approach:

This evaluation demonstrates Khulisa’s capability to execute a rigorous evaluation of a complex, regional project covering multiple countries and stakeholders (i.e., implementers, governments, civil society, local authorities, and beneficiaries) at national, provincial, district, sub-district, facility, and community levels.

To answer the eight evaluation objectives, the Khulisa team used a non-experimental design that excluded the use of a comparison group, but which allowed for measurement of project trends and achievements against pre-defined project targets and objectives. The EPE’s main data collection methods were:

1. Desk review and mining of programme data: Khulisa reviewed relevant grant documents and data to understand the evolution of priority areas and the development and continuum of program interventions and activities. The team also extracted program data from the WHC-TIMS Project Management Office (PMO) database and reports from January 2016 through December 2017, with a focus on the grant performance indicator data.
2. On-site key Informant Interviews (KIIs): Through field visits in all 10 countries, our team conducted interviews with key informants (individuals and groups) to obtain perceptions and feedback around the design, implementation, and results of the TIMS program and its core interventions.
3. On-site indicator validation: Khulisa conducted a mini data-quality assessment to validate data for the indicator, “Number of Key Populations (KPs) screened for TB”, reported by sub-recipients to WHC, and by WHC to the Global Fund.

Impact:

The evaluation’s conclusions, lessons learned, and recommendations were developed to inform phase II changes in the TIMS Grant. The findings and the recommendations highlighted the need for improvements in the following areas:

Principal Recipient Management and Coordination

• Clarified roles and responsibilities between the principal recipient, RCM, CCMs, and the Global Fund is crucial for maximizing TIMS and ensuring effective management of resources.
• Clear and systematic processes for setting agendas, conducting oversight, and disseminating key findings and next steps to all RCM members for improved coordination, enhanced transparency, strengthened accountability, efficient resource allocation, timely decision-making, and enhanced knowledge sharing.
• An online platform accessible to all sub-recipients and partners for enhanced internal communication and collaboration, improved planning and coordination, streamlined document management, real-time updates and notifications, increased accountability and transparency, and knowledge sharing and learning.

Internal Processes

• Sub-recipient selection: Adapt processes to evaluate changes since contract-signing, review financial documentation for sub-recipient selection, ensure proper procedures for activity selection by TIMS consortium members, and document any approved deviations from rules or regulations by relevant bodies.
• Financial efficiency: Formally define and measure the financial efficiency of each TIMS activity.
• Risk management: Ensure that the risk register is regularly/adequately reviewed and revised to identify all risks to the program at all times.
• Sub-recipient management: Implement timely funds disbursement, user-friendly financial reporting, efficient M&E and meeting structure, streamlined in-country activity management, internal communication strategies, midterm evaluation, and supply chain coordination.

Technical and Programmatic

• Health services management and coordination, including better training, more promotion of silicosis testing, improved coordination with laboratories, and improved collaboration with district health teams in each country.
• Engaging more female miners and local organizations focused on gender.
• Clarifying and unifying the regional TB screening tool to improve results of TB screening and active case management.