In a recent article in Evaluation Matters, published by the Independent Development Evaluation (IDEV) at the African Development Bank, Khulisa unpacks evaluation failures from four cases studies in Africa – Southern Africa, West Africa, East Africa and the Sahel.
The article, published in a second volume on Made in Africa evaluation approaches, draws lessons from evaluations that went wrong in these regions. Here is a summary:
- An evaluability assessment is essential prior to commissioning any evaluation. It gives an early realisation of the challenges, specifically data gaps. It also identifies when the program has not progressed enough to be evaluated.
- For multi-stakeholder evaluations, especially when African national governments are involved, it’s necessary to set up a joint baseline steering committee to anticipate and discuss any technical, administrative or political issues that might surface at all levels, and then build the foundation of the next steps during research.
- Social sector evaluations in Africa require far deeper inquiry than just what happened during project implementation, focussing above all on why and how it happened applying mixed methods approach or a combined quantitative and qualitative design.
- Asymmetry of information and knowledge undermines the likelihood of an evaluation’s success and usefulness. One major lesson is that in Africa, the usual evaluation planning is not enough to detect the real issues and lessons. One needs to know the context beforehand and include it in the evaluation design.
- To be successful, evaluations in Africa ought to include a preliminary field phase for preparation purposes.
- Evaluators should not rely on desk work or completion reports from executing agencies to plan their evaluation exercise. They should go to the field and understand the social dynamics and seasonal effects before deciding how to conduct evaluations.
- A last enabling factor of evaluation failure the inability to value indigenous knowledge, and this is more relevant when it comes to typical Made in Africa evaluations. Many evaluations in the Global South miss the assessment of changes in local dynamics and power relations. The undocumented nature of these dynamics and relations makes it difficult for evaluators who have not been exposed before to the context to grasp the true nature of the changes that occurred.