By Mary Pat Selvaggio
Khulisa is currently carrying out a large-scale Performance Audit of a multi-sectoral nutrition programme being implemented at provincial, district, and sub-district level. Our approach provides common measures of performance across seven sectors, across national and sub-national levels, and across numerous implementation partners. We defined 31 standards across eight domains. This blog captures our evolving expertise.
Most performance audit (PA) definitions[1] emerge from financial auditing with a focus on determining the efficiency, economy, and effectiveness of the operations of organisations, programmes, and services.
However, when conducting PAs for public sector social programmes or services, there is a trend to rather focus the audit around adherence or compliance to pre-defined standards or benchmarks (local or international) in the programme’s or service’s management, systems, processes, or procedures.
Indeed, Kinnear[2] notes a simple definition of PA as an independent examination of the “compliance of programs, activities, and functions with predetermined standards”.
Performance Audit vs Program Evaluation
Performance Audits are designed to provide assurance that programme/service quality and delivery meets expected standards. But PAs are not designed to answer ‘how and why’. This is addressed by program evaluations.
So the choice between a ‘performance audit’ and a ‘program evaluation’ depends on what the commissioner wants to know about the program.
If it is to measure compliance, the choice will be an audit. If it is to know how a program has performed and why, the choice will be a program evaluation
S. Kinnear
Khulisa’s experience with Performance Audits
For more than 15 years, Khulisa has conducted PAs in various manifestations of public sector programmes (health and education).
In all our PAs, we first define expected standards for the programme’s structures, functions, and responsibilities, as well as processes, procedures, and systems. These standards are most often based on benchmarks established by international agencies (i.e. WHO, UNICEF, Global Fund, etc.) or national bodies (i.e. Ministries of Health or Ministries of Education).
Khulisa’s first PA experience was in Data Quality Auditing, a type of PA that focuses on the adequacy of data management systems and processes to produce good quality data. Our Data Quality Audits verified the adherence of the service, programme, or organisation to data management standards around:
| DQA standards – areas of inquiry |
| Organisational structure, definition of functions and attribution of responsibilities |
| Capacity and availability of human resources |
| Adequacy of tools used for collecting and reporting data (whether electronic or paper) |
| Procedures around data collection, aggregation, manipulation and reporting. Linkage between existing systems to avoid duplications of efforts |
| Accuracy of data reported |
| Extent of data use in decision making |
Khulisa has also conducted PAs of HIV and TB programmes implemented by PEPFAR partners. These PAs focused on verifying the programmes’ adherence to expected standards in 6 functional areas:
| Standards Assessed | Standards Assessed |
| A. Policies, Plans, Procedures, and Regulations | – relevant legislation, policies, guidelines – relevant donor guidance – relevant other guidance (SOPs, frameworks, etc) |
| B. Human Capacity Development /Systems Strengthening (HCD/SS) | – institutional strategic SS/HCD model – planning/management for SS/HCD – SS/HCD programme implementation SS/HCD effects, outcomes, and services satisfaction – relationship with, and alignment to national guidance |
| C. Organisational Administration and Fiscal Management (consists of consists of core questions from Module 10) | – internal controls, expenditure and procurement – contract management and sub-recipient management – financial management and reporting – asset management – risk management – organisational and programme management structures – relevant documents for human resources management – hr planning, policies and strategies – payroll/timekeeping / staff contracts |
| D. Programme Planning and Implementation | – programme planning (workplans, etc) – programme implementation (M&E, devising targets and strategies) – information systems: record keeping and information management systems (including forms and registers) – human resources management |
| E. Service Delivery | – clinic/patient/beneficiary care – clinical support services (lab, pharmacy, legal/protective services, ancillary programme areas for community-based programmes) – public health (health promotion, IEC materials, population-based planning for services delivery) – supply chain management (stock management for condoms, pharmaceuticals, home care kits, testing kits, etc.) – facility and infrastructure (waiting areas, safety, and security, hygiene and cleanliness, waste management) |
| F. Referrals, Linkages, Integration | – referral mechanisms for continuity of care – service integration and linkages |
Multi-sectoral nutrition programme PA
Currently, Khulisa is carrying out a large-scale PA of a multi-sectoral nutrition programme being implemented at provincial, district, and sub-district level. Our approach, adapted from the Primary Health Care Progression tool[3], aims to provide common measures of performance across 7 sectors, across national and sub-national levels, and across numerous implementation partners. To achieve this, the PA was designed so that findings can be easily “sliced and diced” across sectors (health, water, agriculture, etc.), implementing partners, and levels. In these PAs, we defined 31 standards across 8 domains:
| PA Domain | Areas examined |
| 1. Governance and Leadership | – Strategic policy frameworks, plans, guidelines – effective oversight – coalition-building – regulation – attention to system-design – accountability |
| 2. Adjustment to Population Needs | – Routine information about nutrition status/needs – use of information to set/implement priorities – continual learning/adaptation based on emerging evidence/data |
| 3. Commodities, supplies and Service Infrastructure | – Performance of supply chains – availability of essential consumables, products, and technologies at health facilities / community level |
| 4. Information Systems | – Availability, coordination, and interoperability of civil registration/vital statistics – routine MIS – beneficiary records systems – information infrastructure |
| 5. Workforce | – Sufficiency and distribution of workers to meet population nutrition needs and access to quality services – quality of workforce education and practice. |
| 6. Funding | Management of funds to address recurrent and fixed programme costs incurred at all levels, including payment of staff salaries. |
| 7. Population Nutrition Management | – Priority setting – community engagement – empanelment – population outreach |
| 8. Service Organization and Management | – Effective service operation organization – HR deployment in multidisciplinary teams – routine reporting against targets – QI initiatives – management oversight. |
Over the years, Khulisa’s PAs have evolved to meet the performance information needs of public sector policy makers and programme managers. But always, our approach has been to ensure reliable, objective, and independent information around how programme governance, planning, management, and implementation could be compared with, and improved to meet, international best practices or standards.
[1] https://www.investopedia.com/terms/p/performanceaudit.asp
https://www.nexia-sabt.co.za/thought-leadership/performance-auditing-an-introduction/
https://study.com/academy/lesson/performance-audit-definition-measurement.html
https://assets.kpmg/content/dam/kpmg/pdf/2016/05/Performance-Auditing.pdf
[2] https://www.linkedin.com/pulse/program-evaluation-vs-performance-audit-evalitica-software-/
[3] PHCPI Assessment Tool. https://improvingphc.org/sites/default/files/PHC-Progression%20Model%202019-04-04_FINAL.pdf