Validity is an ongoing process over time, and the deeper and more rigorous the analysis and greater the range of samples, the stronger the case for validity. Four lessons emerged from Khulisa’s experience developing, refining and starting the process of validating a school functionality evaluation tool.
The lessons below, written by Khulisa staff Margaret Roper, Leticia Taimo, Jennifer Bisgard, and Katherine Tjasink was published in the full peer-reviewed African Evaluation Journal article, provide guidance to evaluators embarking on a tool validation process:
1. Ensure sufficient time to develop, test and refine the tool
Assuming it is not possible to build the full process into one evaluation, because of cost and time limitations, build a tool over time and over a range of existing and relevant projects (where possible) can provide useful insights.
2. Build in time & resources for a validation process from the beginning
Looking retrospectively at our process, establishing content validity required several rounds of reviews and pre-tests, and iterative tool refinement, to come to a point where the tool encompassed the correct constructs in line with the literature, government norms and standards, and with contextually appropriate scales of measurement and set criteria for standards. When designing or adapting a tool to context, it is advisable to plan for reliability and validation from the start. The process requires many team members with different skill-sets and technical specialists (in our case, tool development specialists, statisticians and education specialists) to assist in the process.
3. Make provision for the rigorous training of researchers
The tool relies mostly on observations, which are biased to the observer’s frame of reference. Thus, it is important to establish that different researchers are collecting data in a consistent way. This involves rigorously training for the researchers and checking for inter-rater reliability. It typically involves a 3-day training process, where the first day consists of familiarisation with the tool and training on the ethics of collecting data in schools (including photographing children), the second day involves experiential learning where researchers collect real data on site and the third day includes feedback to researchers and revisions (if required) to the tool.
4. Use photographs as evidence to triangulate researchers’ ratings
We learned that including sources of verification, in our case the option for photographs to be taken, was a key element to ensure consistency of scoring. For example, by examining photographs taken of the toilets, one can determine whether researchers are rating these in the same way. If this is not the case, there is a need to explore why not (e.g. do the researchers require more training? or, is the question or measurement criterion not clear?). Given the sensitivity of this type of data source, it is important that such data are adequately protected in line with the relevant laws and legislation.
Download Khulisa’s full article from the African Evaluation Journal here for free. https://aejonline.org/index.php/aej/article/view/423/738