By Dr. Martha Muco and Jesse Webb
Khulisa is currently conducting an evaluation for the U.S. Department of State’s Bureau of Population, Refugees, and Migration (PRM), titled “Evaluating the Effectiveness of Programs for Protection of Refugee Youth in Urban Settings in Africa”. This case study involves evaluating PRM partners and programs that support one of the world’s most vulnerable populations: refugees between the ages of 15 and 24 living in some of Africa’s largest cities. The evaluation includes two case studies with data collection in three cities: Cairo, Egypt; Alexandria, Egypt; and Kampala, Uganda.
Collecting data from vulnerable populations often requires a keen sensitivity to their situation. People on the move or in remote areas, children, and persecuted minorities are often without access to basic technology, and dependent on the NGOs and international organizations that are supporting them. When collecting data from these groups, evaluators must cater to their circumstances to ensure data is accurate and the respondents’ information is protected and secure.
When proposing an appropriate data collection methodology for the PRM study, we thought carefully about what would work best for this hard-to-reach group of refugees. We wanted to design a mixed-method study, with a combination of qualitative and quantitative research. But we also knew most conventional quantitative data collection methods, such as online, telephone, or written surveys, would not be effective for interviewing refugee youth. The respondents’ personal identifying information must be secure, and they might not have access to technology. Tools would have to be written in several languages to accommodate refugees from northeast Africa. We also had to consider the privacy rules and protocols of the PRM partners supporting the respondents we needed to interview – specifically beneficiaries under 18, which are considered minors.
For all these reasons, we decided to use intercept interviews as the methodology for reaching refugee youth in Cairo, Alexandria, and Kampala. Intercept interviews (II) are in-person surveys conducted at the point of interaction with beneficiaries – in this case, interactions with the implementing partners involved in the evaluation. The instrument we used was structured with limited, closed-ended questions to adapt to the environment in which the data was collected. Most of these interviews are done standing outside, or at the premises of the activities and interventions implemented by PRM partners. The data was collected using handheld tablets, deployed with the latest technology, which worked with or without internet access.
Why We Chose the Intercept Interview Methodology
Although II is not a frequently used survey methodology in the evaluation world, we decided it was the proper choice in this case for several reasons:
· Respondents were intercepted and interviewed in-person, avoiding the need for smart phones or internet access.
· Our enumerators interviewed respondents without requesting any personal information except for their first name (optional) and age, which was required due to the specific age range of the evaluation.
· In most cases II allowed us to intercept beneficiaries randomly, avoiding possible biases in the process of recruiting respondents.
· Khulisa selected a skilled group of enumerators, many of whom were refugees themselves, who spoke the necessary languages and could verbally translate the 18-question survey from English.
· The enumerators’ handheld tablets allowed data to be transmitted in real time (whenever internet access was available).
The II methodology in this project complements several tips suggested by the World Bank in a July 2021 blog on collecting data from vulnerable populations:
· We hired local refugees, who formed part of the target of the evaluation, to administer the survey.
· We worked with enumerators to revise the language used in the tools, to ensure the enumerators could communicate clearly in local languages and with younger respondents.
· We sensitized enumerators to the mental health needs of the survey participants and the do-not-harm principles of evaluation.
The II methodology can also pose a few challenges. In-person surveys are labor-intensive and sample sizes tend to be relatively small. We also had to do a lot of outreach in advance: recruiting and training enumerators on specific protocols and how to work with minors; and educating PRM partners about the II methodology.
Despite these challenges, our study achieved a significant sample size – 320 interviews in a short period of time (two weeks in each country) – and we faced minimal challenges during data collection. We’re now in the data cleaning and data analysis and triangulation phase of the study, and we know II was the best, most practical, and most adaptive option to collect quantitative data for this particular evaluation.