A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program

Community health workers (CHWs) are increasingly used to Background: increase access to primary healthcare, and considered to be a key health worker cadre to achieve the UNAIDS 90-90-90 target. Despite the recent policy interest in effectively designing, implementing, and evaluating new CHW programs, there is limited evidence on how long-standing CHW programs are performing. Using the CHW Performance Logic model as an evaluation framework, this study aims to assess the performance of Swaziland’s long-standing national CHW program, called the rural health motivator (RHM) program. This study was carried out in the Manzini and Lubombo Methods: regions of Swaziland. We conducted a survey of 2,000 households selected through two-stage cluster random sampling and a survey among a stratified simple random sample of 306 RHMs. Additionally, semi-structured qualitative interviews were conducted with 25 RHMs. While RHMs are instructed Results: to visit every household assigned to them at least once a month, only 15.7% (95% CI: 11.4 – 20.4%) of RHMs self-reported to be meeting this target. Less than half (46.3%; 95% CI: 43.4 – 49.6%) of household survey respondents, who reported to have ever been visited by a RHM, rated their overall satisfaction with RHM services as eight or more points on a 10-point scale (ranging from “very dissatisfied” to “very satisfied”). A theme arising from the qualitative interviews was that community members only rarely seek care from RHMs, with care-seeking tending to be constrained to emergency situations. Conclusions: The RHM program does not meet some of its key performance objectives. Two opportunities to improve RHM performance identified by the evaluation were increasing RHM's stipend and improving the supply of equipment and material resources needed by RHMs to carry out their tasks.


Introduction
Many low-and middle-income countries, particularly in sub-Saharan Africa, face a severe shortage of skilled healthcare workers 1 . Community health workers (CHWs) are increasingly being used to address this shortage of more extensively trained health workers in order to increase access to primary healthcare services 2, 3 . While there has been a recent policy interest in designing, implementing, and evaluating new CHW programs 4-8 , many large CHW programs that have existed for decades have not yet been rigorously evaluated. One such program is Swaziland's national CHW program, known as the rural health motivator (RHM) program. Existing since 1976, the RHM program currently employs over 5,000 RHMs and aims to cover every household in the nation with basic primary healthcare and health information 9 .
HIV is causing the highest burden of any disease in Swaziland 10 , and is a major challenge to the country's health system. UNAIDS and the World Health Organization recently set a new goal for ending the HIV epidemic: the 90-90-90 target 11 . Under this target, countries aim to ensure that, by 2020, 90% of people living with HIV know their HIV status, 90% of all people whose HIV infection has been diagnosed receive sustained antiretroviral therapy (ART), and 90% of all those receiving ART are virally suppressed. Expanded utilization of CHWs is considered essential to achieving this goal 12 , particularly through offering community-based HIV testing and shifting certain components of long-term ART care from healthcare facilities to the community, for example through ART home delivery [13][14][15][16] . Yet, while RHMs are providing many HIV-relevant services, including the provision of condoms, information on HIV, and following up with pre-ART and ART patients who have missed an HIV care appointment 17 , HIV treatment and care in Swaziland is still largely facility-based. Successful shifting of further HIV testing, treatment and care tasks from healthcare facilities to RHMs would likely require the RHM program to perform reliably and at a high level. Using the CHW Performance Logic Model as an evaluation framework 18 , this study therefore aims to (i) assess the performance of the RHM program, and (ii) identify ways in which program performance can be improved.

Study setting
This study was conducted in the Lubombo and Manzini regions, which are two of Swaziland's four administrative regions. Shiselweni and Lubombo are the most rural and poorest regions in Swaziland, while Manzini and Hhohho are comparatively more urban and wealthy 19,20 . In the latest census from 2007, 206,400 people lived in Lubombo and 313,900 in Manzini, jointly accounting for 52% of Swaziland's total population 19 . According to Swaziland's last HIV incidence and measurement survey 21, 22 , conducted in 2010 and 2011, adult HIV prevalence was 32.4% in Lubombo and 33.6% in Manzini region. The corresponding national estimate was 32.1%.

Community Health Worker programs in Swaziland
A number of CHW programs are currently active in Swaziland. At the time of the study, all CHW programs other than the RHM program had a cadre of less than 50 CHWs. While this study also collected data on three non-RHM CHW programs (the HIV expert  client program, the Mothers2Mothers mentors, and a community  outreach team for HIV-testing and voluntary male medical circumcision), this manuscript focuses on the RHM program given its size, and thus importance to Swaziland's health system.

The RHM program
Established in 1976, the RHM program employed 5,214 RHMs in 2015. As per their official job responsibilities, RHMs are assigned the following activities during their household visits: 1) referring ill household members to a healthcare facility; 2) providing health information on a variety of health topics; 3) providing condoms; 4) encouraging household members to take up preventive healthcare services and antenatal care; 5) follow up with those community members who have missed an HIV care appointment at the healthcare facility; 6) attending medical emergencies (e.g., emergency deliveries); 7) assisting with growth monitoring programs of children under five years of age; 8) dietary counseling; and 9) promoting adult literacy 17 . RHMs are instructed to visit 25 households assigned to them at least once a month.

Quantitative methodology
Quantitative data were collected through a population-based household survey and a questionnaire for RHMs (Supplementary File 1 and Supplementary File 2). The household survey employed two-stage stratified cluster random sampling. In the first stage, we selected a random sample of 50 enumeration areas (EAs) in each Lubombo and Manzini. In each region, 37 of the enumeration areas were classified as rural by the Swaziland Statistics Office, and 13 as urban. In each EA, we selected 20 households through systematic random sampling. Data collectors administered a questionnaire in SiSwati to each household member aged 11 years or older who was present at the time of the household visit and who provided written consent to participate in the survey. Due to feasibility constraints, the data collection team did not revisit households if no household members were present at the time of the visit.
The RHM questionnaire was administered in SiSwati to all RHMs working in the EAs that were selected for the household survey. Since the EAs selected for the household survey were only a relatively small subsample of all EAs in the Lubombo and Manzini region of Swaziland, 306 (12.0%) out of a total of 2,543 RHMs in these two regions were interviewed. The RHM questionnaire was administered at the RHM's household by the same cadre of data collectors, which conducted the household survey.
Both the household and RHM survey were conducted between June 2015 and September 2015. Quantitative analyses consisted of descriptive statistics (means and proportions) and were conducted in Stata version 13.0 (College Station, TX, USA).

Qualitative methodology
Qualitative data were collected through semi-structured interviews with 25 RHMs (Supplementary File 3). These RHMs comprised a criterion-based stratified purposive sample. Strata used were region (13 RHMs from Manzini and 12 from Lubombo region) and urban versus rural (13 from rural areas and 12 from urban areas in each region). Additional sampling criteria were age and sex of RHMs, attempting to yield a sample that is similar to the age and sex distribution of the RHM cadre in general. In addition, we conducted semi-structured qualitative interviews with the chief RHM program manager in the program office in Mbabane, Swaziland, and five RHM trainers in the regional offices of the RHM program.
Five recent graduates of the University of Swaziland Social Science Program who were fluent in SiSwati and English conducted the interviews. The data collectors were Swazi and aged between 20 and 35 years. The interviews lasted between 30 and 45 minutes and were conducted in SiSwati. The interviewers taped the interviews, and transcribed them verbatim in SiSwati. The transcripts were then translated into English by the local study coordinator, who is also an author of this paper (MM). He also conducted a quality check of each transcript. Two authors (MV and PG) conducted content analysis using an inductive approach to coding 23 . We identified broad themes after an initial review of the data, and then conducted iterative reviews to further refine themes and their relationships to each other. All coding was done using NVivo 11 (QSR International, Melbourne, Australia).

Evaluation framework
The evaluation framework that was used for this performance evaluation is the CHW Performance Logic Model (Figure 1), which has been described in detail elsewhere 18 . The model was used to inform the design of the questionnaires and interview guides. More specifically, the data collection tools contained questions on the dimensions (white rectangles in Figure 1), which in turn were grouped under sections corresponding to the dimensions of the model (results, activities, and inputs). Questions in the household survey questionnaire focused on CHW program outcomes by asking about the household members' experiences with the RMH program and the degree to which they sought care from RHMs. Meanwhile the RHM questionnaire focused on CHW program outputs (e.g., self-reported performance, and job satisfaction and motivation), and support provided to RHMs by the community and health system (and actors within these systems). Data on inputs was obtained from program reports and personal meetings with the RHM program management. We have structured the results section according to the logic model dimensions, moving from the inside (CHW performance outcomes) to the outside (inputs) of the model depicted in Figure 1.

Sample characteristics
The RHM questionnaire was administered to a total of 306 RHMs, 96.1% of whom were female (Table 1). On average, RHMs were 52.9 years old (SD: 11.6 years) with 16 RHMs (5.2%) older than 70 years. RHMs had lived in their communities for an average of 34.6 years (SD: 16.5 years) and had worked in the RHM program for 15.5 years (SD: 12.9 years). 30.5% of RHMs reported to have done work other than for the RHM program during the previous 12 months. The characteristics of the 25 RHMs with whom we conducted semi-structured qualitative interviews were similar to those of the sample of RHMs who were included in the RHM survey. The population-based household survey was administered to 2,342 household members across 2,000 households. 97.7% of household survey respondents had lived in the surveyed community for more than one year.

CHW performance
As described in the methods, we assessed performance of the RHM program on the output and outcome level of the CHW Performance Logic Model 18 . Table 2 summarizes our quantitative findings.   Figure 2). 20.4% of respondents rated their satisfaction as less than five on this scale. Nonetheless, the vast majority (96.1%) of respondents would recommend the RHM program to other communities.
Outcomes: Care-seeking from RHMs. 76.7% of RHMs indicated that households had approached them for help or advice. However, in the qualitative interviews, a topic that emerged is that although households did approach RHMs, it was either rare or infrequent. In cases where RHMs were approached, it was usually for acute emergency care: Interviewer: "How often are you contacted for help or advice?" RHM: "It is rare … sometimes when someone is in labor then they call me for help" (Manzini) In the less common scenario where RHMs indicated that they were contacted frequently, it tended to be for material assistance such as medication, diapers, or gloves:   1 This question was only asked to household survey respondents who reported that their household had ever been visited by a RHM (n=1,151). 2 Satisfaction was measured on a scale ranging from 1 ("very dissatisfied") to 10 ("very satisfied").
a result of them being part of the RHM program. 74.3% indicated that their standing had increased, while only 16.5% stated that their standing had decreased, with the remainder answering that their standing had remained unchanged. In the qualitative interviews, when asked about the effect of their work as a RHM on their community standing, RHMs who indicated an increase in community standing suggested that RHMs' responsibilities mean that community members respect them more. In cases where RHMs indicated that their community standing remained unchanged or had decreased, these were accompanied by the perception that they did not meet the expectations of community members:  Technical support. The initial training for new RHMs lasts 12 weeks full-time. In addition, the program runs in-service trainings, which re-emphasize certain topics taught during the initial training and usually also cover some new material. These refresher trainings last for two to five days and are conducted once a year for each RHM. Only 10.5% of RHMs surveyed reported to never have attended an in-service training. Most RHMs (94.7%) either agreed (48.3%) or strongly agreed (46.4%) that the training provided by the program is sufficient to competently perform their work as a RHM. 81.9% rated the quality of their in-service training as being high.

Program-level activities
Incentives. The majority of RHMs expressed dissatisfaction with the compensation offered. 57.8% either disagreed (38.0%) or strongly disagreed (19.8%) with the statement that "Given the amount of work I do as a rural health motivator, I am being paid a fair amount". This is also reflected in the qualitative data, in which RHMs frequently mentioned that they do not feel that they are sufficiently compensated. A typical opinion expressed in this regard is:

"I do not feel I am being paid a fair amount because there is a lot of work that we do. Sometimes the families desert the ill patients and leave them in their own dirt until the day a RHM comes along and bathes the patient, feeds them....so the work is quite a lot" (RHM, Lubombo)
Very few RHMs reported to have received non-monetary compensation from the RHM program.

Family
RHMs who expressed that their families were somewhat supportive or very supportive of their work 95.7 (92.7 -97.7)

Technical Support
RHMs who indicated that their job responsibilities were either well explained or very well explained to them 99.0 (97.1 -99.8) RHMs who either agreed or strongly agreed that they received all the training necessary for them to perform their jobs 94.7 (91.5 -96.9) RHMs who rated the quality of their in-service training as being high 2 81.9 (76.8 -86.2)

Monetary
RHMs reporting to not being paid for their work 11.4 (8.1 -15.5) RHMs who disagree or strongly disagree with the statement "Given the amount of work I do as a RHM, I am being paid a fair amount" 57.8 (50.0 -63.4)

Non-monetary
RHMs who received any non-monetary payments from the program 11.5 (8.  1 This question was only asked if the RHM reported to have regularly interacted with facility-based healthcare workers (93.0%). 2 This was defined as reporting ≥8 on a 10-point scale from "very bad quality" to "very high quality". 3 The denominator for these percentages is the number of RHMs who reported having received nonmonetary payments from the program. satisfied (17.4%) with the level of supervision that they receive, 65.3% indicated that they would like to receive more supervision. Qualitatively, in cases where RHMs expressed interest in additional supervision, the reason tended to be that they felt additional feedback would help motivate them further and support continued learning, as illustrated by the following quote: "I would like more supervision because it would help me learn and grow my skills as a RHM. Additionally, it helps to keep me motivated and to put in more effort in my work" (RHM, Manzini) Provision of material resources. 60.6% of RHMs either disagreed (40.1%) or strongly disagreed (20.5%) that the program provides all the equipment, supplies, and material resources necessary to perform their duties.

Inputs
Human resources. The RHM program had 5,214 RHMs in 2015, of which roughly half (2,803) lived and worked in the Lubombo or Manzini region. In addition, the program had one program manager, one program officer, one administrative assistant, 18 RHM trainers (who are trained nurses), and two drivers.

Capital resources
The RHM program occupies four offices in the country, one in each of Swaziland's four regions. The program also owns two cars. Table 5 shows the running costs of the RHM program for 2011 using data from the Kingdom of Swaziland Budget versus Expenditure Report 2012 24 , which was the latest data available to us. We present these costs in terms of purchasing power parity dollars (PPP$). One PPP$ is calculated such that it had the same purchasing power in Swaziland in 2011 as one US dollar had in the United States in that year. Roughly two thirds of the program

Discussion
This evaluation identified a number of weaknesses in the RHM program's performance. First, despite being in close geographic proximity to their clients, the Swazi population appears to prefer seeking care from other healthcare workers than the RHM cadre. As found in particular through our qualitative interviews, community members rarely seek care from RHMs, and if they do, this tends to be for emergency care when care from other health care providers is unavailable. Second, client satisfaction with the RHM program appears to be comparatively low. The survey data on client satisfaction is likely to suffer from some degree of courtesy or social desirability bias whereby community members give a more favorable assessment of the RHMs' care to abide by a perceived social norm of showing satisfaction and gratitude rather than criticism 25 . Despite the possibility of this bias, a comparatively low proportion (46.3%) of community members rated their overall satisfaction with RHM services as eight or more points on a 10-point scale ranging from very dissatisfied to very satisfied. Third, RHMs do not appear to provide the quantity of care that the program aims to provide. Data on the number of households visited by RHMs are selfreported and may, thus, also suffer from an upward bias as RHMs

Resource Mobilization
RHMs who disagreed or strongly disagreed that the RHM program provides all the equipment, supplies, and material resources necessary for them to perform their duties 60.6 (54.8 -66.1) Abbreviations: RHM=rural health motivator are likely to want to appear as fulfilling their duties. Despite this likely bias, only 15.7% of RHMs reported achieving the program target of visiting all assigned households at least once a month. Overall, improving the performance of the RHM cadre may be necessary to successfully shift HIV care tasks from facility-based to RHM-led care.
Our assessment of the RHM program on the program-and systemlevel dimensions of the CHW Performance Logic Model provides some insight into factors that might be lowering RHM performance. In general, RHMs report that they are satisfied with the quantity and quality of training and supervision provided to them. However, RHMs are dissatisfied with the level of monetary compensation, with 57.8% of RHMs indicating that the level of their pay is unfair given the amount of work they do. In 2015, RHMs earned 350 Swazi Lilangeni (approximately US$ 22.50) per month. Additionally, in the qualitative interviews, RHMs reported that they face transport costs and bank fees to collect and cash their paycheck. Swaziland's national poverty line lies at US$ 3.10 per day 26 . Ignoring costs to collect and cash their paycheck, RHMs earn approximately US$ 0.74 per day, which is only 23.9% of the daily income needed to be earning at the national poverty line. Expectations of RHM performance need to be examined in light of this comparatively low level of pay. The low pay is likely also an obstacle for shifting HIV care tasks to RHMs, as many of these tasks, such as ART home-delivery, require reliable and constant care. A theme arising from our qualitative interviews, however, was that RHMs view themselves as volunteers rather than employees given their low level of pay. It would thus seem likely that other income-generating activities take priority over RHM work, which in turn may lead to prolonged gaps in RHM care delivery.
Apart from monetary compensation, RHMs were also dissatisfied with the material resources provided to them by the RHM program for performing their duties. In the qualitative interviews, RHMs frequently mentioned that community members expect them to provide certain material resources, such as diapers, medications (particularly paracetamol), bandages, and disposable gloves. RHMs felt that not being able to meet this expectation was an important barrier in maintaining a good relationship with the community, and to cover the households that they were assigned. Thus, providing the expected material resources to RHMs and/or altering the expectations of community members to receive such resources from RHMs may increase RHM performance. Improving the RHM-client relationship is of particular importance if RHMs are to provide more HIV care given the continued high HIV-related stigma in Swaziland 27 .
We used the CHW Performance Logic Model to guide this performance evaluation. While the logic model aims to be a useful tool for planning, consensus-building, implementation, and evaluation of CHW programs 18 , we can only comment on our experience with the model's usefulness for CHW program evaluations. A key characteristic of the model is that it tries to comprehensively include all factors that may influence CHW performance. As such, the logic model differs strongly from the more simplistic framework of inputs -processes -outputs that we have previously used for a performance evaluation of a CHW program in Dar es Salaam, Tanzania 28 . In our view, the comprehensive nature of the logic model is its key strength. Given the sheer number of possible factors that may plausibly influence CHW program performance, most evaluators will have to make a decision regarding the scope of their evaluation. The CHW Performance Logic Model could help evaluators clearly define the evaluation's scope, and be more explicit about their choice of which factors and domains they include in the evaluation. Nonetheless, the model's comprehensive nature could be a disadvantage if evaluators find the number of possible factors to evaluate overwhelming. In our view, the main disadvantage of the model is that it does not provide any guidance to evaluators on which factors are the most important determinants of CHW performance. As such, a prioritization of the categories and factors in the model based on relevant theory and evidence, rather than an un-weighted list of all factors that plausibly influence CHW performance, would substantially improve the utility of the model. Another limitation of the model is that many of the performance measures and factors assessed under the model's dimensions lack established measures and scales. In addition, there are doubts as to whether a dimension is measured appropriately, which also results in some degree of subjectivity in interpreting what level of CHW program performance the observed achievement on a measure represents.
Other limitations of this study include that the data from the RHM questionnaire are likely to suffer from a degree of self-reporting bias whereby RHMs may, for example, over-report aspects of their work that they perceive as desirable (e.g., the number of households visited). Similarly, household survey respondents may have been hesitant to express criticism of RHMs because they wanted to maintain a good relationship with the RHMs (who are fellow community members chosen by the community and the village chiefs), or simply due to an intrinsic tendency to be courteous. Lastly, while the RHM program is a national program, this assessment has focused on only two of four regions in Swaziland due to feasibility constraints. However, these two regions constitute more than half (52%) of Swaziland's population, and the program structures for management and implementation of the RHM cadre do not differ between regions. We, therefore, feel confident that the findings of this study apply to the RHM program as a whole.

Conclusions
This evaluation found that the RHM program does not meet some of its performance targets. For instance, RHMs are currently not an important point of first call for seeking care for an illness, and the RHMs do not appear to achieve their household coverage target. If the RHM program is to adopt specific HIV-related tasks, then Swaziland's HIV response would likely benefit from policy and management changes aimed at improving RHM performance. While it is beyond the purview of this study to provide an exhaustive list of suitable reforms, two simple changes identified by this evaluation that may lead to an improvement in RHM performance are i) an increase in monetary compensation, and ii) the provision of material resources to RHMs (e.g., paracetamol, diapers, and bandages) to enable RHMs to meet their community's expectations.

Data availability
Please note that some items have been removed/edited due to potentially identifiable information. The datasets contain both CSV and .dta files. Qualitative interview manuscripts are not shared publicly because they cannot be effectively de-identified given the relatively small number of staff involved in the studied community health worker programs. Individuals interested in accessing the transcripts should contact the corresponding author.
Author contributions PG and MV analyzed the data and wrote the first draft of the manuscript. The authors (PG, MV, JWD, TB, TJB) jointly designed the study and data collection tools. All authors (PG, MV, JWD, TB, TJB) provided important edits to the manuscript and approved the final version.