A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program

Pascal Geldsetzer; Maria Vaikath; Jan-Walter De Neve; Till Bärnighausen; Thomas J. Bossert

doi:10.12688/f1000research.11361.1

Home Browse A qualitative and quantitative performance evaluation of Swaziland’s...

ALL Metrics

-

Views

-

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program

[version 1; peer review: 2 approved with reservations]

Pascal Geldsetzer ¹^*, Maria Vaikath¹^*, Jan-Walter De Neve^1,2, Till Bärnighausen^1-3, Thomas J. Bossert¹

Pascal Geldsetzer ¹^*, Maria Vaikath¹^*, [...] Jan-Walter De Neve^1,2, Till Bärnighausen^1-3, Thomas J. Bossert¹

^* Equal contributors

PUBLISHED 02 May 2017

Author details Author details

¹ Department of Global Health and Population, Harvard T.H. Chan School of Public Health, Boston, MA, 02115, USA
² Institute of Public Health, Heidelberg University, Heidelberg, 69120, Germany
³ Africa Health Research Institute, University of KwaZulu-Natal, Mtubatuba, 3935, South Africa

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Background: Community health workers (CHWs) are increasingly used to increase access to primary healthcare, and considered to be a key health worker cadre to achieve the UNAIDS 90-90-90 target. Despite the recent policy interest in effectively designing, implementing, and evaluating new CHW programs, there is limited evidence on how long-standing CHW programs are performing. Using the CHW Performance Logic model as an evaluation framework, this study aims to assess the performance of Swaziland’s long-standing national CHW program, called the rural health motivator (RHM) program. Methods: This study was carried out in the Manzini and Lubombo regions of Swaziland. We conducted a survey of 2,000 households selected through two-stage cluster random sampling and a survey among a stratified simple random sample of 306 RHMs. Additionally, semi-structured qualitative interviews were conducted with 25 RHMs. Results: While RHMs are instructed to visit every household assigned to them at least once a month, only 15.7% (95% CI: 11.4 – 20.4%) of RHMs self-reported to be meeting this target. Less than half (46.3%; 95% CI: 43.4 – 49.6%) of household survey respondents, who reported to have ever been visited by a RHM, rated their overall satisfaction with RHM services as eight or more points on a 10-point scale (ranging from “very dissatisfied” to “very satisfied”). A theme arising from the qualitative interviews was that community members only rarely seek care from RHMs, with care-seeking tending to be constrained to emergency situations. Conclusions: The RHM program does not meet some of its key performance objectives. Two opportunities to improve RHM performance identified by the evaluation were increasing RHM's stipend and improving the supply of equipment and material resources needed by RHMs to carry out their tasks.

Keywords

Community health worker, performance evaluation, Swaziland, rural health motivator

Corresponding author: Pascal Geldsetzer

Competing interests: No competing interests were disclosed.

Grant information: Data used in this study were collected for other activities supported by the American people through the United States Agency for International Development (USAID) with funding from the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). The data were collected by the Harvard T.H. Chan School of Public Health through the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project. The USAID ASSIST Project is managed by University Research Co., LLC (URC) under the terms of Cooperative Agreement AID-OAA-A-12-00101. The authors’ views expressed in this paper do not necessarily reflect the views of USAID or the United States Government.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2017 Geldsetzer P et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

How to cite: Geldsetzer P, Vaikath M, De Neve JW et al. A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program [version 1; peer review: 2 approved with reservations]. F1000Research 2017, 6:607 (https://doi.org/10.12688/f1000research.11361.1) First published: 02 May 2017, 6:607 (https://doi.org/10.12688/f1000research.11361.1) Latest published: 02 May 2017, 6:607 (https://doi.org/10.12688/f1000research.11361.1)

Introduction

Many low-and middle-income countries, particularly in sub-Saharan Africa, face a severe shortage of skilled healthcare workers¹. Community health workers (CHWs) are increasingly being used to address this shortage of more extensively trained health workers in order to increase access to primary healthcare services^2,3. While there has been a recent policy interest in designing, implementing, and evaluating new CHW programs^4–8, many large CHW programs that have existed for decades have not yet been rigorously evaluated. One such program is Swaziland’s national CHW program, known as the rural health motivator (RHM) program. Existing since 1976, the RHM program currently employs over 5,000 RHMs and aims to cover every household in the nation with basic primary healthcare and health information⁹.

HIV is causing the highest burden of any disease in Swaziland¹⁰, and is a major challenge to the country’s health system. UNAIDS and the World Health Organization recently set a new goal for ending the HIV epidemic: the 90-90-90 target¹¹. Under this target, countries aim to ensure that, by 2020, 90% of people living with HIV know their HIV status, 90% of all people whose HIV infection has been diagnosed receive sustained antiretroviral therapy (ART), and 90% of all those receiving ART are virally suppressed. Expanded utilization of CHWs is considered essential to achieving this goal¹², particularly through offering community-based HIV testing and shifting certain components of long-term ART care from healthcare facilities to the community, for example through ART home delivery^13–16. Yet, while RHMs are providing many HIV-relevant services, including the provision of condoms, information on HIV, and following up with pre-ART and ART patients who have missed an HIV care appointment¹⁷, HIV treatment and care in Swaziland is still largely facility-based. Successful shifting of further HIV testing, treatment and care tasks from healthcare facilities to RHMs would likely require the RHM program to perform reliably and at a high level. Using the CHW Performance Logic Model as an evaluation framework¹⁸, this study therefore aims to (i) assess the performance of the RHM program, and (ii) identify ways in which program performance can be improved.

Methods

Study setting

This study was conducted in the Lubombo and Manzini regions, which are two of Swaziland’s four administrative regions. Shiselweni and Lubombo are the most rural and poorest regions in Swaziland, while Manzini and Hhohho are comparatively more urban and wealthy^19,20. In the latest census from 2007, 206,400 people lived in Lubombo and 313,900 in Manzini, jointly accounting for 52% of Swaziland’s total population¹⁹. According to Swaziland’s last HIV incidence and measurement survey^21,22, conducted in 2010 and 2011, adult HIV prevalence was 32.4% in Lubombo and 33.6% in Manzini region. The corresponding national estimate was 32.1%.

Community Health Worker programs in Swaziland

A number of CHW programs are currently active in Swaziland. At the time of the study, all CHW programs other than the RHM program had a cadre of less than 50 CHWs. While this study also collected data on three non-RHM CHW programs (the HIV expert client program, the Mothers2Mothers mentors, and a community outreach team for HIV-testing and voluntary male medical circumcision), this manuscript focuses on the RHM program given its size, and thus importance to Swaziland’s health system.

The RHM program

Established in 1976, the RHM program employed 5,214 RHMs in 2015. As per their official job responsibilities, RHMs are assigned the following activities during their household visits: 1) referring ill household members to a healthcare facility; 2) providing health information on a variety of health topics; 3) providing condoms; 4) encouraging household members to take up preventive healthcare services and antenatal care; 5) follow up with those community members who have missed an HIV care appointment at the healthcare facility; 6) attending medical emergencies (e.g., emergency deliveries); 7) assisting with growth monitoring programs of children under five years of age; 8) dietary counseling; and 9) promoting adult literacy¹⁷. RHMs are instructed to visit 25 households assigned to them at least once a month.

Quantitative methodology

Quantitative data were collected through a population-based household survey and a questionnaire for RHMs (Supplementary File 1 and Supplementary File 2). The household survey employed two-stage stratified cluster random sampling. In the first stage, we selected a random sample of 50 enumeration areas (EAs) in each Lubombo and Manzini. In each region, 37 of the enumeration areas were classified as rural by the Swaziland Statistics Office, and 13 as urban. In each EA, we selected 20 households through systematic random sampling. Data collectors administered a questionnaire in SiSwati to each household member aged 11 years or older who was present at the time of the household visit and who provided written consent to participate in the survey. Due to feasibility constraints, the data collection team did not revisit households if no household members were present at the time of the visit.

The RHM questionnaire was administered in SiSwati to all RHMs working in the EAs that were selected for the household survey. Since the EAs selected for the household survey were only a relatively small subsample of all EAs in the Lubombo and Manzini region of Swaziland, 306 (12.0%) out of a total of 2,543 RHMs in these two regions were interviewed. The RHM questionnaire was administered at the RHM’s household by the same cadre of data collectors, which conducted the household survey.

Both the household and RHM survey were conducted between June 2015 and September 2015. Quantitative analyses consisted of descriptive statistics (means and proportions) and were conducted in Stata version 13.0 (College Station, TX, USA).

Qualitative methodology

Qualitative data were collected through semi-structured interviews with 25 RHMs (Supplementary File 3). These RHMs comprised a criterion-based stratified purposive sample. Strata used were region (13 RHMs from Manzini and 12 from Lubombo region) and urban versus rural (13 from rural areas and 12 from urban areas in each region). Additional sampling criteria were age and sex of RHMs, attempting to yield a sample that is similar to the age and sex distribution of the RHM cadre in general. In addition, we conducted semi-structured qualitative interviews with the chief RHM program manager in the program office in Mbabane, Swaziland, and five RHM trainers in the regional offices of the RHM program.

Five recent graduates of the University of Swaziland Social Science Program who were fluent in SiSwati and English conducted the interviews. The data collectors were Swazi and aged between 20 and 35 years. The interviews lasted between 30 and 45 minutes and were conducted in SiSwati. The interviewers taped the interviews, and transcribed them verbatim in SiSwati. The transcripts were then translated into English by the local study coordinator, who is also an author of this paper (MM). He also conducted a quality check of each transcript. Two authors (MV and PG) conducted content analysis using an inductive approach to coding²³. We identified broad themes after an initial review of the data, and then conducted iterative reviews to further refine themes and their relationships to each other. All coding was done using NVivo 11 (QSR International, Melbourne, Australia).

Evaluation framework

The evaluation framework that was used for this performance evaluation is the CHW Performance Logic Model (Figure 1), which has been described in detail elsewhere¹⁸. The model was used to inform the design of the questionnaires and interview guides. More specifically, the data collection tools contained questions on the dimensions (white rectangles in Figure 1), which in turn were grouped under sections corresponding to the dimensions of the model (results, activities, and inputs). Questions in the household survey questionnaire focused on CHW program outcomes by asking about the household members’ experiences with the RMH program and the degree to which they sought care from RHMs. Meanwhile the RHM questionnaire focused on CHW program outputs (e.g., self-reported performance, and job satisfaction and motivation), and support provided to RHMs by the community and health system (and actors within these systems). Data on inputs was obtained from program reports and personal meetings with the RHM program management. We have structured the results section according to the logic model dimensions, moving from the inside (CHW performance outcomes) to the outside (inputs) of the model depicted in Figure 1.

Figure 1. The Community Healthcare Worker Performance Logic Model.

Adapted from Naimoli et al.¹⁸

Ethics

This study was approved by the Swaziland Ethics Committee on March 31^st 2015 (reference number: MH/599C/FWA 000 15267/IRB 000 9688), and received an exemption by the institutional review board of the Harvard T. H. Chan School of Public Health on March 31^st 2015. Written informed consent was obtained from all study participants.

Results

Sample characteristics

The RHM questionnaire was administered to a total of 306 RHMs, 96.1% of whom were female (Table 1). On average, RHMs were 52.9 years old (SD: 11.6 years) with 16 RHMs (5.2%) older than 70 years. RHMs had lived in their communities for an average of 34.6 years (SD: 16.5 years) and had worked in the RHM program for 15.5 years (SD: 12.9 years). 30.5% of RHMs reported to have done work other than for the RHM program during the previous 12 months. The characteristics of the 25 RHMs with whom we conducted semi-structured qualitative interviews were similar to those of the sample of RHMs who were included in the RHM survey. The population-based household survey was administered to 2,342 household members across 2,000 households. 97.7% of household survey respondents had lived in the surveyed community for more than one year.

Table 1. Socio-demographic characteristics of household and RHM survey respondents.

Characteristic	RHM survey (n=306)	RHM interviews (n=25)	Household survey (n=2,342)
Female	96.1%	80.0%	31.2%
Mean age (years)	52.9 (11.6)	47.4 (12.3)	37.0 (18.8)
Region
Lubombo	51.0%	40.0%	58.3%
Manzini	49.0%	60.0%	41.7%
Head of household	43.3%	62.5%	63.0%
Married	73.3%	66.7%	44.5%
Mean no. of years lived in this community	34.6 (16.5)	35.3 (17.2)	22.8 (18.1)
Lived in this community for at least 1 year	100.0%	100.0%	97.7%
Educational achievement None Primary school Secondary school > Secondary school	5.6% 43.5% 35.0% 16.0%	4.2% 37.5% 37.5% 20.8%	16.9% 30.2% 38.9% 14.1%
Currently in full- or part- time education	3.6%	4.0%	20.0%

Standard deviations are shown in brackets. Abbreviations: RHM=Rural Health Motivator; No. = number

CHW performance

As described in the methods, we assessed performance of the RHM program on the output and outcome level of the CHW Performance Logic Model¹⁸. Table 2 summarizes our quantitative findings.

Outcomes: Satisfaction with the RHM program. Household survey respondents’ overall satisfaction with RHM services was mixed, with 46.3% of respondents rating their satisfaction as greater or equal to eight on a 10-point scale ranging from very dissatisfied to very satisfied (Table 2 and Figure 2). 20.4% of respondents rated their satisfaction as less than five on this scale. Nonetheless, the vast majority (96.1%) of respondents would recommend the RHM program to other communities.

Table 2. Outcomes and outputs.

Outcome level	Percentage/Mean (95% CI)
Satisfaction with the RHM program
% of community members¹ satisfied with the services provided by the RHMs in their community²	46.3 (43.4 – 49.6)
% of community members¹ satisfied with the accessibility of the RHMs in their community²	49.8 (46.6 – 53.1)
% of community members¹ satisfied with the quality of the advice and care given by the RHMs in their community²	49.4 (46.1 – 52.7)
% of community members¹ who would recommend the RHM program to other communities²	96.1 (94.6 – 97.2)
Care-seeking
% of RHMs who report that they have been approached by community members for help or advice	76.7 (71.3 – 81.1)
RHMs’ standing in the community
% of RHMs reporting that the RHM program increased their community standing	74.3 (68.2 – 78.4)
Output level	Percentage/Mean (95% CI)
Quantity of work performed
Mean no. of households RHMs report to have been assigned	29.8 (28.14 – 31.49)
% of RHMs who report to have visited all assigned households in the last one month	15.7 (11.4 – 20.4)
% of RHMs who report to have visited all assigned households in the last six months	57.8 (50.8 – 64.6)
% of RHMs reporting to take off frequently 2 weeks or more	7.6 (4.8 – 11.1)
% of RHMs who agree or strongly agree that the amount of work they are expected to finish each week is reasonable	92.1 (88.4 – 94.8)
Job satisfaction
% of RHMs who are satisfied or very satisfied with their job	92.4 (88.7 – 95.1)
% of RHMs who are proud to be working for the RHM program	95.0 (91.9 – 97.2)
% of RHMs who would recommend the RHM program to others as a good organization to work for	93.7 (90.3 – 96.2)
% of RHMs who are glad to be working for the RHM program rather than other CHW programs	92.0 (88.4 – 94.8)
% of RHMs who occasionally or often think about leaving their job	26.2 (21.3 – 31.5)

Abbreviations: CI = Confidence interval; RHM = Rural Health Motivator; % = percentage.

¹ This question was only asked to community members who reported to have ever been visited by a RHM.

² This was defined as reporting ≥8 on a 10-point scale from “very dissatisfied” to “very satisfied”.

Figure 2. Histograms of satisfaction with the rural health motivator (RHM) program among household survey respondents^1,2.

¹This question was only asked to household survey respondents who reported that their household had ever been visited by a RHM (n=1, 151). ²Satisfaction was measured on a scale ranging from 1 (“very dissatisfied”) to 10 (“very satisfied”).

Outcomes: Care-seeking from RHMs. 76.7% of RHMs indicated that households had approached them for help or advice. However, in the qualitative interviews, a topic that emerged is that although households did approach RHMs, it was either rare or infrequent. In cases where RHMs were approached, it was usually for acute emergency care:

Interviewer: “How often are you contacted for help or advice?”
RHM: “It is rare … sometimes when someone is in labor then they call me for help” (Manzini)

In the less common scenario where RHMs indicated that they were contacted frequently, it tended to be for material assistance such as medication, diapers, or gloves:

Interviewer: “How often are you contacted for help or advice?”
RHM: “About 3 times a week. They usually want disposable diapers, gloves, or ORS [oral rehydration therapy]” (Lubombo)

Outcomes: RHMs’ standing in the community. In general, RHMs felt that their standing within their communities had increased as a result of them being part of the RHM program. 74.3% indicated that their standing had increased, while only 16.5% stated that their standing had decreased, with the remainder answering that their standing had remained unchanged. In the qualitative interviews, when asked about the effect of their work as a RHM on their community standing, RHMs who indicated an increase in community standing suggested that RHMs’ responsibilities mean that community members respect them more. In cases where RHMs indicated that their community standing remained unchanged or had decreased, these were accompanied by the perception that they did not meet the expectations of community members:

“No, I think [my community standing] is the same especially because people complain that we do not bring them anything except information; they want material things” (RHM, Lubombo)

Outputs: Quantity of work performed. According to the RHM program management, RHMs are responsible for 25 households, which they are to visit at least once a month. In the RHM survey, RHMs reported to be responsible for visiting an average of 29.8 households. Less than a quarter of RHMs (15.7%) reported to have visited all households assigned to them in the last one month, and 57.8% stated they had visited all assigned households at least once in the last six months. The vast majority of RHMs (92.1%) reported that the workload expected of them is reasonable.

Part of the qualitative interviews with RHMs focused on the reasons for not being able to visit all assigned households at least once a month. Four main factors were mentioned most frequently by RHMs: 1) the availability of the client, 2) physical distance to the household, 3) clients’ acceptability of the RHMs, and 4) the inability of RHMs to meet the expectations of some clients. Typical quotes illustrating each of these factors are:

Client availability: “Sometimes there are no people in the household I visit and I have to return on another day” (RHM, Lubombo)
Physical distance to the household: “I find it to be very easy since the households I am responsible for are nearby and I do not need to walk a long distance” (RHM, Manzini)
Acceptability of RHMs: “It is easiest with the homes where people are educated about the health issues and understand our work as RHMs; in homes where this is not the case, they are normally hostile towards us…” (RHM, Manzini)
Inability to meet clients’ expectations: “It is very difficult… people expect motivators to come with material things like [disposable diapers] napkins for their bedridden relatives, but we do not have these things. This disappoints the people and they start to develop an attitude towards us.” (RHM, Manzini)

Outputs: Job satisfaction. Roughly half of RHMs reported to be satisfied or very satisfied with their job. Most RHMs (93.7%) would recommend the RHM program as a good organization to work for, and 95.0% of RHMs answered that they were proud to be working for the RHM program. Roughly a quarter (26.2%) of RHMs reported to occasionally or often think about leaving their job.

Program-level activities

Table 3 summarizes the results for the indicators used to evaluate program-level activities (as defined by the CHW Performance Logic Model¹⁸).

Table 3. Program-level activities.

Social Support	Percentage (95% CI)
Community
RHMs who expressed that their community is somewhat supportive or very supportive of their work	89.8 (85.8 – 93.0)
RHMs who received the following types of support from their community
Verbal support	95.1 (92.0 – 97.2)
Financial support	3.3 (1.6 – 6.0)
Equipment for work	9.2 (6.2 – 13.0)
In-kind support	10.9 (7.6 – 14.9)
Special privileges	5.9 (3.5 – 9.2)
Facility
RHMs who interact regularly with facility-based healthcare workers	93.0 (89.5 – 95.6)
RHMs who expressed that facility-based healthcare workers were somewhat supportive or very supportive of their work¹	96.5 (93.6 – 98.3)
RHMs who felt that the facility-based healthcare workers value their work¹	95.4 (92.3 – 97.5)
Family
RHMs who expressed that their families were somewhat supportive or very supportive of their work	95.7 (92.7 – 97.7)
*Technical Support*
RHMs who indicated that their job responsibilities were either well explained or very well explained to them	99.0 (97.1 – 99.8)
RHMs who either agreed or strongly agreed that they received all the training necessary for them to perform their jobs	94.7 (91.5 – 96.9)
RHMs who rated the quality of their in-service training as being high²	81.9 (76.8 – 86.2)
*Incentives*
Monetary
RHMs reporting to not being paid for their work	11.4 (8.1 – 15.5)
RHMs who disagree or strongly disagree with the statement “Given the amount of work I do as a RHM, I am being paid a fair amount”	57.8 (50.0 – 63.4)
Non-monetary
RHMs who received any non-monetary payments from the program	11.5 (8.1 – 15.6)
RHMs who reported to have received the following types of non-monetary payments from the program³
Food	17.1 (6.6 – 33.6)
Livestock	5.7 (0.6 – 19.2)
Free access to social support services	22.9 (10.4 – 40.1)
Equipment (e.g., mobile phones and uniforms)	35.3 (19.7 – 53.5)
Exemption from other community duties	37.1 (21.5 – 55.1)

Abbreviations: RHM=rural health motivator.

¹ This question was only asked if the RHM reported to have regularly interacted with facility-based healthcare workers (93.0%).

² This was defined as reporting ≥8 on a 10-point scale from “very bad quality” to “very high quality”.

³ The denominator for these percentages is the number of RHMs who reported having received non-monetary payments from the program.

Social support. The majority of RHMs indicated that they were somewhat or very well supported by members in their communities (89.8%), by their families (95.7%), and by facility-based healthcare workers (96.5%). The vast majority of RHMs (95.4%) felt that facility-based colleagues value their work.

Technical support. The initial training for new RHMs lasts 12 weeks full-time. In addition, the program runs in-service trainings, which re-emphasize certain topics taught during the initial training and usually also cover some new material. These refresher trainings last for two to five days and are conducted once a year for each RHM. Only 10.5% of RHMs surveyed reported to never have attended an in-service training. Most RHMs (94.7%) either agreed (48.3%) or strongly agreed (46.4%) that the training provided by the program is sufficient to competently perform their work as a RHM. 81.9% rated the quality of their in-service training as being high.

Incentives. The majority of RHMs expressed dissatisfaction with the compensation offered. 57.8% either disagreed (38.0%) or strongly disagreed (19.8%) with the statement that “Given the amount of work I do as a rural health motivator, I am being paid a fair amount”. This is also reflected in the qualitative data, in which RHMs frequently mentioned that they do not feel that they are sufficiently compensated. A typical opinion expressed in this regard is:

“I do not feel I am being paid a fair amount because there is a lot of work that we do. Sometimes the families desert the ill patients and leave them in their own dirt until the day a RHM comes along and bathes the patient, feeds them....so the work is quite a lot” (RHM, Lubombo)

Very few RHMs reported to have received non-monetary compensation from the RHM program.

System-level activities

Table 4 summarizes the results for the indicators used to evaluate system-level activities.

Table 4. System-level activities.

Leadership and Governance	Percentage (95% CI)
RHMs who agreed or strongly agreed that the RHM program is supportive of them	97.7 (95.2 – 99.1)
RHMs who agreed or strongly agreed that they are able to easily communicate with members from all levels of the RHM program	97.0 (94.4 – 98.6)
RHMs who agreed or strongly agreed that that the RHM program rules make it easy for them to do their jobs	97.0 (94.4 – 98.6)
RHMs who indicated that they receive feedback from their supervisors	91.8 (88.1 – 94.6)
RHMs who were satisfied or very satisfied with the level of supervision they receive	76.0 (70.8 – 80.7)
RHMs who would like to receive more supervision	65.3 (59.6 – 70.6)
*Resource Mobilization*
RHMs who disagreed or strongly disagreed that the RHM program provides all the equipment, supplies, and material resources necessary for them to perform their duties	60.6 (54.8 – 66.1)

Abbreviations: RHM=rural health motivator

Leadership and governance. Among RHMs, 55.8% agreed and 41.9% strongly agreed that the RHM program management was supportive of their work. Most either agreed (53.0%) or strongly agreed (44.0%) with the statement that “the RHM program rules make it easy for me to do a good job”. Similarly, virtually all RHMs (97.0%) expressed that it was generally easy to communicate with members from all levels of the RHM program. Concerning supervision, 91.8% of RHMs indicated that supervisors provide feedback on their work. While 76.0% of RHMs were satisfied (58.6%) or very satisfied (17.4%) with the level of supervision that they receive, 65.3% indicated that they would like to receive more supervision. Qualitatively, in cases where RHMs expressed interest in additional supervision, the reason tended to be that they felt additional feedback would help motivate them further and support continued learning, as illustrated by the following quote:

“I would like more supervision because it would help me learn and grow my skills as a RHM. Additionally, it helps to keep me motivated and to put in more effort in my work” (RHM, Manzini)

Provision of material resources. 60.6% of RHMs either disagreed (40.1%) or strongly disagreed (20.5%) that the program provides all the equipment, supplies, and material resources necessary to perform their duties.

Inputs

Human resources. The RHM program had 5,214 RHMs in 2015, of which roughly half (2,803) lived and worked in the Lubombo or Manzini region. In addition, the program had one program manager, one program officer, one administrative assistant, 18 RHM trainers (who are trained nurses), and two drivers.

Capital resources The RHM program occupies four offices in the country, one in each of Swaziland’s four regions. The program also owns two cars.

Costs. Table 5 shows the running costs of the RHM program for 2011 using data from the Kingdom of Swaziland Budget versus Expenditure Report 2012²⁴, which was the latest data available to us. We present these costs in terms of purchasing power parity dollars (PPP$). One PPP$ is calculated such that it had the same purchasing power in Swaziland in 2011 as one US dollar had in the United States in that year. Roughly two thirds of the program costs are spent on salaries for the RHMs. As of 2015, RHMs earned 350 Swazi Lilangeni per month, which is approximately US$ 22.50 (PPP$ 73.22).

Table 5. Cost of the RHM program in 2011.

Cost item	PPP$¹
Human resources
RHM salaries²	10,006,500
Head office (program manager, program officer, drivers)	106,529
Capital resources	766,238
RHM uniforms	251,038
RHM training	4,078,071
Total	15,208,376

Abbreviations: PPP$ = Purchasing power parity-adjusted dollars; RHM, rural health motivator.

¹ This is the PPP$ value for 2011 (i.e., not further adjusted for inflation since 2011). The PPP

conversion factor for Swaziland for 2011 was obtained from the United Nations Statistics Division²⁹.

² In 2011, the RHM program employed 4,765 RHMs.

Dataset 1.Household (head and member) survey raw data.

Dataset 2.Rural health motivator survey raw data.

Discussion

This evaluation identified a number of weaknesses in the RHM program’s performance. First, despite being in close geographic proximity to their clients, the Swazi population appears to prefer seeking care from other healthcare workers than the RHM cadre. As found in particular through our qualitative interviews, community members rarely seek care from RHMs, and if they do, this tends to be for emergency care when care from other health care providers is unavailable. Second, client satisfaction with the RHM program appears to be comparatively low. The survey data on client satisfaction is likely to suffer from some degree of courtesy or social desirability bias whereby community members give a more favorable assessment of the RHMs’ care to abide by a perceived social norm of showing satisfaction and gratitude rather than criticism²⁵. Despite the possibility of this bias, a comparatively low proportion (46.3%) of community members rated their overall satisfaction with RHM services as eight or more points on a 10-point scale ranging from very dissatisfied to very satisfied. Third, RHMs do not appear to provide the quantity of care that the program aims to provide. Data on the number of households visited by RHMs are self-reported and may, thus, also suffer from an upward bias as RHMs are likely to want to appear as fulfilling their duties. Despite this likely bias, only 15.7% of RHMs reported achieving the program target of visiting all assigned households at least once a month. Overall, improving the performance of the RHM cadre may be necessary to successfully shift HIV care tasks from facility-based to RHM-led care.

Our assessment of the RHM program on the program- and system-level dimensions of the CHW Performance Logic Model provides some insight into factors that might be lowering RHM performance. In general, RHMs report that they are satisfied with the quantity and quality of training and supervision provided to them. However, RHMs are dissatisfied with the level of monetary compensation, with 57.8% of RHMs indicating that the level of their pay is unfair given the amount of work they do. In 2015, RHMs earned 350 Swazi Lilangeni (approximately US$ 22.50) per month. Additionally, in the qualitative interviews, RHMs reported that they face transport costs and bank fees to collect and cash their paycheck. Swaziland’s national poverty line lies at US$ 3.10 per day²⁶. Ignoring costs to collect and cash their paycheck, RHMs earn approximately US$ 0.74 per day, which is only 23.9% of the daily income needed to be earning at the national poverty line. Expectations of RHM performance need to be examined in light of this comparatively low level of pay. The low pay is likely also an obstacle for shifting HIV care tasks to RHMs, as many of these tasks, such as ART home-delivery, require reliable and constant care. A theme arising from our qualitative interviews, however, was that RHMs view themselves as volunteers rather than employees given their low level of pay. It would thus seem likely that other income-generating activities take priority over RHM work, which in turn may lead to prolonged gaps in RHM care delivery.

Apart from monetary compensation, RHMs were also dissatisfied with the material resources provided to them by the RHM program for performing their duties. In the qualitative interviews, RHMs frequently mentioned that community members expect them to provide certain material resources, such as diapers, medications (particularly paracetamol), bandages, and disposable gloves. RHMs felt that not being able to meet this expectation was an important barrier in maintaining a good relationship with the community, and to cover the households that they were assigned. Thus, providing the expected material resources to RHMs and/or altering the expectations of community members to receive such resources from RHMs may increase RHM performance. Improving the RHM-client relationship is of particular importance if RHMs are to provide more HIV care given the continued high HIV-related stigma in Swaziland²⁷.

We used the CHW Performance Logic Model to guide this performance evaluation. While the logic model aims to be a useful tool for planning, consensus-building, implementation, and evaluation of CHW programs¹⁸, we can only comment on our experience with the model’s usefulness for CHW program evaluations. A key characteristic of the model is that it tries to comprehensively include all factors that may influence CHW performance. As such, the logic model differs strongly from the more simplistic framework of inputs – processes – outputs that we have previously used for a performance evaluation of a CHW program in Dar es Salaam, Tanzania²⁸. In our view, the comprehensive nature of the logic model is its key strength. Given the sheer number of possible factors that may plausibly influence CHW program performance, most evaluators will have to make a decision regarding the scope of their evaluation. The CHW Performance Logic Model could help evaluators clearly define the evaluation’s scope, and be more explicit about their choice of which factors and domains they include in the evaluation. Nonetheless, the model’s comprehensive nature could be a disadvantage if evaluators find the number of possible factors to evaluate overwhelming. In our view, the main disadvantage of the model is that it does not provide any guidance to evaluators on which factors are the most important determinants of CHW performance. As such, a prioritization of the categories and factors in the model based on relevant theory and evidence, rather than an un-weighted list of all factors that plausibly influence CHW performance, would substantially improve the utility of the model. Another limitation of the model is that many of the performance measures and factors assessed under the model’s dimensions lack established measures and scales. In addition, there are doubts as to whether a dimension is measured appropriately, which also results in some degree of subjectivity in interpreting what level of CHW program performance the observed achievement on a measure represents.

Other limitations of this study include that the data from the RHM questionnaire are likely to suffer from a degree of self-reporting bias whereby RHMs may, for example, over-report aspects of their work that they perceive as desirable (e.g., the number of households visited). Similarly, household survey respondents may have been hesitant to express criticism of RHMs because they wanted to maintain a good relationship with the RHMs (who are fellow community members chosen by the community and the village chiefs), or simply due to an intrinsic tendency to be courteous. Lastly, while the RHM program is a national program, this assessment has focused on only two of four regions in Swaziland due to feasibility constraints. However, these two regions constitute more than half (52%) of Swaziland’s population, and the program structures for management and implementation of the RHM cadre do not differ between regions. We, therefore, feel confident that the findings of this study apply to the RHM program as a whole.

Conclusions

This evaluation found that the RHM program does not meet some of its performance targets. For instance, RHMs are currently not an important point of first call for seeking care for an illness, and the RHMs do not appear to achieve their household coverage target. If the RHM program is to adopt specific HIV-related tasks, then Swaziland’s HIV response would likely benefit from policy and management changes aimed at improving RHM performance. While it is beyond the purview of this study to provide an exhaustive list of suitable reforms, two simple changes identified by this evaluation that may lead to an improvement in RHM performance are i) an increase in monetary compensation, and ii) the provision of material resources to RHMs (e.g., paracetamol, diapers, and bandages) to enable RHMs to meet their community’s expectations.

Data availability

Please note that some items have been removed/edited due to potentially identifiable information. The datasets contain both CSV and .dta files.

Dataset 1: Household (head and member) survey raw data. doi, 10.5256/f1000research.11361.d158777³⁰

Dataset 2: Rural health motivator survey raw data. doi, 10.5256/f1000research.11361.d158778³¹

Qualitative interview manuscripts are not shared publicly because they cannot be effectively de-identified given the relatively small number of staff involved in the studied community health worker programs. Individuals interested in accessing the transcripts should contact the corresponding author.

Author contributions

PG and MV analyzed the data and wrote the first draft of the manuscript. The authors (PG, MV, JWD, TB, TJB) jointly designed the study and data collection tools. All authors (PG, MV, JWD, TB, TJB) provided important edits to the manuscript and approved the final version.

Competing interests

No competing interests were disclosed.

Grant information

Data used in this study were collected for other activities supported by the American people through the United States Agency for International Development (USAID) with funding from the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). The data were collected by the Harvard T.H. Chan School of Public Health through the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project. The USAID ASSIST Project is managed by University Research Co., LLC (URC) under the terms of Cooperative Agreement AID-OAA-A-12-00101. The authors’ views expressed in this paper do not necessarily reflect the views of USAID or the United States Government.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript

Supplementary material

Supplementary File 1: Household survey questionnaires consisting of a questionnaire for all household members aged 11 years and older and an additional questionnaire for the household head.

Click here to access the data.

Supplementary File 2: Questionnaire for the survey of rural health motivators.

Click here to access the data.

Supplementary File 3: Interview guide for RHMs, RHM trainers, and the RHM program management.

Click here to access the data.

Faculty Opinions recommended

References

1. World Health Organization: World Health Statistics 2016. Geneva: World Health Organization, 2016. Reference Source
2. Singh P, Sachs JD: 1 million community health workers in sub-Saharan Africa by 2015. Lancet. 2013; 382(9889): 363–5. PubMed Abstract | Publisher Full Text
3. Hongoro C, McPake B: How to bridge the gap in human resources for health. Lancet. 2004; 364(9443): 1451–6. PubMed Abstract | Publisher Full Text
4. Mwai GW, Mburu G, Torpey K, et al.: Role and outcomes of community health workers in HIV care in sub-Saharan Africa: a systematic review. J Int AIDS Soc. 2013; 16(1): 18586. PubMed Abstract | Publisher Full Text | Free Full Text
5. Viswanathan M, Kraschnewski J, Nishikawa B, et al.: Outcomes of community health worker interventions. Evid Rep Technol Assess (Full Rep). Rockville, MD: RTI International–University of North Carolina Evidence-based Practice Center. 2009; 181: 1–144; A1-2, B1-14, passim. PubMed Abstract | Free Full Text
6. Gilmore B, McAuliffe E: Effectiveness of community health workers delivering preventive interventions for maternal and child health in low- and middle-income countries: a systematic review. BMC Public Health. 2013; 13: 847. PubMed Abstract | Publisher Full Text | Free Full Text
7. Lewin S, Munabi-Babigumira S, Glenton C, et al.: Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases. Cochrane Database Syst Rev. 2010; (3): Cd004015. PubMed Abstract | Publisher Full Text
8. Perry HB, Zulliger R, Rogers MM: Community health workers in low-, middle-, and high-income countries: an overview of their history, recent evolution, and current effectiveness. Annu Rev Public Health. 2014; 35: 399–421. PubMed Abstract | Publisher Full Text
9. East Central and Southern African Health Community: Task shifting in Swaziland: A case study. Washington, DC: Futures Group, Health Policy Initiative, Task Order 1, 2010. Reference Source
10. GBD 2013 DALYs and HALE Collaborators, Murray CJ, Barber RM, et al.: Global, regional, and national disability-adjusted life years (DALYs) for 306 diseases and injuries and healthy life expectancy (HALE) for 188 countries, 1990-2013: quantifying the epidemiological transition. Lancet. 2015; 386(10009): 2145–91. PubMed Abstract | Publisher Full Text | Free Full Text
11. UNAIDS: 90-90-90 An ambitious treatment target to help end the AIDS epidemic. Geneva: UNAIDS, 2014. Reference Source
12. UNAIDS, One Million Community Health Workers: UNAIDS joins forces with the One Million Community Health Workers campaign to achieve the 90–90–90 treatment target. 2016, (accessed 24 March 2016). Reference Source
13. World Health Organization, PEPFAR, UNAIDS: Task shifting - Global recommendations and guidelines. Geneva: World Health Organization, 2008. Reference Source
14. Jaffar S, Amuron B, Foster S, et al.: Rates of virological failure in patients treated in a home-based versus a facility-based HIV-care model in Jinja, southeast Uganda: a cluster-randomised equivalence trial. Lancet. 2009; 374(9707): 2080–9. PubMed Abstract | Publisher Full Text | Free Full Text
15. Selke HM, Kimaiyo S, Sidle JE, et al.: Task-shifting of antiretroviral delivery from health care workers to persons living with HIV/AIDS: clinical outcomes of a community-based program in Kenya. J Acquir Immune Defic Syndr. 2010; 55(4): 483–90. PubMed Abstract | Publisher Full Text
16. Geldsetzer P, Francis JM, Ulenga N, et al.: The impact of community health worker-led home delivery of antiretroviral therapy on virological suppression: A non-inferiority cluster-randomized health systems trial in Dar es Salaam, Tanzania. BMC Health Serv Res. 2017; 17(1): 160, In press. PubMed Abstract | Publisher Full Text | Free Full Text
17. ICAP-Swaziland: RHM Review 2012. Mbabane: Columbia University, 2012.
18. Naimoli JF, Frymus DE, Wuliji T, et al.: A Community Health Worker "logic model": towards a theory of enhanced performance in low- and middle-income countries. Hum Resour Health. 2014; 12: 56. PubMed Abstract | Publisher Full Text | Free Full Text
19. Central Statistical Office: 2007 Population and Housing Census. Mbabane, Swaziland: The Kingdom of Swaziland, UNFPA, 2010; 6. Reference Source
20. Central Statistical Office, Macro International Inc: Swaziland - Demographic and Health Survey 2006-2007. 2007. Reference Source
21. Ministry of Health: Swaziland HIV incidence measurement survey (SHIMS). Mbabane: Kingdom of Swaziland, 2012. Reference Source
22. Bicego GT, Nkambule R, Peterson I, et al.: Recent patterns in population-based HIV prevalence in Swaziland. PLoS One. 2013; 8(10): e77101. PubMed Abstract | Publisher Full Text | Free Full Text
23. Sandelowski M: Whatever happened to qualitative description? Res Nurs Health. 2000; 23(4): 334–40. PubMed Abstract | Publisher Full Text
24. The Kingdom of Swaziland: Budget versus Expenditure Report. Mbabane, Swaziland, 2012.
25. Glick P: How reliable are surveys of client satisfaction with healthcare services? Evidence from matched facility and household data in Madagascar. Soc Sci Med. 2009; 68(2): 368–79. PubMed Abstract | Publisher Full Text
26. Oxford Poverty and Human Development Initiative: OPHI Country Briefing Dec 2015: Swaziland. Oxford, UK: Oxford Department of International Development, University of Oxford, 2015.
27. Tsai AC: Socioeconomic gradients in internalized stigma among 4,314 persons with HIV in sub-Saharan Africa. AIDS Behav. 2015; 19(2): 270–82. PubMed Abstract | Publisher Full Text | Free Full Text
28. Lema IA, Sando D, Magesa L, et al.: Community health workers to improve antenatal care and PMTCT uptake in Dar es Salaam, Tanzania: a quantitative performance evaluation. J Acquir Immune Defic Syndr. 2014; 67(Suppl 4): S195–201. PubMed Abstract | Publisher Full Text | Free Full Text
29. United Nations Statistics Division. Purchasing power parities (PPP) conversion factor, local currency unit to international dollar. (accessed September 8 2016) 2016. Reference Source
30. Geldsetzer P, Vaikath M, de Neve JW, et al.: Dataset 1 in: A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program. F1000Research. 2017. Data Source
31. Geldsetzer P, Vaikath M, de Neve JW, et al.: Dataset 2 in: A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program. F1000Research. 2017. Data Source

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 02 May 2017

Author details Author details

¹ Department of Global Health and Population, Harvard T.H. Chan School of Public Health, Boston, MA, 02115, USA
² Institute of Public Health, Heidelberg University, Heidelberg, 69120, Germany
³ Africa Health Research Institute, University of KwaZulu-Natal, Mtubatuba, 3935, South Africa

Competing interests

No competing interests were disclosed.

Grant information

Data used in this study were collected for other activities supported by the American people through the United States Agency for International Development (USAID) with funding from the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). The data were collected by the Harvard T.H. Chan School of Public Health through the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project. The USAID ASSIST Project is managed by University Research Co., LLC (URC) under the terms of Cooperative Agreement AID-OAA-A-12-00101. The authors’ views expressed in this paper do not necessarily reflect the views of USAID or the United States Government.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (1)

version 1

Published: 02 May 2017, 6:607

https://doi.org/10.12688/f1000research.11361.1

Copyright

© 2017 Geldsetzer P et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

0

SEE MORE DETAILS

CITE

how to cite this article

Geldsetzer P, Vaikath M, De Neve JW et al. A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program [version 1; peer review: 2 approved with reservations]. F1000Research 2017, 6:607 (https://doi.org/10.12688/f1000research.11361.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 02 May 2017

Views

8

Reviewer Report 09 Nov 2017

Eilish McAuliffe, School of Nursing, Midwifery and Health Systems, University College Dublin, Dublin, Ireland

Approved with Reservations

https://doi.org/10.5256/f1000research.12265.r25132

General comment
This study sets out to evaluate the performance of Swaziland’s Rural Health Motivator Programme. As most resource-poor countries with human resources shortages in their health sectors are employing CHWs as a strategy to respond to the health ... Continue reading

General comment
This study sets out to evaluate the performance of Swaziland’s Rural Health Motivator Programme. As most resource-poor countries with human resources shortages in their health sectors are employing CHWs as a strategy to respond to the health needs of communities, evaluations of this strategy are highly relevant for health policy makers. Gathering data from households and CHWs has the potential to generate a much better understanding of how these programmes work or don’t work. This study is impressive in the large amount of data gathered from different stakeholder perspectives. It is a well-written paper, but with some critical omissions as set out below.

Introduction
A brief discussion on published evaluations of CHW programmes (e.g. well established programmes such as the Health Extension Workers programme in Ethiopia and the Lady Health Workers in Pakistan) would help to situate this research in the field of study and enable the authors to better articulate the study’s contribution to the field in their discussion and conclusions part of the paper.
Some description of RHMs and how they differ from CHWs would be helpful.

Methods
A rationale needs to be provided for the study design. Why were household surveys and RHM surveys conducted? Was it the authors intention to cross reference the results? What was the specific purpose of the qualitative interviews? What is the rationale for the sample size 25? No information is provided on how the interviews map to the CHW Performance Logic Model. It is not clear if the interviews were conducted after the analysis of the quantitative data and whether they were used to explore issues arising from questionnaire results in more depth or to address some other aspect of the Logic Model. The rationale for the study design and the specific purpose of each of the methods needs to be included.
Authors need to include some explanation of the CHW Performance Logic Model and why it was considered most relevant for this study.
Performance is the key outcome variable in this study, yet the authors have not defined what this term means in the context of this study.
The authors state that if no household members was available at first visit, they did not revisit. They ought to give some consideration to the potential bias this may have introduced into the study. For instance, it may be that those who were employed and out at work were less likely to be available than those who were unemployed. No information is provided about the time the visits took place, so it is difficult to know if this bias could have occurred. Some further expansion of the methods and/or consideration of this issue in the limitations of the study is needed.

Results
A strength of the study is the use of three sources of data and the opportunity this presents to triangulate data. The relatively large sample sizes should allow for a robust interrogation of the data. It is therefore disappointing that the analysis is rather superficial and confined mainly to descriptive stats. There are some obvious questions that remain unanswered because the relevant data have not been triangulated. For example, the RHMs reporting of rate of household visits could have been triangulated the household members response to Questions 2.8-2.14 to give a much better understanding of the productivity of the RHMs. Also there are many questions in the household member study that explore the performance of RHMs the results of which are not presented at all in this paper. If these results have been presented elsewhere, it would be helpful to refer to the publication.

The statement “30.5% of RHMs reported to have done work other than for the RHM program during the previous 12 months” requires further explanation. Does this mean these RHMs worked elsewhere before joining the program or that they concurrently take on other work while conducting their RHM duties?

57.8% of sample disagree that they are paid a fair amount for the work they do, yet there are several non-monetary payments. Is it possible that these compensate.

It is mentioned that half of respondents were satisfied or very satisfied with their job and then “Roughly a quarter (26.2%) of RHMs reported to occasionally or often think about leaving their job”. Was this issue explored in the qualitative interviews? Why is such a high proportion thinking about leaving their jobs?

I also have concerns about the treatment of the qualitative data. It is not clear if quotes provided are indicative of a majority view or not. Whilst it would not be appropriate to attempt to quantify qualitative data, some sense of how widely held the opinions represented in the quotes are in the sample of interviewees would be appropriate.

The following quote is provided as illustrative of physical distance being a barrier to RHMs visiting the households, but the quote is illustrative of the ease of visiting due to short distance and is therefore not appropriate to illustrate the point above
““I find it to be very easy since the households I am responsible for are nearby and I do not need to walk a long distance””

Discussion
There are several problems with the discussion section of this paper.

In the first paragraph of the discussion on social desirability bias needs to be re-written. The authors suggest that social desirability bias may have impacted the answers to two specific questions, but the results (i.e. low satisfaction with RHM and low rates of productivity reported by RHNs) suggest the opposite is more likely. This needs further discussion and explanation.
There seems to be an element of selectivity about what results are discussed. For example, taking two results presented in the results section: “The majority of RHMs expressed dissatisfaction with the compensation offered. 57.8% either disagreed (38.0%) or strongly disagreed (19.8%) with the statement that “Given the amount of work I do as a rural health motivator, I am being paid a fair amount”. and “65.3% indicated that they would like to receive more supervision. Qualitatively, in cases where RHMs expressed interest in additional supervision, the reason tended to be that they felt additional feedback would help motivate them further and support continued learning”. The first of these results is discussed and a recommendation put forward to increase the monetary reward of RHMs. The second result represents a greater majority and is clearly linked to improved motivation, yet it is not mentioned in the discussion and additional supervision is not put forward as a recommendation.
There are many apparent contradictions in the data that are not discussed e.g. low satisfaction levels with RHM services, but very high percentage of households would recommend the RHM programme to other communities
The discussion section contains statements for which no evidence is provided in the results section e.g. “The low pay is likely also an obstacle for shifting HIV care tasks to RHMs, as many of these tasks, such as ART home-delivery, require reliable and constant care.”
“Qualitative interview manuscripts are not shared publicly because they cannot be effectively de-identified given the relatively small number of staff involved in the studied community health worker programs.” This is not a convincing explanation as 25 interviewees from a sample of 306 would suggest that their anonymity could be protected. If the data cannot be provided, then the extracted text coded to the main reported themes should be made available.
In the limitations, the authors provide a comprehensive account of the weaknesses of the CHW Performance Logic Model. This begs the question as to why they decided to use such a flawed model as the framework for the evaluation, given that this information on the Model was available to them.

Conclusions.
Again some selectivity evident as in the discussion, but some important points are also made regarding the deficiencies in the RHM programme.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Views

10

Reviewer Report 07 Nov 2017

Frédérique Vallières, School of Psychology, Trinity College Dublin, Dublin, Ireland; Center for Global Health, Trinity College Dublin, Dublin, Ireland

Approved with Reservations

https://doi.org/10.5256/f1000research.12265.r26804

Thank you for the opportunity to review this interesting paper by Geldsetzer et al. The authors conducted a cross-sectional, mixed-methods study examining the performance of Swaziland’s RHM programme. The study had two underlying objectives (i) to assess the performance of ... Continue reading

Thank you for the opportunity to review this interesting paper by Geldsetzer et al. The authors conducted a cross-sectional, mixed-methods study examining the performance of Swaziland’s RHM programme. The study had two underlying objectives (i) to assess the performance of the RHM programme, using the CHW Performance Logic Model, and (ii) to identify ways in which performance could be improved. Overall, this is a well-written paper with many strengths including an impressive sample of 2000 households, representative of two of Swaziland’s four administrative regions, and a further 25 semi structured interviews. However, the paper would benefit from a number of important revisions, especially pertaining to a clear definition of performance, the methodological approaches used to define and measure this, and more rigorous analytical approaches in order to adequately address the study’s objectives.

Suggested Revisions

Introduction

While evaluating Swaziland’s RHM program is a worthy endeavour, I would argue that it is unfair to state that ‘many [of the] large programs that have existing for decades have not yet been rigorously evaluated’ (p.3) as the rationale for the current study. For example, there have been a number of evaluations of Ethiopia’s CHEW programme¹ and Pakistan’s LHW programme². Many of these studies also arguably employ more rigorous evaluation approaches than the current study. It is probably sufficient to say that no evaluations of the RHM in Swaziland have taken place to date, and if RHMs are expected to contribute to achieving 90-90-90, then a better understanding of the programme’s progress is required.
More detail is required on the RHM programme in general. Are the RHMs remunerated? If so, who pays them? If not, what non-financial incentives are in place for them, if any? Who supervises them? Are they affiliated with a health centre? Are they trained? If so, for how long? Are they predominantly women? Are they recognised as part of the formal health system?
Additional background on the context of Swaziland, specifically as it relates to their lack of human resources for health, and their need to task shift/share, would also improve the introduction and offer more context for the reader.
A clearer rationale for the importance of assessing performance of CHWs is required. Performance of health workers is a difficult construct to define, and there would be large variation in the literature in terms of how performance is (i) defined and (ii) measured. The paper would benefit greatly from engaging with this literature. Specifically, the paper should engage the literature examining the relationship between job satisfaction, motivation, intention to leave, etc. and performance, in order to build justification for the inclusion of these variables within the questionnaire design.
Did the authors consider other frameworks? If so, why was the CHW Performance Logic Model chosen as the framework for this study? A stronger rationale for why this framework was chosen is required as this model does not measure performance per se, but rather puts forward a theoretical pathway towards performance. What evidence is there to suggest that this model is valid (i.e. predictive validity) in terms of predicting CHW performance? This is also methodological decision and does not belong in the Introduction.

Methods

Study Setting

Clarify why Lubombo and Manzini were chosen as the two study areas. Was this random selection? If not, how were these two chosen? Is there any reason to believe that differences might exist between these two areas? What percentage of Swaziland’s population resides within these two areas?
Who operates the other CHW programmes in Swaziland? While the authors specify that each have a workforce of less than 50CHWs, how many CHWs in Swaziland are part of other programmes (i.e. not RHMs)? If you have the data for these, why not include these and control for ‘CHW programme’ in your analysis?
No need to repeat that the RHM programme was established in 1976 and employs 5000 RHMs again here, as it is already in the introduction.

Study Design & Materials

Clearly state that this study uses a cross-sectional, mixed-methods design.
As above, a clearer operational definition of what is meant by ‘performance’ is needed. What is meant by high or good performance here? How was ‘performance’ defined and measured for the purpose of this study? Why, for example, not include indicators that are aligned to the activities outlined under the RHM program section of the introduction (page 2 para 5)? It seems only fair that performance of the RHMs should, at least in part, be measured against those activities and tasks they are expected to complete. This is a major element missing from this study.
A much more detailed description of the study tools are required. As it stands, it is difficult to evaluate the validity of the scales used without knowing the details of how the survey employed during data collection was designed. For example, have the ‘job satisfaction’, ‘social support’, ‘supervision’, and ‘motivation’ items used in the RHM questionnaire (Part 12 and Part 13, respectively) been used in other studies? If so, which ones? Have these scales been validated? According to these validation studies, how are they meant to be coded/scored?
- Some of the items under Part 13 do not seem to be measuring motivation (i.e. lack face validity). For example, items 13.8 & 13.9 seem to more so be measuring conscientiousness at work, than motivation.

Sampling

It would be quite unlikely in an entirely random sample to have the same number of rural (n=37) and urban (n=13) EAs in each region. Especially since the study setting section describes Manzini as comparatively more urban and wealthy than Lubombo.
What is the rationale for interviewing family members aged 11 years or older? Also, one would expect that those over the age of 18 would provide written consent, but those between 11-17 would provide assent, provided parental consent. Please clarify whether this was the case.
On page 4, first paragraph, please clarify the sampling for the SSIs. The authors state that 13 RHMs from Manzini and 12 from Lubombo were selected, but then go on to specify that 13 were from rural and 12 were from urban areas, in each region. Were there 50 SSIs in total?

Data Analysis

The quantitative data analysis is rather superficial (i.e. descriptive), and unfortunately, does not exploit the richness of the dataset. Why were inferential statistics not employed to look at correlations between certain factors that are known to predict performance (i.e. job satisfaction, motivation at work, supportive supervision) and performance (however this is defined here)? Regression methods, with ‘performance’ as the dependent variable, could be used to more rigorously assess variables that are associated with performance, and are aligned with addressing the second objective of your study: (ii) identify ways in which performance could be improved. The logic here being that improvements made to these factors could result in corresponding changes ‘performance’. Moreover, you would be able to control for differences across Lubombo and Manzini, CHW programes etc.

How were scores calculated? For example, on what basis was it decided that anyone who scored below 8/10 on the Likert scale should not be classified as ‘satisfied with the services provided by the RHMs in their community’? Why recode the answers into dichotomous variables, instead of using the mean score? These methodological decisions need to be justified/described in much greater detail.
- How was missing data treated?

The qualitative data analysis, given the use of a comprehensive interview guide, with broadly pre-determined themes (i.e. the dimensions of the CHW Performance Logic Model), strikes me as having been analysed using more deductive, rather than inductive approaches.
More detail is required as to how you: ‘conducted iterative reviews’ and ‘further refined themes’ and established ‘their relationship to each other’. As it stands, the qualitative description of the analysis is insufficient to ensure replicability.

Results

The results in part read like a report against a logframe or results-based framework than a research paper. As above, the lack of clarity around how dichotomous categories were recoded from the Likert scale makes the results difficult to interpret.
In line with the above comment on inductive vs. deductive approaches, I’m not convinced that HHs approaching RHMs was ‘rare or infrequent’ (pg 6, para 2) is an emerging theme or topic, when the question asked was “How often are you contacted for help or advice?”.
Figure 2 is not very telling. Instead of presenting as a histogram, consider presenting as mean scores for ‘satisfaction’ across the various items in the HH questionnaire, with the Likert Scale range of scores presented on the y-axis.
“Inputs” section is odd here and should form part of the narrative describing the RHM programme in the introduction. RHM salaries in Table 5 should also be reported in USD to give the reader some idea of what this figure represents.
Overall, I’m not convinced that the results presented are sufficient to answer the research objectives (i) to assess the performance of the RHM programme and (ii) to identify ways in which performance could be improved. The results could however, shed some light on how the RHM programme is faring in terms of known determinants of performance.

Discussion

While every programme has room for improvement, I would argue that the results presented are not only suggestive of ‘a number of weaknesses in the RHM’ programme. Sure, resource mobilization and poor pay are an issue (as they are in most places!), but overall, the evidence also suggests a programme that has maintained high community support, a high level of quality training, frequent and supportive levels of supervision from the health facilities, and relatively low levels of people who were thinking of leaving their job. These are all worthy accomplishments for a health programme where resources are extremely limited. Moreover, these are all factors that the literature would strongly suggest are important in predicting health worker performance. While I understand that this type of narrative is what keeps foreign aid funding flowing into programmes, our responsibility as researchers is to present the evidence as objectively as possible. Here, the evidence provided is quite strong for many successful elements of the RHM programme, and these should be discussed too.
Page 10 para 1: “overall, improving the performance of the RHM cadre may be necessary to successfully shift HIV care tasks from facility-based to RHM-led care” seems like an over-interpretation of findings. The evidence presented more so suggests that adequate compensation and more material resources are likely necessary before asking RHMs to take on yet another task within the health system.
The evidence presented offers some insight into why RHMs may not be meeting their HH targets (having to return to homes more than once if a family is absent, attitudes towards HRMs). This should be discussed accordingly as the reason for this observation does not appear to lie solely with the RHM.
The breakdown of income earning on page 10 is interesting, but would be better suited to a description of the RHM programme in the literature review.
Page 10, para 2: “Our assessment of the RHM program…provides some insight into factors that might be lowering RHM performance”. There is no evidence presented to suggest that ‘performance’ decreased in any way. Please rephrase.
Page 10 para 2: “a theme arising from our qualitative interviews, however, was that RHMs view themselves as volunteers rather than…”. I did not see this evidence presented in the results of the paper. Generally speaking, new results should not be presented in the discussion.
Limitations: I disagree that many of the performance-related factors assessed under the CHW Performance Logic Model lack established measures and scales. The I/O psychology literature for example, contains a number of well-developed, cross-culturally validated scales of social support, motivation at work, job satisfaction, etc. scales.

Conclusion

Regarding the statement that, “the evaluation found that the RHM program does not meet some of its performance targets”: Are RHMs really intended to act as the first point of care in this context? Other than not visiting with all 25-30 households on a monthly basis, what other performance target(s) does this statement refer to?

Other (including Figures & Tables)

2^nd para, page 4, there is no need to explain in text what the contributions of each author were. There should be an ‘Author Contributions’ section which allows for this.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

No
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Karim AM, Admassu K, Schellenberg J, Alemu H, et al.: Effect of ethiopia's health extension program on maternal and newborn health care practices in 101 rural districts: a dose-response study.PLoS One. 2013; 8 (6): e65160 PubMed Abstract | Publisher Full Text
2. Douthwaite M, Ward P: Increasing contraceptive use in rural Pakistan: an evaluation of the Lady Health Worker Programme.Health Policy Plan. 2005; 20 (2): 117-23 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Applications of Psychology to Global Health. Namely, Oganisational Psychology, Global Mental Health, Psychotraumatology, and Latent Variable Modelling,

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 02 May 2017

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 1 02 May 17	read	read

Frédérique Vallières, Trinity College Dublin, Dublin, Ireland; Trinity College Dublin, Dublin, Ireland
Eilish McAuliffe, University College Dublin, Dublin, Ireland

Comments on this article

All Comments(0)

Add a comment

Sign up for content alerts

Browse by related subjects

Back to all reports

Reviewer Report

8 Views

09 Nov 2017 | for Version 1

Eilish McAuliffe, School of Nursing, Midwifery and Health Systems, University College Dublin, Dublin, Ireland

8 Views Cite this report Responses(0)

Approved With Reservations

General comment
This study sets out to evaluate the performance of Swaziland’s Rural Health Motivator Programme. As most resource-poor countries with human resources shortages in their health sectors are employing CHWs as a strategy to respond to the health needs of communities, evaluations of this strategy are highly relevant for health policy makers. Gathering data from households and CHWs has the potential to generate a much better understanding of how these programmes work or don’t work. This study is impressive in the large amount of data gathered from different stakeholder perspectives. It is a well-written paper, but with some critical omissions as set out below.

Introduction
A brief discussion on published evaluations of CHW programmes (e.g. well established programmes such as the Health Extension Workers programme in Ethiopia and the Lady Health Workers in Pakistan) would help to situate this research in the field of study and enable the authors to better articulate the study’s contribution to the field in their discussion and conclusions part of the paper.
Some description of RHMs and how they differ from CHWs would be helpful.

Methods
A rationale needs to be provided for the study design. Why were household surveys and RHM surveys conducted? Was it the authors intention to cross reference the results? What was the specific purpose of the qualitative interviews? What is the rationale for the sample size 25? No information is provided on how the interviews map to the CHW Performance Logic Model. It is not clear if the interviews were conducted after the analysis of the quantitative data and whether they were used to explore issues arising from questionnaire results in more depth or to address some other aspect of the Logic Model. The rationale for the study design and the specific purpose of each of the methods needs to be included.
Authors need to include some explanation of the CHW Performance Logic Model and why it was considered most relevant for this study.
Performance is the key outcome variable in this study, yet the authors have not defined what this term means in the context of this study.
The authors state that if no household members was available at first visit, they did not revisit. They ought to give some consideration to the potential bias this may have introduced into the study. For instance, it may be that those who were employed and out at work were less likely to be available than those who were unemployed. No information is provided about the time the visits took place, so it is difficult to know if this bias could have occurred. Some further expansion of the methods and/or consideration of this issue in the limitations of the study is needed.

Results
A strength of the study is the use of three sources of data and the opportunity this presents to triangulate data. The relatively large sample sizes should allow for a robust interrogation of the data. It is therefore disappointing that the analysis is rather superficial and confined mainly to descriptive stats. There are some obvious questions that remain unanswered because the relevant data have not been triangulated. For example, the RHMs reporting of rate of household visits could have been triangulated the household members response to Questions 2.8-2.14 to give a much better understanding of the productivity of the RHMs. Also there are many questions in the household member study that explore the performance of RHMs the results of which are not presented at all in this paper. If these results have been presented elsewhere, it would be helpful to refer to the publication.

The statement “30.5% of RHMs reported to have done work other than for the RHM program during the previous 12 months” requires further explanation. Does this mean these RHMs worked elsewhere before joining the program or that they concurrently take on other work while conducting their RHM duties?

57.8% of sample disagree that they are paid a fair amount for the work they do, yet there are several non-monetary payments. Is it possible that these compensate.

It is mentioned that half of respondents were satisfied or very satisfied with their job and then “Roughly a quarter (26.2%) of RHMs reported to occasionally or often think about leaving their job”. Was this issue explored in the qualitative interviews? Why is such a high proportion thinking about leaving their jobs?

I also have concerns about the treatment of the qualitative data. It is not clear if quotes provided are indicative of a majority view or not. Whilst it would not be appropriate to attempt to quantify qualitative data, some sense of how widely held the opinions represented in the quotes are in the sample of interviewees would be appropriate.

The following quote is provided as illustrative of physical distance being a barrier to RHMs visiting the households, but the quote is illustrative of the ease of visiting due to short distance and is therefore not appropriate to illustrate the point above
““I find it to be very easy since the households I am responsible for are nearby and I do not need to walk a long distance””

Discussion
There are several problems with the discussion section of this paper.

In the first paragraph of the discussion on social desirability bias needs to be re-written. The authors suggest that social desirability bias may have impacted the answers to two specific questions, but the results (i.e. low satisfaction with RHM and low rates of productivity reported by RHNs) suggest the opposite is more likely. This needs further discussion and explanation.
There seems to be an element of selectivity about what results are discussed. For example, taking two results presented in the results section: “The majority of RHMs expressed dissatisfaction with the compensation offered. 57.8% either disagreed (38.0%) or strongly disagreed (19.8%) with the statement that “Given the amount of work I do as a rural health motivator, I am being paid a fair amount”. and “65.3% indicated that they would like to receive more supervision. Qualitatively, in cases where RHMs expressed interest in additional supervision, the reason tended to be that they felt additional feedback would help motivate them further and support continued learning”. The first of these results is discussed and a recommendation put forward to increase the monetary reward of RHMs. The second result represents a greater majority and is clearly linked to improved motivation, yet it is not mentioned in the discussion and additional supervision is not put forward as a recommendation.
There are many apparent contradictions in the data that are not discussed e.g. low satisfaction levels with RHM services, but very high percentage of households would recommend the RHM programme to other communities
The discussion section contains statements for which no evidence is provided in the results section e.g. “The low pay is likely also an obstacle for shifting HIV care tasks to RHMs, as many of these tasks, such as ART home-delivery, require reliable and constant care.”
“Qualitative interview manuscripts are not shared publicly because they cannot be effectively de-identified given the relatively small number of staff involved in the studied community health worker programs.” This is not a convincing explanation as 25 interviewees from a sample of 306 would suggest that their anonymity could be protected. If the data cannot be provided, then the extracted text coded to the main reported themes should be made available.
In the limitations, the authors provide a comprehensive account of the weaknesses of the CHW Performance Logic Model. This begs the question as to why they decided to use such a flawed model as the framework for the evaluation, given that this information on the Model was available to them.

Conclusions.
Again some selectivity evident as in the discussion, but some important points are also made regarding the deficiencies in the RHM programme.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

10 Views

07 Nov 2017 | for Version 1

Frédérique Vallières, School of Psychology, Trinity College Dublin, Dublin, Ireland; Center for Global Health, Trinity College Dublin, Dublin, Ireland

10 Views Cite this report Responses(0)

Approved With Reservations

Thank you for the opportunity to review this interesting paper by Geldsetzer et al. The authors conducted a cross-sectional, mixed-methods study examining the performance of Swaziland’s RHM programme. The study had two underlying objectives (i) to assess the performance of the RHM programme, using the CHW Performance Logic Model, and (ii) to identify ways in which performance could be improved. Overall, this is a well-written paper with many strengths including an impressive sample of 2000 households, representative of two of Swaziland’s four administrative regions, and a further 25 semi structured interviews. However, the paper would benefit from a number of important revisions, especially pertaining to a clear definition of performance, the methodological approaches used to define and measure this, and more rigorous analytical approaches in order to adequately address the study’s objectives.

Suggested Revisions

Introduction

While evaluating Swaziland’s RHM program is a worthy endeavour, I would argue that it is unfair to state that ‘many [of the] large programs that have existing for decades have not yet been rigorously evaluated’ (p.3) as the rationale for the current study. For example, there have been a number of evaluations of Ethiopia’s CHEW programme¹ and Pakistan’s LHW programme². Many of these studies also arguably employ more rigorous evaluation approaches than the current study. It is probably sufficient to say that no evaluations of the RHM in Swaziland have taken place to date, and if RHMs are expected to contribute to achieving 90-90-90, then a better understanding of the programme’s progress is required.
More detail is required on the RHM programme in general. Are the RHMs remunerated? If so, who pays them? If not, what non-financial incentives are in place for them, if any? Who supervises them? Are they affiliated with a health centre? Are they trained? If so, for how long? Are they predominantly women? Are they recognised as part of the formal health system?
Additional background on the context of Swaziland, specifically as it relates to their lack of human resources for health, and their need to task shift/share, would also improve the introduction and offer more context for the reader.
A clearer rationale for the importance of assessing performance of CHWs is required. Performance of health workers is a difficult construct to define, and there would be large variation in the literature in terms of how performance is (i) defined and (ii) measured. The paper would benefit greatly from engaging with this literature. Specifically, the paper should engage the literature examining the relationship between job satisfaction, motivation, intention to leave, etc. and performance, in order to build justification for the inclusion of these variables within the questionnaire design.
Did the authors consider other frameworks? If so, why was the CHW Performance Logic Model chosen as the framework for this study? A stronger rationale for why this framework was chosen is required as this model does not measure performance per se, but rather puts forward a theoretical pathway towards performance. What evidence is there to suggest that this model is valid (i.e. predictive validity) in terms of predicting CHW performance? This is also methodological decision and does not belong in the Introduction.

Methods

Study Setting

Clarify why Lubombo and Manzini were chosen as the two study areas. Was this random selection? If not, how were these two chosen? Is there any reason to believe that differences might exist between these two areas? What percentage of Swaziland’s population resides within these two areas?
Who operates the other CHW programmes in Swaziland? While the authors specify that each have a workforce of less than 50CHWs, how many CHWs in Swaziland are part of other programmes (i.e. not RHMs)? If you have the data for these, why not include these and control for ‘CHW programme’ in your analysis?
No need to repeat that the RHM programme was established in 1976 and employs 5000 RHMs again here, as it is already in the introduction.

Study Design & Materials

Clearly state that this study uses a cross-sectional, mixed-methods design.
As above, a clearer operational definition of what is meant by ‘performance’ is needed. What is meant by high or good performance here? How was ‘performance’ defined and measured for the purpose of this study? Why, for example, not include indicators that are aligned to the activities outlined under the RHM program section of the introduction (page 2 para 5)? It seems only fair that performance of the RHMs should, at least in part, be measured against those activities and tasks they are expected to complete. This is a major element missing from this study.
A much more detailed description of the study tools are required. As it stands, it is difficult to evaluate the validity of the scales used without knowing the details of how the survey employed during data collection was designed. For example, have the ‘job satisfaction’, ‘social support’, ‘supervision’, and ‘motivation’ items used in the RHM questionnaire (Part 12 and Part 13, respectively) been used in other studies? If so, which ones? Have these scales been validated? According to these validation studies, how are they meant to be coded/scored?
- Some of the items under Part 13 do not seem to be measuring motivation (i.e. lack face validity). For example, items 13.8 & 13.9 seem to more so be measuring conscientiousness at work, than motivation.

Sampling

It would be quite unlikely in an entirely random sample to have the same number of rural (n=37) and urban (n=13) EAs in each region. Especially since the study setting section describes Manzini as comparatively more urban and wealthy than Lubombo.
What is the rationale for interviewing family members aged 11 years or older? Also, one would expect that those over the age of 18 would provide written consent, but those between 11-17 would provide assent, provided parental consent. Please clarify whether this was the case.
On page 4, first paragraph, please clarify the sampling for the SSIs. The authors state that 13 RHMs from Manzini and 12 from Lubombo were selected, but then go on to specify that 13 were from rural and 12 were from urban areas, in each region. Were there 50 SSIs in total?

Data Analysis

The quantitative data analysis is rather superficial (i.e. descriptive), and unfortunately, does not exploit the richness of the dataset. Why were inferential statistics not employed to look at correlations between certain factors that are known to predict performance (i.e. job satisfaction, motivation at work, supportive supervision) and performance (however this is defined here)? Regression methods, with ‘performance’ as the dependent variable, could be used to more rigorously assess variables that are associated with performance, and are aligned with addressing the second objective of your study: (ii) identify ways in which performance could be improved. The logic here being that improvements made to these factors could result in corresponding changes ‘performance’. Moreover, you would be able to control for differences across Lubombo and Manzini, CHW programes etc.

How were scores calculated? For example, on what basis was it decided that anyone who scored below 8/10 on the Likert scale should not be classified as ‘satisfied with the services provided by the RHMs in their community’? Why recode the answers into dichotomous variables, instead of using the mean score? These methodological decisions need to be justified/described in much greater detail.
- How was missing data treated?

The qualitative data analysis, given the use of a comprehensive interview guide, with broadly pre-determined themes (i.e. the dimensions of the CHW Performance Logic Model), strikes me as having been analysed using more deductive, rather than inductive approaches.
More detail is required as to how you: ‘conducted iterative reviews’ and ‘further refined themes’ and established ‘their relationship to each other’. As it stands, the qualitative description of the analysis is insufficient to ensure replicability.

Results

The results in part read like a report against a logframe or results-based framework than a research paper. As above, the lack of clarity around how dichotomous categories were recoded from the Likert scale makes the results difficult to interpret.
In line with the above comment on inductive vs. deductive approaches, I’m not convinced that HHs approaching RHMs was ‘rare or infrequent’ (pg 6, para 2) is an emerging theme or topic, when the question asked was “How often are you contacted for help or advice?”.
Figure 2 is not very telling. Instead of presenting as a histogram, consider presenting as mean scores for ‘satisfaction’ across the various items in the HH questionnaire, with the Likert Scale range of scores presented on the y-axis.
“Inputs” section is odd here and should form part of the narrative describing the RHM programme in the introduction. RHM salaries in Table 5 should also be reported in USD to give the reader some idea of what this figure represents.
Overall, I’m not convinced that the results presented are sufficient to answer the research objectives (i) to assess the performance of the RHM programme and (ii) to identify ways in which performance could be improved. The results could however, shed some light on how the RHM programme is faring in terms of known determinants of performance.

Discussion

While every programme has room for improvement, I would argue that the results presented are not only suggestive of ‘a number of weaknesses in the RHM’ programme. Sure, resource mobilization and poor pay are an issue (as they are in most places!), but overall, the evidence also suggests a programme that has maintained high community support, a high level of quality training, frequent and supportive levels of supervision from the health facilities, and relatively low levels of people who were thinking of leaving their job. These are all worthy accomplishments for a health programme where resources are extremely limited. Moreover, these are all factors that the literature would strongly suggest are important in predicting health worker performance. While I understand that this type of narrative is what keeps foreign aid funding flowing into programmes, our responsibility as researchers is to present the evidence as objectively as possible. Here, the evidence provided is quite strong for many successful elements of the RHM programme, and these should be discussed too.
Page 10 para 1: “overall, improving the performance of the RHM cadre may be necessary to successfully shift HIV care tasks from facility-based to RHM-led care” seems like an over-interpretation of findings. The evidence presented more so suggests that adequate compensation and more material resources are likely necessary before asking RHMs to take on yet another task within the health system.
The evidence presented offers some insight into why RHMs may not be meeting their HH targets (having to return to homes more than once if a family is absent, attitudes towards HRMs). This should be discussed accordingly as the reason for this observation does not appear to lie solely with the RHM.
The breakdown of income earning on page 10 is interesting, but would be better suited to a description of the RHM programme in the literature review.
Page 10, para 2: “Our assessment of the RHM program…provides some insight into factors that might be lowering RHM performance”. There is no evidence presented to suggest that ‘performance’ decreased in any way. Please rephrase.
Page 10 para 2: “a theme arising from our qualitative interviews, however, was that RHMs view themselves as volunteers rather than…”. I did not see this evidence presented in the results of the paper. Generally speaking, new results should not be presented in the discussion.
Limitations: I disagree that many of the performance-related factors assessed under the CHW Performance Logic Model lack established measures and scales. The I/O psychology literature for example, contains a number of well-developed, cross-culturally validated scales of social support, motivation at work, job satisfaction, etc. scales.

Conclusion

Regarding the statement that, “the evaluation found that the RHM program does not meet some of its performance targets”: Are RHMs really intended to act as the first point of care in this context? Other than not visiting with all 25-30 households on a monthly basis, what other performance target(s) does this statement refer to?

Other (including Figures & Tables)

2^nd para, page 4, there is no need to explain in text what the contributions of each author were. There should be an ‘Author Contributions’ section which allows for this.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

No
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Karim AM, Admassu K, Schellenberg J, Alemu H, et al.: Effect of ethiopia's health extension program on maternal and newborn health care practices in 101 rural districts: a dose-response study.PLoS One. 2013; 8 (6): e65160 PubMed Abstract | Publisher Full Text
2. Douthwaite M, Ward P: Increasing contraceptive use in rural Pakistan: an evaluation of the Lady Health Worker Programme.Health Policy Plan. 2005; 20 (2): 117-23 PubMed Abstract | Publisher Full Text

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Applications of Psychology to Global Health. Namely, Oganisational Psychology, Global Mental Health, Psychotraumatology, and Latent Variable Modelling,

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Respond to this report

Responses (0)

[1] 1. World Health Organization: World Health Statistics 2016. Geneva: World Health Organization, 2016. Reference Source

[2] 2. Singh P, Sachs JD: 1 million community health workers in sub-Saharan Africa by 2015. Lancet. 2013; 382(9889): 363–5. PubMed Abstract | Publisher Full Text

[3] 3. Hongoro C, McPake B: How to bridge the gap in human resources for health. Lancet. 2004; 364(9443): 1451–6. PubMed Abstract | Publisher Full Text

[4] 4. Mwai GW, Mburu G, Torpey K, et al.: Role and outcomes of community health workers in HIV care in sub-Saharan Africa: a systematic review. J Int AIDS Soc. 2013; 16(1): 18586. PubMed Abstract | Publisher Full Text | Free Full Text

[5] 5. Viswanathan M, Kraschnewski J, Nishikawa B, et al.: Outcomes of community health worker interventions. Evid Rep Technol Assess (Full Rep). Rockville, MD: RTI International–University of North Carolina Evidence-based Practice Center. 2009; 181: 1–144; A1-2, B1-14, passim. PubMed Abstract | Free Full Text

[6] 6. Gilmore B, McAuliffe E: Effectiveness of community health workers delivering preventive interventions for maternal and child health in low- and middle-income countries: a systematic review. BMC Public Health. 2013; 13: 847. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Lewin S, Munabi-Babigumira S, Glenton C, et al.: Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases. Cochrane Database Syst Rev. 2010; (3): Cd004015. PubMed Abstract | Publisher Full Text

[8] 8. Perry HB, Zulliger R, Rogers MM: Community health workers in low-, middle-, and high-income countries: an overview of their history, recent evolution, and current effectiveness. Annu Rev Public Health. 2014; 35: 399–421. PubMed Abstract | Publisher Full Text

[9] 9. East Central and Southern African Health Community: Task shifting in Swaziland: A case study. Washington, DC: Futures Group, Health Policy Initiative, Task Order 1, 2010. Reference Source

[10] 10. GBD 2013 DALYs and HALE Collaborators, Murray CJ, Barber RM, et al.: Global, regional, and national disability-adjusted life years (DALYs) for 306 diseases and injuries and healthy life expectancy (HALE) for 188 countries, 1990-2013: quantifying the epidemiological transition. Lancet. 2015; 386(10009): 2145–91. PubMed Abstract | Publisher Full Text | Free Full Text

[11] 11. UNAIDS: 90-90-90 An ambitious treatment target to help end the AIDS epidemic. Geneva: UNAIDS, 2014. Reference Source

[12] 12. UNAIDS, One Million Community Health Workers: UNAIDS joins forces with the One Million Community Health Workers campaign to achieve the 90–90–90 treatment target. 2016, (accessed 24 March 2016). Reference Source

[13] 13. World Health Organization, PEPFAR, UNAIDS: Task shifting - Global recommendations and guidelines. Geneva: World Health Organization, 2008. Reference Source

[14] 14. Jaffar S, Amuron B, Foster S, et al.: Rates of virological failure in patients treated in a home-based versus a facility-based HIV-care model in Jinja, southeast Uganda: a cluster-randomised equivalence trial. Lancet. 2009; 374(9707): 2080–9. PubMed Abstract | Publisher Full Text | Free Full Text

[15] 15. Selke HM, Kimaiyo S, Sidle JE, et al.: Task-shifting of antiretroviral delivery from health care workers to persons living with HIV/AIDS: clinical outcomes of a community-based program in Kenya. J Acquir Immune Defic Syndr. 2010; 55(4): 483–90. PubMed Abstract | Publisher Full Text

[16] 16. Geldsetzer P, Francis JM, Ulenga N, et al.: The impact of community health worker-led home delivery of antiretroviral therapy on virological suppression: A non-inferiority cluster-randomized health systems trial in Dar es Salaam, Tanzania. BMC Health Serv Res. 2017; 17(1): 160, In press. PubMed Abstract | Publisher Full Text | Free Full Text

[17] 17. ICAP-Swaziland: RHM Review 2012. Mbabane: Columbia University, 2012.

[18] 18. Naimoli JF, Frymus DE, Wuliji T, et al.: A Community Health Worker "logic model": towards a theory of enhanced performance in low- and middle-income countries. Hum Resour Health. 2014; 12: 56. PubMed Abstract | Publisher Full Text | Free Full Text

[19] 19. Central Statistical Office: 2007 Population and Housing Census. Mbabane, Swaziland: The Kingdom of Swaziland, UNFPA, 2010; 6. Reference Source

[20] 20. Central Statistical Office, Macro International Inc: Swaziland - Demographic and Health Survey 2006-2007. 2007. Reference Source

[21] 21. Ministry of Health: Swaziland HIV incidence measurement survey (SHIMS). Mbabane: Kingdom of Swaziland, 2012. Reference Source

[22] 22. Bicego GT, Nkambule R, Peterson I, et al.: Recent patterns in population-based HIV prevalence in Swaziland. PLoS One. 2013; 8(10): e77101. PubMed Abstract | Publisher Full Text | Free Full Text

[23] 23. Sandelowski M: Whatever happened to qualitative description? Res Nurs Health. 2000; 23(4): 334–40. PubMed Abstract | Publisher Full Text

[24] 24. The Kingdom of Swaziland: Budget versus Expenditure Report. Mbabane, Swaziland, 2012.

[25] 25. Glick P: How reliable are surveys of client satisfaction with healthcare services? Evidence from matched facility and household data in Madagascar. Soc Sci Med. 2009; 68(2): 368–79. PubMed Abstract | Publisher Full Text

[26] 26. Oxford Poverty and Human Development Initiative: OPHI Country Briefing Dec 2015: Swaziland. Oxford, UK: Oxford Department of International Development, University of Oxford, 2015.

[27] 27. Tsai AC: Socioeconomic gradients in internalized stigma among 4,314 persons with HIV in sub-Saharan Africa. AIDS Behav. 2015; 19(2): 270–82. PubMed Abstract | Publisher Full Text | Free Full Text

[28] 28. Lema IA, Sando D, Magesa L, et al.: Community health workers to improve antenatal care and PMTCT uptake in Dar es Salaam, Tanzania: a quantitative performance evaluation. J Acquir Immune Defic Syndr. 2014; 67(Suppl 4): S195–201. PubMed Abstract | Publisher Full Text | Free Full Text

[29] 29. United Nations Statistics Division. Purchasing power parities (PPP) conversion factor, local currency unit to international dollar. (accessed September 8 2016) 2016. Reference Source

[30] 30. Geldsetzer P, Vaikath M, de Neve JW, et al.: Dataset 1 in: A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program. F1000Research. 2017. Data Source

[31] 31. Geldsetzer P, Vaikath M, de Neve JW, et al.: Dataset 2 in: A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program. F1000Research. 2017. Data Source

A qualitative and quantitative performance evaluation of Swaziland’s Rural Health Motivator program

Abstract

Keywords

Introduction

Methods

Study setting

Community Health Worker programs in Swaziland

The RHM program

Quantitative methodology

Qualitative methodology

Evaluation framework

Figure 1. The Community Healthcare Worker Performance Logic Model.

Ethics

Results

Sample characteristics

Table 1. Socio-demographic characteristics of household and RHM survey respondents.

CHW performance

Table 2. Outcomes and outputs.

Figure 2. Histograms of satisfaction with the rural health motivator (RHM) program among household survey respondents1,2.

Program-level activities

Table 3. Program-level activities.

System-level activities

Table 4. System-level activities.

Inputs

Table 5. Cost of the RHM program in 2011.

Discussion

Conclusions

Data availability

Author contributions

Competing interests

Grant information

Supplementary material

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

The problem

How to fix it

The problem

How to fix it

Competing Interests Policy

Stay Updated

Figure 2. Histograms of satisfaction with the rural health motivator (RHM) program among household survey respondents^1,2.