Evaluation of the rapid, multi-country, parallel process, multi-tasking approach to startup of short-term technical assistance to improve service delivery in newborn and child health in the context of USAID’s Zika response in four Eastern and Southern Caribbean countries

Background

The rapid spread of the Zika virus (ZIKV) infection to countries in the Latin American and Caribbean (LAC) region in 2015 and 2016 and its association with serious health consequences such as microcephaly prompted the World Health Organization (WHO) to declare the ZIKV infection a public health emergency of international concern (PHEIC) on February 1, 2016¹. Although the WHO declared an end to the PHEIC in November of the same year, the ongoing Zika transmission in the region as well as the potential for future outbreaks required continuous vigilance to improve health systems preparedness and capacity in prevention, surveillance, and management of emerging infectious diseases^2–4.

In 2016, the United States Agency for International Development (USAID), the international community, and country governments began responding to the ZIKV epidemic in the LAC region⁵. In 2017, the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project implemented by University Research Co., LLC (URC) in Chevy Chase, MD, started activities in the Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Jamaica, Nicaragua, and Paraguay to strengthen Zika-related health services to deliver evidence-based, person-centered quality care with a focus on pregnant women, newborns, and women of reproductive age⁶. In May 2018, USAID requested that ASSIST expand its efforts to four more countries in the Eastern and Southern Caribbean (ESC): Antigua and Barbuda, Dominica, St. Kitts and Nevis, and St. Vincent and the Grenadines. The request came in the last year of a three-year USAID Zika response plan in the LAC region. During this time, there was also a substantial decline in the number of Zika cases observed in the region. Therefore, the decision was made to design the effort as a one-year short-term technical assistance (STTA) activity aimed at improving detection and addressing the unique health needs of children potentially affected by Zika in the four countries. The STTA planned to achieve these objectives by strengthening newborn and well-baby care systems and early childhood development (ECD) programs in all functional health facilities providing childbirth services and well-baby care. These efforts were in line with national and USAID goals to improve public health emergency response and to strengthen the resilience of health systems to address future emergencies.

Rapid Multi-Country, Parallel Process, Multi-Tasking Approach for Project Startup

Given the short timeline and overarching objectives, ASSIST developed and applied the Rapid, Multi-country, Parallel Process, Multi-tasking Approach to Project Startup (RMPP-MAPS) in the four countries. The RMPP-MAPS required simultaneous initiation and execution of multiple planning processes; including but not limited to, achieving buy-in from Ministries of Health and participating facilities, co-developing the workplan and sustainability strategy, developing technical content and operational approach, identifying essential needs of facility teams to implement the activity, and initiating the procurement processes to address the logistical needs of the activity.

Basis and rationale for the RMPP-MAPS

The fundamental concept underlying ASSIST’s health systems improvement approach is understanding that “every system is perfectly designed to achieve exactly the results it achieves⁷.” ASSIST views health systems as complex yet adaptive systems designed to undergo continuous reiterative processes of change and innovation. That process forms the basis of a six-component “Country Integrated Design” model (1-Improvement Design; 2-Implementation; 3-Sustainability; 4-Scale-up; 5-Institutionalization, and 6- Learning) that has been proven to effect positive outcomes over the three decades of USAID work (through the ASSIST and preceding projects) in the field of health system strengthening and quality improvement in health care services delivery⁸. To achieve desired outcomes, all six components of the model must be integrated in planning improvement interventions. Thus, the process necessitates the engagement of all stakeholders throughout implementation. Depending on the scale of the problem and/or context the process can take up to several years. A typical model of ASSIST and other global health partners includes setting up, staffing country offices, and a step-by-step implementation approach. For the ESC countries, ASSIST’s task was to implement the improvement activity in multiple countries within a single year. Registering URC field offices within the countries and establishing project teams to locally support the operation was not feasible in the given timeframe. This constraint posed a significant challenge for logistics and implementation. To overcome the challenge, ASSIST undertook an innovative RMPP-MAPS which allowed simultaneous planning and execution of multiple startup processes in all four countries (instead of in linear, sequential fashion) and using real-time data or “feedback loops” to track progress.

Capturing any contextual similarities and differences early in the project is critical for the success of RMPP-MAPS. For the latter purpose, the team administered scoping surveys to national and regional stakeholders in advance, followed by initial country visits. Given the similar socio-economic, health care infrastructure or development levels, the common language, and geographic proximity, the RMPP-MAPS approach was applied universally across all four countries. The main differences, as relevant to the startup, were in the availability of local human resources (e.g., shortage of nurses) and qualified vendors for outsourcing operational and logistical tasks. In addition, the IT infrastructure readiness level, such as availability of computers, internet connectivity, or even availability of electricity in health facilities, varied within and between islands. The infrastructure-related challenges were due to differing severity of destruction caused by devastating hurricanes that passed through the region the prior year. For example, Dominica’s health infrastructure suffered more severe damages than the infrastructure in the other three countries.

Despite the above differences, the RMPP-MAPS was able to integrate real-time feedback, joint brainstorming, and adaptive implementation strategies to address unique contextual challenges and ensure simultaneous implementation of startup of activities and timely initiation of facility-level QI activities.

In documenting and evaluating this novel approach, URC has published observations and recommendations for similar rapid startup of donor-funded short-term technical assistance activities. The scope of this paper is to report on the findings of the evaluation of the startup of STTA, its activities, implementation, facilitating and constraining factors, and to capture lessons learned.

Results of the start-up phase

The start-up activities in all four countries were conducted simultaneously except for the country visits, which were conducted sequentially by the start-up team visiting one country after the other. All four countries committed to taking on the work and to allocating the human resources needed for it. In the start-up discussions, differences between the resource availability in different countries became clear. For example, some countries had shortages in staff required to act as coaches, while others did not. These were factored into the workplan through arrangements with the Caribbean Regional Nurses Association (CRNA). By the end of the start-up phase all four countries embarked on implementing the workplan.

Evaluation design

A process evaluation design and a qualitative study methodology were employed to address the evaluation objectives⁹. The evaluation was retrospective with primary data collected through in-depth interviews with persons or key informants (KI) involved in the startup and secondary data retrieved from the review of project documents. Documents reviewed included: (1) startup documents obtained from the ASSIST Project (e.g., meeting records, activity plans, timelines, presentations, trip reports, scoping survey, etc.), (2) online public resources (e.g., USAID Zika program overviews, press-releases, ESC country-specific information, literature related to Zika outbreak in the region, summary of relevant work by implementing partners, etc.), and (3) documents obtained from program managers (e.g., Maternal and Child Survival Program reports and documents). Primary and secondary data were triangulated as necessary to document and develop a cohesive picture of the activity. See Extended Data¹⁰ for the documents reviewed.

Identification and engagement of key informants

KIs were defined as individuals who had direct or indirect knowledge, experience, and/or involvement with any stage of the project development or startup activities. The USAID ASSIST Project provided a list of recommended KIs to the independent research consultant (B.A.) who conducted the interviews. The list included technical partners, country program planners, project coordinators, chairs and members of community and hospital nursing divisions, and maternal and child health committees. An initial email was sent to all prospective participants by the USAID ASSIST Project to introduce the consultant and inform the eligible KI about the evaluation. The consultant then contacted each KI via email, provided details about the evaluation purpose and rationale, and invited him or her to participate in an interview via Zoom (or in some cases, over the telephone). When necessary, the USAID ASSIST Project team facilitated contact with stakeholders to confirm their participation as a KI.

In-depth individual interviews with key informants

The case study approach and evaluation or learning framework guided the development of key informant interview (KII) tools. Based on initial discussions with ASSIST program officers, the evaluation focus, objectives, main topics of interest were defined followed by identification of all stakeholders and key informants involved at each step of the startup. The interview guide was then structured to have sections around identified topics with questions and probes to facilitate detailed responses as necessary to answer evaluation questions of interest.

A semi-structured interview guide (Extended data¹⁰) consisted of sections on the focus and objectives of the evaluation, timeline, and activities that took place during the defined startup timeframe. Probe questions focused on identifying and describing startup activities in chronological order, elucidating respondent’s views or understanding of the approach and methods employed, perceptions related to timeline or speed of the process, discussion of facilitating and hindering factors, capturing lessons learned and future recommendations. Project documentation was used to more precisely construct activities and identify additional topics of interest for inclusion in the interview guide. Questions were adapted according to the type of respondent (national or regional partner, implementing partner, or leading STTA provider) and to fit the extent of participants’ involvement in the startup. All interviews were conducted in English and lasted approximately 45 minutes. The interviews were conducted by an independent, external research consultant (B.A.), who was not part of USAID ASSIST Project and was not involved in a project startup or implementation. The consultant holds a Ph.D. in Epidemiology and has expertise in public health research and program evaluation..

Data analysis

All interviews were audio-recorded. Audio files were uploaded into password-protected computer files. Automated transcripts were then generated from interviews and transcribed using transcription software (https://otter.ai). Transcripts were then manually checked and edited to ensure accuracy. The accuracy of transcription software was dependent on the respondent’s English accent (e.g., native English speakers in US vs. English of respondents from Caribbean countries), technical factors such as whether the interview was taken via phone, an online communication platform, or in-person, as well as recorded sound quality, presence of background noise, etc., and varied between 80 and 90 percent. Transcripts were not sent back to interviewees.

Thematic or pattern-based coding was applied to the data manually. Data were systematically grouped according to recurrent patterns and themes as related to central topics being evaluated (by B.A.), and codes were applied to identify segments of the interviews where these themes were discussed. The information obtained from the desk review was triangulated with the data retrieved from in-depth interviews to validate the information collected and more precisely map startup activities.

Ethical statement

Informed consent was obtained from each KI prior to the interview via email (consent form available as Extended data¹⁰). In addition, consent was obtained orally prior to recording each interview. Participants were informed of the voluntary nature of participation and the right to withdraw at any time or decline questions they did not wish to answer. Participants were also informed that their responses would be confidential (i.e., specific quotes or answers not linked to any specific respondent in the report). Interview audio, voice, and transcript records were stored securely in password-protected electronic files.

This evaluation was performed as part of the learning component of the USAID ASSIST Project and was not considered a research study subject to IRB review.

Limitations

Recall bias is a potential study limitation as interviews took place four to six months after the startup in the middle of the implementation phase. Participants sometimes had difficulties recalling details related to activities, dates, and sequences of events, and could not always distinguish startup from implementation activities. There is also a possibility for social desirability bias in response to sensitive questions (e.g., what the participant did not like about startup). The unavailability of some respondents due to scheduling or technical challenges is also a potential limitation.

Characteristics of evaluation respondents

Invitations to participate in the startup evaluation interviews were sent out to a total of 34 individuals, out of whom 28 (82.4%) were successfully interviewed. Interviews took place between January 29, 2019 and April 7, 2019. Interviewed participants comprised of key stakeholders involved in the startup, including technical assistance providers, implementers, and partners, based in USA and the Caribbean region. The distribution of respondents included: four USAID Washington staff; four URC staff working on the USAID ASSIST Project; one Jhpiego staff member who worked on the USAID Maternal and Child Survival Program (MCSP); three staff of the American Academy of Pediatrics (AAP); one representative of the Caribbean Regional Midwives Association (CRMA); one representative of USAID/Guyana (overseeing USAID’s regional activities in ESC); as well as stakeholders from the four technical assistance recipient countries. Respondent distribution is further illustrated in Table 1.

Stakeholders’ roles

Key stakeholders and their roles and responsibilities are shown in Figure 1. The ASSIST team in collaboration with country partners played a key role in the development of the overall scope, approach, activity plan, and implementation of the STTA. Jhpiego MCSP, a collaborating partner, worked closely with the ASSIST team to coordinate approaches and activities in the region and share lessons and experiences, as well as documents and training materials for adaptation and use in STTA. AAP, the project sub-awardee, was represented by Global Health Program Officers (management team) and pediatric consultants (technical team) and was engaged to bring in the subject matter expertise, specifically, to deliver training curricula on essential care for every baby (ECEB), and developmental pediatrics within the context of Zika. CRMA, the project sub-awardee, is a regional organization with members throughout the Caribbean region and was represented by an executive team. CRMA supported the selection and recruitment of quality improvement (QI) coaches. USAID’s Zika Program guided the overall direction and scope of the STTA.

Figure 1. Map of stakeholders and communication/relationship linkages.

Notes: The central part of the chart shows the country Ministry of Health as the principal technical assistance recipient and stakeholder and the USAID ASSIST Project as the leading provider and implementing partner of USAID. Key supporting implementing partners and other national stakeholders are arranged around the above two main stakeholders. Arrows reflect the relationship and communication flows between different stakeholders. Sidebars display the key roles each stakeholder played in the startup. USAID Guyana and CRMA are in the middle between the main stakeholders as regional liaisons and partners. MCSP, USAID Maternal and Child Survival Program; AAP, American Academy of Pediatrics; CRMA, Caribbean Regional Midwives Association; MOH, Ministry of Health; PAHO, Pan American Health Organization; CMO, Chief Medical Officer; WI-HER LLC, Women Influencing Health, Education, and Rule of Law; NGO, non-governmental organization; ECD, early childhood development; ECEB, essential care for every baby; QI, quality improvement.

Startup activities and timeline

The startup consisted of multiple processes implemented in parallel in the four countries between May 11, 2018 (when the assignment was given to the USAID ASSIST Project) and September 3, 2018. In general, all activities fall under four main categories: (1) conception and initiation activities, (2) workplan and timeline development, (3) technical content development, and (4) operations. A detailed summary list of activities under each category is shown in Table 2.

The timeline of key activities is shown in Figure 2. In less than four months, the USAID ASSIST Project was able to achieve buy-in from all countries, develop a work plan with approval from the USAID and the four Ministries of Health, develop technical content and capacity building materials for the first TA visit, initiate agreements with three sub-awardees and event planners, establish facility-based improvement teams in each country, determine indicators to track improvements in service delivery, and finalize financial, organizational and logistical arrangements for both the scoping and the first technical assistance visits.

Figure 2. Timeline of key activities.

AAP, American Academy of Pediatrics; CRMA, Caribbean Regional Midwives Association; MOH, Ministry of Health; SOW, Scope of Work; IID, improvement indicator database.

Key themes and issues identified from interviews

In this section, we describe and discuss some of the common themes and issues identified from interviews as related to key processes, challenges encountered, and any enabling and hindering factors.

Achieving buy-in. Given the limited timeframe, it was critical to secure buy-in for USAID Zika expansion efforts from key stakeholders in the new ESC countries from the very beginning. Achieving buy-in involved: (1) describing the problem and providing a convincing rationale for why it needs to be addressed; (2) informing stakeholders of the availability of funds and technical expertise that can be leveraged by countries to co-develop and implement a solution; and (3) mapping out the project’s direct outputs/outcomes and extended benefits. Involving country stakeholders at each step was critical to achieving buy-in both at the decision making and implementation levels. However, there were several constraining factors when it came to engaging country stakeholders. First, ASSIST did not have a prior presence in these four countries. As explained by one respondent, there were "reservations" on the part of countries since they were not familiar with ASSIST or its work. Second, by the time USAID approached countries, the Zika epidemic and the heightened urgency around Zika was essentially over. Finally, the countries faced competing priorities such as dealing with the aftermath of devastating hurricanes that ravaged the region in 2017. In order to address the countries’ needs, the activity focused on detection and support of babies potentially affected by Zika through strengthening essential newborn and well-baby care systems and early childhood development programs.

According to USAID and ASSIST respondents, initially, there were delays in establishing regular communications with the countries. Different “levels of engagement” were noted, with some country partners being more responsive compared to others. These challenges were mitigated through ASSIST’s continuous engagement with the USAID representative from the regional office, who facilitated direct contact with country stakeholders via email and phone. ASSIST also coordinated face-to-face meetings at a regional conference in Trinidad and during country scoping visits to ensure full commitment from country partners.

From the countries’ perspectives, the demonstrated experience of the USAID ASSIST Project and the proposal laid out during the initial discussions were noted as influential in deciding to get on board with the activity. As one country senior Ministry of Health (MOH) officer described:

Additional contributing factor for accepting the USAID Zika STTA was the recognition of the need to have ongoing Zika surveillance and up-to-date national policy documents and guidelines for developmental surveillance, screening, referral, and support of children potentially affected by Zika. Country pediatricians have noted that the existing policy documents and guidelines on Zika were developed during the height of the epidemic and did not reflect the new evidence. For example, in Antigua and Barbuda, the guidelines for infants exposed in utero to ZIKV were developed in 2016 with technical assistance from Pan American Health Organization (PAHO). This was useful for clinical providers in responding to the outbreak at the time; however, the knowledge and evidence about Zika has since expanded. Additionally, terminology, including the definition of congenital syndrome associated with Zika (CSaZ), was not reflected. Therefore, the USAID Zika technical assistance was viewed by leading pediatricians in these countries as a “fantastic” opportunity to update relevant policy documents with the help of ASSIST and AAP.

Defining STTA focus, scope and approach. Interviews with USAID respondents revealed that USAID designed a response framework around the four major lines of effort, each addressing a specific factor in preventing and controlling the spread of the Zika epidemic. The four lines of effort were: (1) vector control; (2) social and behavior change, communication, and community engagement; (3) service delivery; and (4) research, development, and innovation. Depending on the stage of the epidemic and the needs of the countries, implementing partners were guided by USAID to focus their efforts on one or more particular lines of effort to strategically align response activities amongst the various partners. Since ASSIST started its efforts in ESC countries, nearly two to three years after the outbreak had peaked, the country priorities shifted towards improving service delivery around identification of cases of CSaZ and other developmental malformations among infants and children exposed to ZIKV in utero and strengthening newborn and well-baby care systems to address other similar emergencies.

Therefore, the focus at this stage of the response was to improve and strengthen national health systems to enhance national capacities to address public health emergencies. Based on responses from USAID and country participants, the following factors have played a role in shaping the focus, scope and approach of the STTA: (1) timing or the current phase of the Zika epidemic; (2) timeframe of the USAID Zika response contract; (3) financial and human resources; and (4) country-level factors (Figure 3).

Figure 3. Key determining factors for the STTA focus, scope and approach.

TA, technical assistance; CSaZ, congenital syndrome associated with Zika; STTA, short-term technical assistance; MCSP, USAID Maternal and Child Survival Program; AAP, American Academy of Pediatrics; CRMA, Caribbean Regional Midwives Association; PAHO, Pan American Health Organization; WI-HER LLC, Women Influencing Health, Education, and Rule of Law; URC, University Research Co., LLC; WPP, World Pediatric Project.

All MOH focal persons and relevant decision-makers were engaged in the process of co-developing the work plan and finalizing the technical and geographic scope of the work.

Technical partners with specific areas of expertise were contracted after the scope and the types of technical expertise that would be needed had been defined. However, some partners wished they were involved in initial discussions. According to technical partners, earlier engagement would have helped identify clear expectations, roles, and responsibilities; while giving the partners time to plan their part of work according to the country contexts. The time left after achieving country buy-ins and initial discussions with countries to define TA focus and scope was less than a month.

Perspectives on the RMPP-MAPS. The approach was based on the ASSIST’s “model for improvement”¹¹ and “adaptive management”¹² strategies. These strategies emphasize identification of gaps and testing of local solutions to problems in real time, adjusting plans throughout the process, and implementing the results of regular monitoring of improvement interventions. Among the advantages of a rapid approach, respondents noted faster implementation, early achievement of results, expedited scaling and delivery of patient-favorable outcomes into the health services, and the efficient use of resources. Disadvantages included a possible loss of specificity and the potential for missing subtle details in the design during the early stages of response. Only a few respondents raised concerns about the possibility of diminished quality of response. Respondents emphasized that a continuous feedback mechanism and real-time adaptive management and learning strategies helped to address quality or other concerns promptly. While constant feedback and inputs from all partners were actively sought and incorporated at each step, the limited time frame made it impossible to wait until all components and specific details were in place or agreed upon by all stakeholders. Rather, the strategy adopted by ASSIST was to proceed with all necessary startup activities simultaneously to avoid delays while refining and/or making adjustments along the way as needed.

Scale up and engagement of local stakeholders. The small population and geographic size of the four countries facilitated the full scale of the improvement activity from the start. Except for the few community-based clinics in Barbuda and Dominica that were damaged by hurricanes, all facilities were included in the improvement intervention. To promote early adoption and institutionalization of changes and cross-sectoral collaboration ASSIST engaged health workers at all levels of the health system and encouraged the involvement of early childhood developmental programs under the Ministry of Education, and other community-based organizations (e.g., Roving Caregivers Program in Dominica). Both scaling up and institutionalization are essential components of ASSIST’s health service improvement model. These features of the STTA were noted by many country respondents as unique when compared to other technical assistance projects they had experience within their countries. According to country representatives, often in such projects, information "stays at the top" while "people at the bottom" are left poorly informed and tasks are "thrown" at them. Many pointed out that frontline and mid-level health workers (and, in some cases, community health aides and nursing assistants) had benefited greatly from training on clinical care of Zika-affected newborns and children. The science of QI empowered health workers to improve processes within the newborn and well-baby care system and equipped them with the skills and knowledge to apply improvement principles to other priority clinical-content areas.

Ensuring the participation of all stakeholders was not without challenges. The health systems in three of the four countries (except St. Vincent and the Grenadines) were suffering from severe shortages of health care workers, particularly nurses. Many nurses and other health care workers had migrated out to other countries following recent hurricanes, exacerbating pre-existing shortages. Hence, some health facilities were concerned additional project-related responsibilities would overwhelm staff members. Due to this shortage of local health workers and heavy workload, country partners and ASSIST used retired nurse midwives who were already familiar with the health system. St. Vincent and the Grenadines was the only country that did not request external help to provide on-site coaching support to health facilities. According to a MOH officer, there was no shortage of nurses in the country. St. Vincent and the Grenadines relied on the existing nurse supervisors to take on coaching responsibilities. St. Vincent and the Grenadines did, however, experience the same challenge as the other countries when it came to "pulling the staff" for a week-long technical session while ensuring no interruption in health services.

Country teams and facilities had to shuffle various schedules and calendars around clinic days or public holidays to prevent disruption of health services and accommodate project activities.

Country team leaders noted that promoting a better understanding of the improvement activity, the benefits of adopting changes, and support from ASSIST helped to overcome the initial resistance. As one country focal point put it, they had to “sell the idea” to staff. A dedicated leadership team and local champions were crucial in creating a “positive competitive environment” to motivate increased participation and involvement. The result was increased interest and demand for participation in technical learning sessions observed from the second TA visit, with more health providers attending the workshops than initially planned.

Technical content development. Technical content development focused on two major areas: (1) the science of QI led by the ASSIST team; and ECEB training with a focus on ZIKV led by AAP. One of the challenges identified by respondents was the lack of technical experts with expertise in both ECEB and ZIKV. According to AAP, the pool of experts trained in ECEB was limited and finding available experts to work on developing the curriculum on “a very short turnaround time” presented significant challenges. Nevertheless, the AAP team was instrumental in “pulling together” their existing materials and information resources, as well as a team of consultants to develop high-level technical tools. In mitigating short timeframe challenges, AAP and other implementing partners emphasized the importance of early involvement of technical partners in the rapid startup. Additionally, providing technical consultants with as much background or context data on local health systems and related issues in advance, was highlighted as an essential best practice.

Operations. Operations included functions related to defining administrative support (documentation, reporting, contracting, etc.), financial management (budgeting, accounting, payments, etc.), streamlining communications between multiple stakeholders, and logistical needs (procurement/supply, event organization, etc.). Given the lack of field offices and URC staff within the countries, administrative and financial management was centralized from ASSIST headquarters in Chevy Chase, MD, USA with country-specific adjustments made where necessary.

Logistical activities were outsourced to external organizations. The main challenge was identifying reliable event planning or logistics services on the ground who have intimate knowledge of local markets, suppliers, distributors, or experience with procurement management and competitive purchasing processes. For example, no reputable event planning organizations meeting the minimum required qualifications were identified in St. Vincent and the Grenadines. A unique logistical challenge for St. Vincent and the Grenadines were the delays arranging transportation for the "nurses from the Grenadines to come to the mainland" (some of the Grenadines’ nine islands are small and remote). Therefore, ASSIST contracted two independent consultants to take on the role of event planners. The ASSIST team worked with MOH focal points in all four countries to help identify and provide recommendations on potential vendors and co-implement activity to help "build local ownership and capacity."

Financial management followed USAID accounting procedures and standards according to the terms of the ASSIST cooperative agreement between USAID and URC. Planning and budgeting took place at the same time as workplan, staffing, and operational needs were being defined. Difficulties were reported in prompt costing of local activities.

Figure 1 illustrates the flow of communication between all stakeholders and partners. The ASSIST team and the focal points from the country teams were the two parties in primary contact through whom all messages from other partners were communicated. This was established to ensure consistency of messages across multiple partners. According to ASSIST, the sub-awardees were not in direct communication with country stakeholders during the early start-up phase to facilitate focused conversations about buy-in or the technical scope. However, relevant updates and decisions were shared in a timely manner. Communication was open and regular, informing all relevant stakeholders, including local stakeholders and partners. There were also regular conference calls involving stakeholders from all islands. Emails to relevant stakeholders and partners were answered within 24 hours, and urgent issues were handled through telephone, Zoom calls, and WhatsApp.

The speed or pace of the startup. Participants reported that given the substantial amount of work in preparation for the first TA session in early September and very short timeframe, the startup had to be rapid and fast-paced, which had both advantages and disadvantages. The rapid startup galvanized stakeholders to brainstorm and make decisions quickly. However, the rapid speed may not have allowed enough time for detailed or thorough follow-up of some tasks (e.g., not all materials were printed out in time for the first technical session). When asked about how the pace of the startup compared to other projects, most respondents indicated that they are not aware of any other projects of the scale, depth, or the speed and pace that occurred with the USAID ASSIST Project. Both ASSIST team members and partners reported exerting a great level of effort to successfully and efficiently manage multiple tasks at the same time within a short timeframe.

Limitations

Recall bias is a potential study limitation as interviews took place four to six months after the startup in the middle of the implementation phase. Participants sometimes had difficulties recalling details related to activities, dates, and sequences of events, and could not always distinguish startup from implementation activities. There is also a possibility for social desirability bias in response to sensitive questions (e.g., what the participant did not like about startup). There is also chance for respondent or KI selection and inclusion process. However, the risk is believed to be minimal as all individuals involved at any stage of the startup were identified as potential KIs, contacted and invited to participate in the study with response rate exceeding 80 percent.

Underlying data

In order to encourage free and open sharing of opinions and perceptions, we assured stakeholders that their responses would be confidential. Since interview questions were shaped around a participant’s specific role and involvement level in the startup, their responses were correspondingly specific to their particular experiences and views. Many organizations were represented by no more than 1–2 respondents. Hence, interview files (transcripts and audio recordings) will contain potentially identifying information and thus, cannot be made openly available. Those interested in accessing this data may contact the corresponding author, at rashadmassoud@outlook.com. Requests will be considered on a case-by-case basis and must include contact information, type of data requested, and justification (e.g., proposal) for how the data will be used and for what purposes. Any information that could potentially identify respondents or organizations will be removed from the data shared.

Extended data

Open Science Framework: Zika Startup Evaluation Manuscript Extended Data, https://doi.org/10.17605/OSF.IO/B7QGS¹⁰

This project contains the following extended data:

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).