Keywords
Ebola, electronic medical records, EMR, innovation, humanitarian
This article is included in the Emerging Diseases and Outbreaks gateway.
This article is included in the Médecins Sans Frontières gateway.
This article is included in the Médecins Sans Frontières: Research on F1000 collection.
Ebola, electronic medical records, EMR, innovation, humanitarian
This article has been updated in response to reviewers' comments as follows:
- Clarifications have been added as to how data was transferred out of the high risk zone
- Kappa coefficients have been used to show the degree of agreement between electronic and paper data collection
- Further cleaning of the raw data to put it in an appropriate format for calculating the kappa statistic revealed a minor error in record allocation. This correction has been made in the text, with ‘85 record pairs for 32 patients with 26 data items’ being changed to ’83 record pairs for 33 patients with 22 data items’
See the authors' detailed response to the review by James Whitworth
See the authors' detailed response to the review by Benjamin O. Black
By November 2015, the West Africa Ebola epidemic had caused 28598 infections and 11299 deaths in the three countries most affected1. In response, Médecins sans Frontières (MSF), drawing on over 20 years of experience in responding to Ebola outbreaks, had treated more than 7500 suspected Ebola cases, including more than 4700 confirmed cases. The outbreak, unprecedented in size, required rapid innovation and adaptation. To cope with the number of cases, MSF scaled up its usual 20–30 bed Ebola management centres (EMCs) to 100–300 beds with over 300 workers in some settings. This increased scale brought challenges in patient and clinical data management resulting from the difficulties of working safely with high numbers of Ebola patients (Panel 1). Little had been published on managing clinical documentation and data transfer from filovirus wards, and there were no established, standardized, or widely used approaches2. Here we describe how MSF developed context-adapted tools to address the challenges of recording Ebola clinical information for in-patient management. We share the key issues and lessons learned in innovating rapidly under pressure in difficult environmental conditions and propose a framework for emergency humanitarian innovation.
• Staff working in the high-risk zone of an EMC (the area where there is high risk of Ebola infection) must wear PPE, which is heavy, limits vision severely, adversely affects manual dexterity, and is extremely hot. Clinicians have a strictly limited time in which they can safely remain in PPE without overheating and increasing infection risk during PPE removal. A lot of time was lost while clinicians in PPE tried to find patients inside the high risk zone of the EMC. Patients were moved through tents in the EMC representing different levels of risk during their treatment and could also move around within the EMC of their own volition (patients can become confused and delirious during Ebola infection), which made locating a specific patient in a large EMC challenging. • Clinicians lacked an efficient means to document patient history and record patients’ progress through the EMC. In some EMCs, clinicians in full PPE shouted data over the fence for someone outside the high-risk zone to record on ‘clean’ paper, or data were recorded in digital photographs, since nothing could be taken out of the high-risk zone without being disinfected in chlorine. The original paper records made in the high-risk zone were incinerated. As a result, adequate data to support good clinical care were rarely available. • Data collection processes varied widely between EMCs, resulting in little data sharing between them and limiting the potential for learning and improving care. There was no centralized organizing authority for data collection forms or formats. • Despite the efforts of the Emergency Telecommunications Cluster*, most EMCs had spotty and intermittent internet connectivity at best. |
*Information on the ETC is at: http://www.etcluster.org/about-etc
In September 2014, MSF medical staff established a collaborative project with a small group of software developers and the Google Social Impact Team. The aim was to develop a tool to enable the collection, visualisation, and sharing of standardised information on Ebola patients and treatment programmes, and thereby make the most efficient use of the limited time that staff are able to remain in personal protective equipment (PPE) to treat patients in the high-risk zone of an EMC.
Informal discussions with current and returning MSF field staff enabled a first set of requirements to be drawn up. Initial scoping, conducted by Google over 2 days, showed that several data collection devices were already under development for Ebola, some of which promised to be rapidly deployable3,4. However, it was determined that none of these would sufficiently meet the requirements that had been identified. These requirements were further refined through extensive user consultation including current MSF national and international field staff, operations managers, and medical specialists with experience in Ebola, as well as public health specialists from Harvard Medical School and The London School of Hygiene and Tropical Medicine. A product specification was generated to meet these requirements (Panel 2) based on the following principles: the solution should be available quickly (within 6 weeks); it should be “just enough” to meet the requirements; it should be useable without programming knowledge; and both the product and its intellectual property (IP) should be freely or cheaply available.
To speed up development and ensure that the final tools would be available free of charge to Ministries of Health or other humanitarian organisations, the development team used pre-existing open-source platforms where possible, rather than developing from scratch. The data model and database from OpenMRS (an open-source Medical Record System; platform v 1.10.1 on an Edison server running Yocto Linux + Debian GNU/Linux 7) were combined with data entry elements from OpenDataKit (open-source mobile data collection tools), together with a bespoke user interface (made with code forked from ODK Collect v1.4.4, running on Android 4.4.2 [KitKat]), designed for a high-risk zone. Panel 3 outlines the tool that was finally developed. A full, open-source, technical specification of the product is available5.
The client-server architecture developed for this project involves a low-energy server implemented on a 36 × 25 × 4 mm Intel Edison computer, built into an enclosure full of batteries and a custom charging circuit to ensure all-hours reliability. Rather than relying on the internet for updates, backups, and maintenance, the backup and updating system relies solely on USB sticks. The client software installed on the tablets can be replaced with any other application, which can then benefit from the hardware and server capacity of the system. Servers can be deployed inside and outside the high risk zone, enabling safe and effortless transfer of data between locations.
Approximately US$1.9 million was spent on development, of which $1.8 million was provided by Google. This included a team of five full-time engineers for 6 months, as well as contractors and consultants. Approximately $500,000 was spent on custom manufacturing of hardware, often at above-market costs due to the urgency of the project. For instance, a factory in California was commissioned to run for 72 hours straight during the Christmas of 2014 to manufacture tablet enclosures. In all, the project took 7 months from concept to deployment.
In January 2015, an alpha version was field-piloted in the MSF EMC in Magburaka, Sierra Leone; feedback from the field team was incorporated and bugs were fixed. In March 2015, a full prototype was deployed in Magburaka. Inpatient data were captured on 204 clinical interactions with 34 patients from 5 March until 10 April 2015 (equating to 95% of those admitted in this relatively quiet period). In this initial deployment, for each clinician-patient interaction the routine paper record system was maintained in parallel; clinical observations and treatments were recorded on paper in the high risk zone by the clinician wearing PPE, then shouted over the fence to a colleague standing in the low-risk zone (an area of an EMC where there is low risk of Ebola transmission and staff do not wear full PPE) who transcribed them to ‘clean’ paper patient charts, and later entered into an Excel database by a data encoder.
Information on adoption, maintenance, and data quality was gathered through review of project documentation, discussions with field staff and key project stakeholders, and analysis of data from the tablets. We carried out a rapid informal mixed methods evaluation to look at adoption/acceptance by health workers, implementation and maintenance challenges, and data quality and usefulness. Observation of staff and six unstructured staff interviews were carried out by a member of the implementation team with experience of implementing and evaluating e-health innovations, and were recorded in note form. Six semi-structured interviews with key project stakeholders (Google, MSF, Harvard) were carried out using a topic guide by an MSF administrator with experience in qualitative research, who recorded and transcribed the interviews. Project documentation from September 2014 to April 2015, including situation reports, vision and requirements documents, were included as an additional data source. Thematic analysis of project documentation, observations and field interview notes, and stakeholder interview transcripts was performed by the administrator supported by a senior team member with experience in health-care evaluation.
Data from the tablets were analysed via a semi-automated match of record “pairs” (clinician-patient interactions where a record for the same interaction existed in both the tablet dataset and the paper→Excel dataset). For data items in both tablet and Excel datasets (temperature and symptoms), the simple data (raw data rather than OpenMRS codes) were extracted from the tablet dataset and multiple records for interactions within 30 minutes were manually merged. If a symptom was recorded as present in any record in a set of multiple records within 30 minutes, the symptom was marked as present. The Excel dataset was checked and corrected against scanned paper records. The aligned tablet and cleaned Excel datasets were combined into a single list, sorted by patient and date/time of interaction, such that the single daily record in Excel was paired with the first tablet record that day. This is a potential limitation, as it assumes that no observations would be recorded on the tablet prior to the principle morning clinical encounter, at which the single daily paper record was made. The record pairs were then recoded to collapse graded symptoms in the tablet record (e.g. 24 hour count of diarrhoea or vomiting episodes, extent of weakness) to only symptom present or absent, and some symptoms combined into a single symptom variable (e.g. abdominal pain was a single variable on paper but recorded separately for upper and lower on the tablet, so a combined variable was used for comparison). Temperature was recoded to <37.5°C or >=37.5°C, and also matched using precise value. Kappa coefficients were calculated using STATA v14.2 (StataCorp, College Station, Texas USA) for each symptom and for all records with an entry in both sources. Observations of vital signs were not part of the Excel dataset, but the presence of these items in the scanned paper records was documented and compared to whether these signs were also recorded in the tablet record. The quality of the scanned paper charts was subjectively assessed.
This evaluation met the criteria of the MSF Ethics Review Board for exemption from ethics review. The product was discussed with and demonstrated to health authorities prior to implementation, who were supportive of this innovation and did not suggest additional ethical scrutiny.
The time taken to deliver the product was substantially more than that anticipated by MSF (7 months, as opposed to 6 weeks). The initial specification was largely but not fully delivered; a patient localising feature (radio-frequency identification tags for patients and a network of readers) was developed but never completed, as the declining patient numbers reduced the potential value of this feature. In addition, exported data require significant work to clean and analyse outside of the application interface. The database was not configured to create a single record for a patient encounter if the user moved in and out of the record, resulting in up to five partial records within a 10 minute period which all related to a single encounter, and thus required to be merged into a single record of that encounter. Records were then exported with content and order based on OpenMRS coding, resulting in 3 data fields per data element and an order that was non-intuitive and would require further development for the product to be usable in field conditions.
Due to the hasty decommissioning of the Google team at the end of the project, there was insufficient time for a comprehensive handover (from Google to MSF) of information and understanding required to operate and maintain the software and hardware that had been developed. In the case of the software and key hardware, MSF was able to hire an ex-Google engineer who worked on this project to further develop the software and complete the process of handover to MSF. However some parts of the hardware (e.g. polycarbonate casing) were deemed too expensive to warrant further investment by MSF, so little attempt was made to obtain the knowledge necessary to produce these.
Deployment of the tablet coincided with a dramatic drop in patient numbers. As a result, the full prototype had little impact on patient care.
Adoption. The final product deployed in Magburaka was well received by medical staff, some of whom had no previous experience with computers or touch-screen devices. Staff described the system as intuitive and reliable for data entry and visualisation, even when power and internet connections were interrupted. Medical staff valued the ability to access more detailed, real-time, clinical data.
Implementation and maintenance. Tablets remained functional after frequent dipping in chlorine disinfectant, and the system remained active for periods of up to 24 hours without electricity.
Data quality and usefulness. 83 record “pairs” for 33 patients with 22 data items (temperature and symptoms) per pair were available to be analysed.
The overall Kappa coefficient for agreement between both sources was 0.62, but reduced to 0.59 when rare bleeding symptoms were excluded, indicating moderate to good agreement. For individual symptoms it the Kappa coefficient ranged from 0.29 to 0.86, with chest pain, diarrhoea and sore throat having only fair agreement, but precise temperature very good agreement. The number of pairs with data available in both sources per symptom ranged from 78–81 of the 83 record pairs. Most patients were admitted for a short period and symptoms were recorded on paper only once per day, so 30/33 (91%) had 1–3 record pairs. One patient had 19 record pairs, but if all record pairs beyond 4 were excluded, the Kappa coefficient was still 0.59.
The tablet contained more unique patient encounters than the paper records – the paper chart usually showed one set of observations per day, while the tablet was used to record additional encounters that were (at best) otherwise written only in patient progress notes (which were not standardised and thus difficult for a data encoder to enter into a standardised database format). However, it is unknown if this recording would still occur if the EMC were very busy.
The tablet contained data fields for vital signs (BP, HR, RR) that were not always recorded on the paper vital signs charts, but rather only on the free-text clinician progress notes (which had not been entered into the Excel database at the time of matching). At least one of these vital signs was entered in the tablet for about 40% of 204 interactions, usually matching the paper record, and sometimes providing data that did not exist in the paper record. In a few instances (11/85 interaction pairs), one or more vital signs in the paper record were not in the tablet dataset.
The paper charts were sometimes hard to read, leading to errors in the Excel database such as a temperature recorded in the paper record that could be read as either 39.2 or 34.2 (the tablet record showed 34.2). In addition, there was variable use in the symptoms chart of symbols such as yes, y, no, n, ✓, ✘, -, |, ?. About 20% of paper-recorded encounters had some ambiguity, including no or only partial zero reporting (that is, specifying that symptoms were absent, rather than an assumption that any symptom not marked as present is absent as opposed to not assessed).
Interviews with field staff and key stakeholders revealed these common themes around challenges faced in the project.
Lack of agreed vision from start: The team’s haste to deploy something quickly that was “just enough” meant that insufficient time was taken to establish internal (MSF) project vision. Instead, the project team adopted a more perfection-driven Google vision (which extended the timeframe), and the user (MSF) questioned only late why something so complex had been made that was potentially useful in the long term, but not in this outbreak.
No explicit approach to project management: MSF project management structure, including governance and an organogram, was defined only when the project started to falter. As a result, the software developers were never sure who was the MSF lead, or who to go to when things went wrong, so at times asked the wrong question of the wrong person and lost focus on the minimum viable product requirements. At a late stage, efforts were made to develop an evaluation plan, but the numbers of cases dropped very substantially soon after.
Business pressures: the pressure to start meant that reflection on team composition happened too late. For example, the team approached numerous medical people with different backgrounds, experiences, and expectations for user input. Since the epidemic evolved rapidly, this focus on “the last medic returning” as the optimal source of information meant that the user stories became rapidly out of date. A consistent medical focal person was appointed to the team to provide ongoing guidance, but this was done too late. Likewise, Google’s need to move its developers onto other projects meant that there was insufficient time for effective knowledge transfer to a technologically-naïve partner.
Lack of understanding between the humanitarian and technology fields: MSF expected to be able to describe what was needed quickly and then leave the software developers to deliver, without realising the time commitment required from them to enable the developers to make the right product. In parallel, the developers expected that health programmes could be measured in terms of number of patients accessed, and struggled to incorporate a health outcome perspective in their objectives and notion of impact.
The tablet facilitated more frequent and slightly more detailed data than the fence→ paper→Excel database routine, and there was high consistency between the results from the two systems. Given that the tablet data record is essentially real-time (in terms of data entry and opportunity for correction), it is likely to be the more accurate record. This assumption requires validation, but there is anecdotal evidence of the errors and uncertainties of the non-electronic data system from several EMCs.
To our knowledge, this is the first time a portable real-time electronic medical record (EMR) has been developed that specifically meets the needs of an extreme biohazard environment in a resource-limited setting with erratic power and internet supply. A client-server architecture normally necessitates a reliable, high-bandwidth internet connection, or a local server with very reliable electrical energy, IT support, and the availability of specialized parts. These requirements make a client-server architecture difficult to implement in rural sub-Saharan Africa.
MSF has gained experience and understanding of the process of technological innovation, including the limits and challenges of deploying new technology and working with technology partners6. The positive collaboration between MSF and Google has sparked interest in the potential for humanitarian-technology collaborations7. Successful deployment of reliable client-server architecture in a rural sub-Saharan African environment represents a proof of concept, such that it is already being deployed by other agencies (The Wellbody Alliance, in collaboration with Harvard Medical School).
However, there were major challenges with the project. Due to the length of time to deliver the product, in the context of an evolving Ebola outbreak, a product was delivered too late to have an impact on patient care or efficiency of EMC management. Lack of knowledge transfer meant that MSF had a product that they did not fully know how to support or adapt for future use.
The challenges we describe are echoed throughout the literature on humanitarian and technological innovation, such as best practice principles for humanitarian innovation defined by ALNAP and lessons learnt from IT innovation as outlined in the Chaos Manifesto8,9. Our experience has led us to identify some concrete lessons specific to humanitarian-technology collaborative projects. Most importantly, time and effort is required to bridge differences in organisational culture between the technology and humanitarian worlds. This investment is essential for establishing a shared vision on deliverables, urgency, and ownership of product. To guide this process, there is a need for a framework for innovation in humanitarian emergencies. The overwhelming priority for medical teams is immediate delivery of care, and innovation projects must fit around this reality; careful consideration must be given to whether there are sufficient resources to deliver the project. Therefore, emergency innovation projects need to be agile, iterative, and free from heavy project management processes. Yet if we are serious about innovation, we need to push the boundaries to ensure it has impact not only on our current patients, but also for future patients. We have outlined the key components of such a framework in Panel 4. This framework should be used as a flexible and practical tool; we caution against adapting systems that could lead to stifling of innovation, especially in complex and challenging environments where fast solutions are needed.
An MSF team is in the process of adapting the software code-base of the Ebola data management solution to develop a generic emergency EMR that can be rapidly modified for different types of emergency. Our vision is to build a modifiable Open-MRS backbone for which new disease-specific apps can be developed within 5 days by a non-coder. The first ‘test-case’ is nutrition, for which we are working on apps for inpatient and ambulatory therapeutic feeding centres. In this project, having learnt from our experience with the Ebola EMR, we have taken the time to apply our framework (Panel 4) from the start.
A complete kit of open-source software, hardware, and documentation for the Ebola EMR is available on-line and can be downloaded for free; the code is also freely available for developers to use5. Several of the developers involved have formed an open-source community that is supporting this code, and can help develop the software for new-use cases10. Harvard Medical School, MSF, and Google are all represented in this consortium. Harvard Medical School have carried out successful deployments of the hardware, which they have combined with new software for community surveillance, triage, and inpatient care. The consortium is focused upon new use-cases, the economics of the solution and driving economies of scale, the incorporation of additional software and hardware toolsets, and the enabling of clinical research during outbreak settings.
The fundamental innovation of our project in the end, was not the new technology that was developed, but rather the adaptation of existing technology so that it would work in an environment that is generally hostile for complex systems. We hope that the lessons learned and the tools developed in this collaboration will help others involved in innovation in humanitarian crises to gain the balance between speed, impact, and sustainability.
Data are deposited in a secure server in MSF Operational Centre Amsterdam and are available via a managed access process under the terms of the MSF Data Sharing Policy11. For contact details and explanation of process please visit http://fieldresearch.msf.org/msf/handle/10144/306501 or email data.sharing@msf.org.
The innovation project was led by IG, with support and input from all authors. KJ and JG wrote the article, with contributions from all authors.
The authors would like to acknowledge Jesse Berns (MSF Switzerland) and Karl Blanchet (London School of Hygiene and Tropical Medicine) for their valuable input into the initial development of the Ebola Clinical Information System. We thank Sarah Venis (MSF UK) for editing assistance; Isobel Aiken for key informant interviews; and Philipp du Cros for project oversight and input into the article.
Views | Downloads | |
---|---|---|
F1000Research | - | - |
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
References
1. Viera AJ, Garrett JM: Understanding interobserver agreement: the kappa statistic.Fam Med. 2005; 37 (5): 360-3 PubMed AbstractCompeting Interests: A colleague from LSHTM is acknowledged for his input into the development of this information system. We have had no direct interaction about this project. HB worked for MSF in Sierra Leone in the same period as the trial but had no involvement in it. She has worked with some of the authors in other locations and contexts.
References
1. Bower H, Whitworth J: Referee Report For:Electronic medical records in humanitarian emergencies – the development of an Ebola clinical information and patient management system [version 2; referees: 1 approved, 1 approved with reservations]. F1000Research. 2016; 5 (1477). Publisher Full TextCompeting Interests: A colleague from LSHTM is acknowledged for his input into the development of this information system. We have had no direct interaction about this project. HB worked for MSF in Sierra Leone in the same period as the trial but had no involvement in it. She has worked with some of the authors in other locations and contexts.
Competing Interests: BOB worked for Médecins Sans Frontières during and after the Ebola outbreak 2014-2016, including in Magburaka, Sierra Leone. However he was not directly involved in the design, implementation or use of the Electronic Medical Record, and was not present in Sierra Leone during the time of the trial. He has worked with some of the authors in other projects and contexts.
Competing Interests: A colleague from LSHTM is acknowledged for his input into the development of this information system. We have had no direct interaction about this project. HB worked for MSF in Sierra Leone in the same period as the trial but had no involvement in it. She has worked with some of the authors in other locations and contexts.
Alongside their report, reviewers assign a status to the article:
Invited Reviewers | ||
---|---|---|
1 | 2 | |
Version 3 (revision) 23 Feb 17 |
read | |
Version 2 (revision) 30 Sep 16 |
read | |
Version 1 23 Jun 16 |
read | read |
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Already registered? Sign in
The email address should be the one you originally registered with F1000.
You registered with F1000 via Google, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Google account password, please click here.
You registered with F1000 via Facebook, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Facebook account password, please click here.
If your email address is registered with us, we will email you instructions to reset your password.
If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance.
Comments on this article Comments (0)