[version 1; peer review: 2 approved]
All authors contributed to study design and methodology. JDF, OR, JC, NT, DV, GD, MJ, CC and COH contributed to lay out the machine learning approaches. MJ helped setup the infrastructure for the work. PK and TP contributed expertise in ophthalmology. TB and JL contributed to project steering and information governance. GR, RR and HM contributed to methodological oversight.
There are almost two million people in the United Kingdom living with sight loss, including around 360,000 people who are registered as blind or partially sighted. Sight threatening diseases, such as glaucoma, diabetic retinopathy and age related macular degeneration have contributed to the 40% increase in outpatient attendances in the last decade but are amenable to early detection and monitoring. With early and appropriate intervention, blindness may be prevented in many cases.
Ophthalmic imaging provides a way to diagnose and objectively assess the progression of a number of pathologies including neovascular (“wet”) age-related macular degeneration (wet AMD) and diabetic retinopathy. Two methods of imaging are commonly used: digital photographs of the fundus (the ‘back’ of the eye) and Optical Coherence Tomography (OCT, a modality that uses light waves in a similar way to how ultrasound uses sound waves). Changes in population demographics and expectations and the changing pattern of chronic diseases creates a rising demand for such imaging. Meanwhile, interrogation of such images is time consuming, costly, and prone to human error. The application of novel analysis methods may provide a solution to these challenges.
This research will focus on applying novel machine learning algorithms to automatic analysis of both digital fundus photographs and OCT in Moorfields Eye Hospital NHS Foundation Trust patients.
Through analysis of the images used in ophthalmology, along with relevant clinical and demographic information, Google DeepMind Health will investigate the feasibility of automated grading of digital fundus photographs and OCT and provide novel quantitative measures for specific disease features and for monitoring the therapeutic success.
Age-related macular degeneration (AMD) is a degenerative retinal disease that can cause irreversible visual loss (
Diabetic retinopathy (DR) is the leading cause of blindness in working age populations in the developed world (Cheung, 2010). It is estimated that up to 50% of people with proliferative DR (characterised by neovascularisation) who do not receive timely treatment will become legally blind within 5 years (Shaw, 2010). Although up to 98% of severe visual loss due to DR can be prevented with early detection and treatment, once it has progressed vision loss is often permanent (Kollias, 2010). Indeed, 4,200 people in England every year are at risk of blindness caused by diabetic retinopathy and there are 1,280 new cases of blindness caused by diabetic retinopathy (
To diagnose these conditions and monitor their progression (and response to treatment) the presence and precise location of the lesions must be determined. Two imaging modalities are commonly used for this purpose: digital photographs of the fundus (the ‘back’ of the eye) and Optical Coherence Tomography (OCT, a modality that uses light waves in a similar way to how ultrasound uses sound waves) (Huang, 1991).
Clinics such as these have generated very large datasets of both digital fundus and OCT images. They are also very busy; ophthalmology clinics often have long waits and average times to urgent first treatments can be greater than two weeks in the busiest clinics.
Machine learning algorithms make use of the rich, varied datasets to find high dimensional interactions between multiple data points (
There have been significant recent advances in the field of machine learning demonstrating algorithms able to learn how to accomplish tasks without instruction (Mnih, 2015; Silver, 2016). Recent healthcare applications of such algorithms have shed light on complex genetic interactions in autism (Uddin, 2014) and monitoring of physiological observations in intensive care (Clifton, 2013).
This study aims to combine traditional statistical methodology and machine learning algorithms to achieve automatic grading and quantitative analysis of both digital fundus photograph and OCT in Moorfields Eye Hospital NHS Foundation Trust patients (London, UK). Moorfields Eye Hospital is the leading provider of eye health services in the UK and a world-class centre of excellence for ophthalmic research and education. Should the research be successful, implementation of the outcomes would improve patient access to treatment and ease pressures on time and resources in ophthalmology clinics.
Exploratory study: investigate whether computer algorithms can detect and classify pathological features on eye imaging, including fundus digital photographs and OCT.
If the exploratory study is successful:
To provide novel image analysis algorithms to identify and quantify specific pathological features in eye imaging, using validated methods and expert clinical consensus.
To provide quantitative measurements disease progression, severity and to monitor the therapeutic success over time.
This is a retrospective, non-interventional exploratory study. Analyses performed in the study will be on fully anonymised (to achieve the primary objective) and pseudonymised (to achieve the secondary objective 2.2) retinal images (including fundus images and OCT). These images will contain no patient identifiable information.
All patients attending Moorfields Eye Hospital NHS Foundation Trust sites between 01/01/2007 and 29/02/2016, and who had digital retinal imaging (including fundus digital photographs and OCT) as part of their routine clinical care, will be eligible for inclusion in this study.
Hard copy examinations (i.e. physical photograph copies prior to digital storage) will be ineligible, and will be excluded by the nature of the original service evaluation requests from the Moorfields team. Data from patients who have previously manually requested that their data should not be shared, even for research purposes in anonymised form, and have informed Moorfields Eye Hospital of this, will be ineligible and removed by Moorfields Eye Hospital staff before research begins.
Approximately 1 million examinations meet the above criteria.
Most recent machine learning algorithms benefit from large datasets on which to train (tens to hundreds of thousands of data instances (Silver, 2016)). Across all machine learning applications the predictive power (as percentage of data instances correctly classified) of the algorithm depends on the size and quality of the dataset.
The sample size is informed by the existing literature and by DeepMind’s previous work in the field of machine learning (Mnih, 2015; Silver, 2016). Today’s most powerful deep neural networks can have millions or billions of parameters, so large amounts of data are needed to automatically infer those parameters during learning. Most problems in the medical domain are highly complex as they arise as an interplay of many clinical, demographic, behavioural and environmental factors that are correlated in non-trivial ways. This is even more true for state-of-the art deep learning methodologies that are expected to give the best results (Szegedy, 2014).
For all patients meeting inclusion/exclusion criteria the following electronic health record data will be required to complete this project successfully:
Digital Fundus Photographs
Digital OCT images
In addition to image data, the anonymised dataset will contain additional information required to train an algorithm (objective 1.1):
Demographic information shown to be associated with eye disease. This is because the retina differs by features such as age, as do the likelihoods, manifestations and progression of specific disease states.
Primary and secondary diagnostic labels describing what pathology is in the image (e.g., wet AMD, diabetic retinopathy) and the associated severity (e.g., grade of retinopathy/maculopathy in diabetic retinopathy).
Treatment information describing aspects of management alongside pathology information and clinical data such as visual acuity.
Model of imaging device.
A second dataset will be pseudonymised to allow further investigation of disease progression and treatment effects. This will include the above information, and add temporal information allowing pseudonymised data to be joined over time with knowledge of the elapsed time between each:
Time elapsed between each data point and the first visit in weeks (Sunday to Sunday).
The dates of each scan will be removed so it will be impossible to identify an individual date or year of patient’s birth.
The anonymisation and pseudonymisation procedures adopted will remove any information not specified to further avoid transfer of patient identifiable information. All anonymisation and pseudonymisation will be formally checked by Moorfields Eye Hospital staff before transfer.
In order to develop the algorithm, Google DeepMind will work with the eye images, split by the known diagnoses, using machine learning and Artificial Intelligence techniques including but not limited to: supervised and semi-supervised convolutional neural networks, recurrent neural networks, unsupervised clustering, reinforcement learning (
For a selection of images additional manual labels will be produced by experts to allow investigation of the clinical benefit that can be achieved through the use of machine learning in eye imaging. Pathological and anatomical features will be annotated by trained graders at Moorfields Eye Hospital and overseen by a consultant ophthalmologist with over 10 years of experience.
Descriptive statistics will be used to describe specific pathological features extracted by the model on retinal images, both continuous and categorical (e.g. presence and size of an abnormality), and to compare them to an expert references.
In addition we will analyse the outcomes against recent Moorfields audit data on human performance to further understand the potential impacts of the model in clinical practice.
This study requires existing retrospective data only; no prospective data are needed nor will be collected from patients, hospitals or healthcare workers. No direct patient contact will occur and necessary data will be anonymised or pseudonymised from this source dataset.
Anonymisation and pseudonymisation of all image files and clinical information is performed and validated by Moorfields Eye Hospital staff at Moorfields Eye Hospital. No patient identifiable data will be transferred to Google DeepMind.
During validation of the anonymisation procedure it was noted that the current anonymisation tool (
Google DeepMind Health has developed and established a state-of-the-art secure patient information handling service utilising Common Criteria EAL4 compliant firewalls and on-disk encryption (using Advanced Encryption Standard with a 256-bit key) of all research data, all housed within an ISO 27001 compliant data centre. After anonymisation data will be transferred to our London, UK data centre. This data handling facility conforms to NHS HSCIC Information Governance Statement of Compliance Toolkit (formally assessed at level 3).
Access will be granted by the custodian of data and no other members of the team. Only those working directly on the data in a research capacity will have access.
The data sharing agreement between Google DeepMind and Moorfields Eye Hospital NHS Foundation Trust lasts for 5 years. After this period the agreement will be reviewed should future work seek to build on this project. After the data sharing agreement expires all data used in the study will be destroyed. No modification will be made based on the data after destruction.
Data destruction will involve the deletion of the encryption/decryption keys for all project volumes, and 8-way random data write to all physical disks within the Google DeepMind Health data infrastructure. A certificate of destruction will be provided to the Trust.
The algorithms developed during the study will not be destroyed. Google DeepMind Health knows of no way to recreate the patient images transferred from the algorithms developed. No patient identifiable data will be included in the algorithms.
The research on anonymised data received formal approval from the Moorfields Eye Hospital research and development office, responsible for approving studies working with anonymised data, on 16 Oct 2015 (reference 15/050).
The research on the pseudonymised dataset received formal Research Ethics Committee approval on 03 Jun 2016 (reference 16/EE/0253). Health Research Authority approval for this is in progress and no research on pseudonymised data will begin until this is approved.
No patient will be approached directly and this work will include no direct patient contact. Only anonymised or pseudonymised retrospective data collected as part of routine clinical care are included. No patient identifiable data will be collected as part of this study. In such cases the ICO code of practice states that explicit consent is not generally required (
The project is non-interventional and does not involve any direct patient contact. We do not anticipate any change to patient management. For pseudonymised data should expert assessment during model validation highlight a clinical error this will be raised to the appropriate clinical team. The primary point of call for adverse events should they arise will be the Trust information governance lead, and the responsible clinical team at Moorfields Eye Hospital will be notified.
The study will be monitored both internally and externally. Internally Google DeepMind managers (TB, JL) will oversee and monitor progress on a day-to-day basis, ensuring the protocol is adhered to and no compliance issues arise.
Clinical and methodological experts (PK, GR, RR) are working with Google DeepMind to further oversee the ethical, clinical and methodological considerations of the project and will advise on at least a weekly basis to ensure no deviation from the described protocol.
Externally the information governance team at the Moorfields Eye Hospital will be consulted before commencing data collection, and weekly thereafter to ensure no deviation from the described protocol.
Google DeepMind has access to the required data to support the research aims of this study. To ensure compliance with the common law principle of data confidentiality, Google DeepMind will only receive anonymised or pseudonymised data from the Trust. DeepMind works with Moorfields Eye Hospital to ensure accuracy and clarity in the data to allow useful and consistent interpretation at all times.
The results will be disseminated through normal academic channels, initially focusing on conference proceedings and the indexed peer reviewed literature relevant to the fields of machine learning, artificial intelligence, ophthalmology and clinical research. Patient and Public Involvement representatives will be involved at each stage of the research.
We propose an exploratory study and initial testing of machine learning algorithms to analyse and quantify pathological and anatomical features in eye images. The results will be compared to expert annotations.
Should development be successful we plan to submit additional applications for permission to refine the model if required and ultimately investigate the performance in a clinical implementation setting.
Mr Will Kay is the Google DeepMind data custodian and is responsible for protection and security of the dataset described in this protocol.
Acknowledgement is made to Moorfields Reading Center and Moorfields Information Governance team who provided input in the IG aspects of study design.
Thank you very much for the opportunity to review the manuscript by De Fauw and co-workers. The topic is of great interest. Moorefields recently announced their research partnership with DeepMind Health. In this manuscript the project plan is detailed. One million examinations (digital fundus photographs and OCT scans) which met the inclusion criteria will be analysed using machine learning and Artificial Intelligence techniques.
The authors did not specify why they did not include angiography as an additional imaging modality which is usually used at baseline or at follow-up exams in patients with AMD and diabetic retinopathy. Please add this information.
Since data security is an important issue to be discussed especially when evaluating such large data sets the authors need to provide more information regarding patient consent. Does everyone who is examined at Moorefields Eye Hospital give a general consent for evaluating their data? Patients are not specifically informed about the project with DeepMind Health, aren't they? So data of patients are only excluded if patients previously (independent of this project) requested that there data should not be shared.
Although there might be some risk regarding "data security" it outweighs the potential of earlier detection and treatment of million of patients.
Reviewer Expertise:
NA
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
There is a good introduction of the two main conditions, AMD and DR and their burden on public health.
Very little information on machine learning to convince the reader that the fundamental processes and pieces of the jigsaw are already available to apply to AMD and DR or in simpler healthcare scenarios.
How does MEH plan to locate the ID of those who have given instructions for their images NOT to be used? Is there a database for this specific parameter?
Good attention to data security and data protection.
Not very much information on specific data outcomes expectations. Eg how detailed will be grading be of a wet AMD lesion.
Overall, a novel concept and worth exploring as it will be able to replace human workforce if successful.
Reviewer Expertise:
NA
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.