F1000ResearchF1000Research2046-1402F1000 Research LimitedLondon, UK10.12688/f1000research.8786.1Method ArticleArticlesBioinformaticsEpidemiologyHealth Systems & Services ResearchStatistical Methodologies & Health InformaticsDigital methodology to implement the ECOUTER engagement process
[version 1; peer review: 2 approved]
WilsonRebecca C.a1ButtersOliver W.1ClarkTom2MinionJoel1TurnerAndrew1MurtaghMadeleine J.1
Data 2 Knowledge Research Group, School of Social and Community Medicine, University of Bristol, Bristol, BS8 2BN, UK
MRC Integrative Epidemiology Unit, University of Bristol, Bristol, BS8 2BN, UKbecca.wilson@bristol.ac.uk
MM developed the ECOUTER process. RW developed technical and implementation methodology, contributed to the ECOUTER Github repository and wrote the ECOUTER wiki. OB and TC contributed to the ECOUTER Github repository. JM developed and monitored ECOUTER processes during pilot implementation. All authors reviewed and edited the manuscript for important intellectual content.
Competing interests: The authors have no professional or personal connections to Mind42 (<a href="http://www.Mind42.com">http://www.Mind42.com</a>.)
ECOUTER (
Employing
COncept
ual schema for policy and
Translation
Engagement in
Research) – French for ‘to listen’ – is a new stakeholder engagement method incorporating existing evidence to help participants draw upon their own knowledge of cognate issues and interact on a topic of shared concern. The results of an ECOUTER can form the basis of recommendations for research, governance, practice and/or policy.
This paper describes the development of a digital methodology for the ECOUTER engagement process based on currently available mind mapping freeware software. The implementation of an ECOUTER process tailored to applications within health studies are outlined for both online and face-to-face scenarios. Limitations of the present digital methodology are discussed, highlighting the requirement of a purpose built software for ECOUTER research purposes.
digital methodologyStackeholder engagementfreeware softwareThe research leading to these results was supported by the Biobank Standardisation and Harmonisation for Research Excellence in the European Union (BioSHaRE-EU) program which received funding from the European Union Seventh Framework Programme (FP7/2007–2013) under grant agreement no 261433, and the University of Bristol’s ‘Thinking Futures Festival’ which received funding from the Economic and Social Research Council, Festival of Social Science.The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.Introduction
Engaging stakeholders is understood to be essential to produce responsible practice in research as well as in business and public provision of social and health services. Stakeholder engagement brings together individuals or groups who have an interest, a stake, in a topic or issue. It makes sense that the people most involved or most affected by research, business or public actions would best understand how these practices affect them. However, achieving effective stakeholder engagement – engagement which represents all stakeholders equally, not just the articulate and powerful, and which engages at a depth and breadth that is appropriate to the issue at hand – is known to be potentially difficult, time consuming and expensive
1. Most existing methods rely on being able to bring people together in real time in a single or small number of locations. Because of these difficulties, stakeholder engagement often represents only a partial understanding of an issue and may not take account of potentially important perspectives. Even when there is a genuine commitment to giving voice to diverse perspectives, stakeholder engagement may exclude the very people it seeks to involve because its structures aren’t sufficiently agile, inclusive or accessible. Our aim was to develop a method and mechanism that was simple and accessible, yet allowed for a depth of analysis needed to uncover and disentangle the complexities and nuances that emerge when bringing together numerous personal understandings and experiences to understand an issue or topic.
Employing COnceptUal schema for policy and Translation Engagement in Research (ECOUTER,
http://www.bristol.ac.uk/ecouter) is a new methodology for stakeholder engagement, utilising concept mapping to collaboratively address a question of interest in a defined stakeholder community (the method is summarised in the
ECOUTER introductory video). Taken from the French verb ‘to listen’, ECOUTER brings together the knowledge, skills and experience of stakeholder contributors and supports a two-way process of informing and generating evidence and understandings of the issue in question from those who know it best. Social science methods of analysis (such as those described by Glaser
2) of contributions made during the engagement process are applied iteratively resulting in qualitative findings and recommended actions. The ECOUTER process as outlined below can lead to the development of recommendations for research, governance, policy and practice.
In this paper we describe the development of a digital methodology for the ECOUTER engagement process and share the instructions for its implementation. A forthcoming paper (Murtagh, MJ., Minion, J.T., Turner, A., Wilson, R.C., Blell, M., Ochieng, C.A., Murtagh, B.M., Roberts, S. and Butters, O.W. ECOUTER (Employing COnceptUal schema for policy and Translation Engagement in Research): a tool to engage research stakeholders, 2016
unpublished report), fully describes the rationale and development of the ECOUTER stakeholder engagement process, including the analysis of a number of use cases.
The ECOUTER process
In practice ECOUTER is a four stage process based on:
Engagement and knowledge exchange: This stage involves defining a central question/issue and relevant stakeholder group(s) to facilitate discussion and contributions. The exchange in question may be undertaken online or face-to-face, though the online mechanism is anticipated to be of greatest utility for engaging stakeholders who are geographically distributed. We therefore describe the essential components for online engagement below.
Analysis: Once the ECOUTER has been conducted, the data are analysed using social science methods.
Concept and recommendation development: Analytic findings are then summarised in a conceptual schema; that is, a map of key concepts, their nature and relationships.
Feedback and refinement: The conceptual schema is fed back to the contributors and wider community along with recommendations for research, governance, policy and practice.
ECOUTER technical development and implementationSoftware specification and selection
An essential aspect of the ECOUTER methodology has been the requirement for contributions to be linked/threaded within a structured discussion space. A mind mapping approach offered an appropriate solution, providing a mechanism for the relationships between comments as well as enabling the comments themselves to be captured and visualised. The nature of the ECOUTER methodology necessitated the capacity to run within a number of stakeholder groups in multiple localities simultaneously. A synchronised, digital, software solution (rather than a paper-based one) was needed. A range of open-source and proprietary software exists for mind mapping. Given the importance of removing cost as a barrier to participation in an ECOUTER, we assessed and trialed a selection of open-source software and freeware solutions based on the research and user requirements outlined below. An online web-based solution rather than an installed computer program was identified as more inclusive, enabling real-time contributions across different platforms (across Windows, Linux, Mac) and from internet-enabled devices (e.g. smartphones, tablets, computers, etc.) regardless of a contributor’s physical location.
ECOUTER initiators required a simple user interface to administer, setup and manage the collaborative mind map. In addition, the data needed to be exported from the software in both image-based and text-based formats prior to analysis, with a mechanism to trace how the collaborative discussion space evolved. From a user perspective, it was essential for the software to have a simple user interface that enabled multi-user contributions within a mind map whilst retaining the anonymity of individual contributors.
The web-based collaborative mind mapping freeware
Mind42 was identified as an appropriate solution to be used within the ECOUTER framework. It has a simple user interface via a website, allowing researchers to initiate an ECOUTER mind map and to manage invited contributors. Multi-user, collaborative mind maps are possible in Mind42, with users remaining anonymous both to other contributors and to the researchers. The software includes versioning and a periodic history, documenting how the mind map evolves.
Mind42 has both a manual web accessible mechanism and an application program interface (API) to export the mind map in multiple formats. These include exports as an image, pdf, text record of all contributions, and formats compatible with other mind mapping software. Furthermore, the Mind42 native data format (a Mind42 .m42 file) is nested and hierarchical following a JSON file format, retaining metadata about the mind map including anonymised identifiers for individual contributors linked to their contributions, the number of individual contributions, and the date and time of individual contributions.
Implementation
The Mind42 software was capable of being implemented in both an online ECOUTER and a face-to-face ECOUTER as summarised below. Full documentation on the
ECOUTER wiki provides complete instructions on how to set up, run and manage an ECOUTER in either format.
Implementation Online. An online ECOUTER implementation has been developed to enable running an ECOUTER over a longer time period (e.g. weeks to months or longer). It has the benefits of allowing people to contribute to discussions regardless of time zone or geographic location, and supports contributors dropping in and out of discussions over the entire time period.
An ECOUTER administrator account is set up on Mind42 allowing ECOUTER facilitators to initiate, manage and moderate a mind map from start to finish. Once a stakeholder group has been defined, invitations are sent to potential participants followed by the creation of individual accounts on Mind42. The information required to create a Mind42 account is minimal: only an email address and a password.
ECOUTER facilitators seed the mind map with themes and evidence (based on review of existing evidence) using the administrator account on Mind42. This task can include linking to online material including videos, photos, papers, articles and other mind maps.
Figure 1 contains an example of a seeded ECOUTER mind map based on the ECOUTER question
What are the ethical, legal and social issues related to trust in data linkage undertaken in a pilot conducted with the
Public and Population Project in Genomics and Society, (P
3G) in late 2014. The ECOUTER question is in a blue box in the map centre, with seeded themes in capitalised text forming the primary branches and subsequent branches containing further seeded comments and evidence in the form of web links.
An example of a seeded ECOUTER mind map.
Mind42 stores versions of the visual mind map periodically. At this stage, back-ups of the mind map in the desired file formats (typically as a .png and .m42 (JSON) file formats) can be taken by the ECOUTER facilitators manually after seeding the map, and regularly during the ECOUTER to prevent data loss. This is particularly important as Mind42 has no advanced user management facility and contributors are able to over-write and delete contributions made by others. Alternatively, auto-backup of the ECOUTER mind map data can be implemented at this stage using the Mind42 API, including a snapshot of the initial seeded mind map. An example back-up script is available on the
ECOUTER Github repository.
ECOUTER facilitators invite registered stakeholders to contribute to the seeded mind map and moderate contributions, checking data backups periodically whilst the ECOUTER is running. Once the ECOUTER period is finished, facilitators close the mind map to new contributions and check the final data backups. An open source script available on the ECOUTER Github repository is used to flatten the native Mind42 data file, creating a human-readable table (.csv) of the mind map metadata, individual text contributions and preserving the final map structure. The data are then imported into computer assisted qualitative research tools (e.g. NVivo) for analysis, with a second copy archived.
Implementation face to face. A face-to-face implementation has been developed to run an ECOUTER over a shorter time period from hours to one day. It has the benefits of allowing people to contribute to discussions within an exhibition-style setting, which may be placed in a high traffic public place, conference or exhibition venue.
ECOUTER facilitators initiate, manage and arrange data backups in the same manner as outlined previously. Facilitators create a number of generic ECOUTER participant accounts on Mind42 through which individuals can contribute to the mind map. Internet enabled laptops and/or tablets, provided as part of an ECOUTER exhibition stand are each logged into the ECOUTER using these accounts on Mind42, thus allowing anonymous contributions to the mind map.
In an exhibition setting it is also possible to publish the mind map online on the Mind42 website so that it is publicly viewable (read-only) including its live evolution. The live mind map can then be displayed using a large-screen television or monitor at the exhibition stand or made available to participants via a QR code or similar.
Discussion and conclusions
The ECOUTER method utilising Mind42 has now been implemented and piloted five times:
September to November 2014 in collaboration with the Public Population Project in Genomics and Society, Montreal, Canada.
What are the ethical, legal and social issues related to trust and data linkage? Online, internationally available ECOUTER implementation over a period of several weeks.
November 2014 during the ESRC Festival of Social Research, Bristol.
Your medical records - hand over or hands off? Facilitated digital face-to-face ECOUTER implementation in a public space on one Saturday in a busy shopping centre.
June 2015 during the Translation in Healthcare conference, Oxford.
Translation and emerging technologies: what are your views on the social, ethical and legal issues? Digital face-to-face ECOUTER implementation during the lunch break of an international academic conference.
July 2015 during the BioSHaRE tool roll out meeting, Milan.
BioSHaRE Tools - Where to now? Manual ECOUTER implementation (paper-based, without Mind42) during a day-long workshop.
May to November 2016 during the data collection clinic of the Avon Longitudinal Study of Parents and Children (ALSPAC, publicly known as Children of the 90’s) cohort study asking study participants
What areas would you like Children of the 90s to research? Online ECOUTER over a period of six months.
Experience from the above pilots established the efficacy using the free mind mapping platform Mind42 during an ECOUTER. While it was an appropriate solution for the initial specification the pilots did, however, highlight a series of critical limitations and technical issues that need to be resolved before the full potential of the ECOUTER methodology can be realised.
Mind42 is free to use because it generates revenue via targeted advertising, with ECOUTER initiators having no control over the advertisements users are exposed to. Advertising has the potential to distract or influence ECOUTER contributors; users are able to remove adverts only by paying a fee to Mind42. In addition, there are concerns around confidentiality and analysis given that the data sits with Mind42, a company located in Austria. Data are therefore are subject to Austrian law.
Furthermore, the inclusion of several additional features are required within a collaborative mind mapping tool, tailored to the ECOUTER process, to facilitate and strengthen data analysis. These include:
Enhancements to facilitate ECOUTER management.
Advanced permission management is essential during an ECOUTER to manage users and user groups. This could be used to help define administrator, moderator and contributor roles during an ECOUTER and ensure secure use of the mind map (e.g. preventing contributions from being modified or deleted by others).
Enhancements to user experience.
Advanced mind map formatting and customisation will enhance readability and user experience. Mind42 has limited formatting capabilities, with only basic methods to format the size of text and colour of mind map branches. As an ECOUTER mind map grows, it can become difficult to navigate the volume of contributions without the use of more advanced formatting features such as bold or italic faces, multiple fonts, font size and text colour. Furthermore, it would be useful for researchers to customise publication grade mind maps for visual impact.
Agree/disagree buttons would allow contributors to agree/disagree with contributions made by others and to enable researchers to gauge agreement with a comment among the stakeholder community.
Enhancements for ECOUTER analysis.
Categorisation of contributors would provide researchers with additional information about participants which may be relevant to the ECOUTER question (e.g. level/area of expertise, gender, age) whilst still retaining their anonymity.
advanced analytics such as activity auditing would assist researchers in understanding and evaluating how the mind mapping tool is used by contributors during an ECOUTER process.
Finally, reliance on third party freeware poses risks to long life-cycle research projects because the software may change substantially in functionality and/or terms and conditions. The software can also shutdown, fail to be maintained or have software errors fixed. An open source self-built solution may be preferable for long term sustainability as an ECOUTER tool and mind mapping service that addresses both researcher and user requirements.
Data and software availability
The latest versions of the ECOUTER implementation scripts are available from the ECOUTER Github repository:
https://github.com/beccawilson/ecouter
Link to the software repository doi on Zenodo:
http://dx.doi.org/10.5281/zenodo.513523
Software license: GPLv3
Acknowledgements
The authors are grateful to all those who have supported the development of ECOUTER through their contributions in the Exploring innovative mechanisms to build trust in human health research biobanking (
Brocher Foundation workshop, June 2013) and the participants and supporting institutions involved in the ECOUTER use cases:
Avon Longitudinal Study of Parents and Children (ALSPAC), University of Bristol;
Economic and Social Research Council (ESRC) Festival of Social Research;
ELSI 2.0: International Collaboratory for Genomics and Society; Centre for Health, Law and Emerging Technologies (
HeLEX), University of Oxford; and,
Public Population Project in Genomics and Society.
ThomsonRMurtaghMKhawFM:
Tensions in public health policy: patient engagement, evidence-based public health and health inequalities.Qual Saf Health Care.2005;14(6):398–400.
1632678110.1136/qshc.2005.0141751744099GlaserBG:
The constant comparative method of qualitative analysis.Soc Prob.1965;12(4):436–445.
10.2307/798843WilsonRCButtersOWClarkT:
ecouter: ECOUTER implementation scripts.Zenodo.2016.
Data Source10.5256/f1000research.9456.r14267Reviewer response for version 1MorrisonMichael1RefereeFinlayTeresa2Co-referee
HeLEX – Centre for Health, Law & Emerging Technologies, Nuffield Department of Population Health, University of Oxford, Oxford, UK
Centre for Health, Law and Emerging Technologies, University of Oxford, Oxford, UK
Competing interests: One of the ECOUTER sessions listed by the authors was carried out at the 2015 Translation in Healthcare conference which was organised by the Centre for Health, Law and Emerging Technologies at the University of Oxford, at which MM and TF are currently employed.
A manuscript reporting the results of the aforementioned ECOUTER session is currently under review as part of a thematic collection of papers based on the Translation in Healthcare conference in the journal BMC Medical Ethics. MM is a guest editor on this collection although he has no direct involvement in the peer review of the ECOUTER paper.
This article describes a novel method for generating responses, opinions and perspectives from stakeholders in a given issue, using a digital engagement processes. The article represents a useful overview of this particular technique and represents a worthwhile contribution to what is an increasingly important area of research. This appraisal comes with the caveat that the article could be improved by addressing the following issues relating to the article content:
Given that a forthcoming paper describing the ECOUTER process and authored by the same research group is explicitly referenced in the introduction, it would be helpful to have a clearer delineation of how the article in F1000Research is distinct from the forthcoming paper (i.e. what is the
specific aim of this paper), or why two separate papers describing the ECOUTER process are warranted.
The authors state that mind-mapping data generated through the ECOUTER process are analysed using ‘social science methods’. This description is unduly vague. As the ECOUTER process generates primarily textual data, it would seem that methods of textual analysis, whether qualitative (e.g. grounded theory-driven thematic analysis) or quantitative (e.g. content analysis) are the most relevant and most applicable subset of social sciences methods (with appropriate methodological additions if visual data is also collected).
The concept of ‘mind mapping’ is employed but is not explained. It is not necessarily self-evident and should be explained in more depth or a suitable reference provided.
Implementation online: As pointed out by other reviewers, email addresses can be used to identify individuals especially when combined with other data.
Reviewer Expertise:
NA
We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
10.5256/f1000research.9456.r14352Reviewer response for version 1VearsDanya F.1Referee
Department of Public Health and Primary Care, Centre for Biomedical Ethics and Law, Katholieke Universiteit Leuven, Belgium, Leuven, Belgium
Competing interests: No competing interests were disclosed.
This paper constitutes a valuable contribution to the literature. The title appropriately represents the article and the abstract nicely summarises the content of the paper. All components of the article have been explained to a high standard and are appropriate to the authors' assessment of the mind mapping tool as an appropriate digital methodology for the ECOUTER engagement process. The conclusions drawn are balanced and justified given the assessments made.
I would encourage the authors to consider elaborating on the following minor points for clarity:
The paper states that the participants' remain anonymous to both the other participants and also the researchers. However, when implementing ECOUTER online using Mind42, an account is generated for the participants using their email address (which often includes participants' names). Can the administrator see who has registered? Can they see which participants are saying what and if they are part of the research team do they think this has any impact on the study?
How can it be assured that the seeded ECOUTER mind map is unbiased and representative of the current state of the literature/debate on the topic?
On page 3 (column 2, paragraph 3) the authors state that the facilitator moderates contributions. Could they very briefly clarify what this involves?
In the discussion and conclusions, they list 5 successfully implemented pilot studies, the last of which is stated to be running from May to November 2016. Is this year correct? If so, I would exercise caution in calling something that was only set up one month ago and is set to run for another five months "successful".
Well done to the authors on a very interesting initiative and a good assessment of the strengths and limitations of their current approach.
Reviewer Expertise:
NA
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.