Assessment of a demonstrator repository for individual clinical trial data built upon DSpace

Selection of DSpace as software for developing a demonstrator repository

Writing a bespoke repository system from scratch was seen as unrealistic, given resource constraints, and in any case less useful than using an existing system – one that would also be available to potential repository managers. A variety of systems were considered as the possible base system for the demonstrator repository (e.g. Figshare, DSpace). These and other systems were characterised with respect to the following standardised criteria⁶:

A formal comparison between the systems was not made⁶, but DSpace was rated as the system with the greatest potential for a demonstrator repository, particularly in an academic context.

A formal comparison between the systems was not made⁶, but DSpace was rated as the system with the greatest potential for a demonstrator repository, particularly in an academic context.

DSpace was selected partly because it appears to be by far the most popular of the various repository systems, with almost 2884 users, 2204 of them listed as ‘academic’ (including the University of Cambridge, Yale, Duke University and the University of Edinburgh amongst many around the world; https://duraspace.org/registry/). Three of 25 repositories for IPD from clinical trial data, evaluated in a recent review, are built upon DSpace (Dryad, Drum, Edinburgh DataShare)².

In addition, DSpace is an open source system and can be modified and extended by users. It claims about 100 contributors to the code base, with the Dryad repository, which runs on DSpace, being an example of how the system can be extended. It is possible to download and run a pre-configured ‘out of the box’ solution, but DSpace also claims to be fully modifiable, even though many of the modifications listed are relatively superficial (e.g. themes, screen configurations, search parameters). The system appeares compliant with most of the relevant standards (e.g. Open Archives Initiative Protocol for Metadata (OAI-PMH)), developed for harvesting metadata descriptions from records), runs on a variety of operating systems and can use either Oracle or PostgreSQL as the back-end database store (https://duraspace.org/dspace/). There also appeared to be an active user group and comprehensive documentation, including a Wiki (https://wiki.duraspace.org/display/DSPACE/). An alternative to DSpace would have been Invenio (https://invenio-software.org/), which delivers the repository units for Zenodo, OpenAIRE and CERN Open Data. Invenio appeared very focused on open data, however, while DSpace seemed to offer more possibilities for supporting more managed access. Further details of the candidate systems considered are given in 6.

Technical infrastructure for the demonstrator repository

A data repository was established between October 2018 and June 2019 within the Coordination Centre for Clinical Trials at the University of Düsseldorf, by BT (first author) using version 6.3 of DSpace. Additional software was installed to supplement DSpace functioning and manage servers and common server functionality.

Full list of the software and hardware used for the repository installations and details of the technical implementation of the demonstrator repository:

DSpace is a framework of a considerable number of different software tools that must work together to achieve an efficient DSpace installation. Prerequisite software tools must be downloaded, installed, tested, configured and integrated with each other. In addition to DSpace itself, the following were installed:

DSpace can be installed at different scales, allowing different amounts of data to be handled. In our usage scenario we assumed the storage of several hundred trials with a size of 10–100 MB per trial, uploaded over several years. We therefore decided to install a mid-range version of DSpace, able to accommodate a large number of clinical trials datasets. The virtual server was established with:

This mid-range system is capable of supporting an application with either a large number of items (roughly 50,000 files and associated metadata) or a large volume of activity (searches, accesses, downloads, etc.).

For testing, publicly available data and documents as from clinical trials were uploaded to the demonstrator repository. The data used are displayed on the welcome page of the DSpace demonstrator repository (http://90.147.75.211:8080/xmlui/).

Quality applied to the reference implementation

The quality criteria used for assessment were developed from an original collection of 34 attributes, themselves derived from previous work and discussion within CORBEL and the IMPACT Observatory project². These criteria were meant to provide a broad characterisation of a repository and included aspects assessing both a repository’s relative maturity and its suitability for clinical research data. From these criteria 8 features were selected as being especially important for clinical researchers wishing to deposit individual participant data (IPD). They were used in a general evaluation of repositories² and were also applied to the DSpace implementation.

These 8 criteria identified as being key to successful management of IPD are listed below².

Other standards and criteria for trustworthy digital repositories have been developed and are being applied, e.g., Data Seal of Approval, International Council for Science World Data Systems^7–10). These criteria usually examine more generic repository features, for example the nature of the security measures in place, the use of encryption, the technical infrastructure, staff competence, etc. Because in this exercise we were not evaluating a repository, but focusing instead on a specific tool, one that would sit within a repository, we did not look at these more general criteria in detail. Of course activities such as monitoring, reviewing and implementing security measures are very important, but we would see them mainly as the concern of the repository managing DSpace rather than DSpace itself. The relationship between the eight criteria used here and other standards and criteria available for repositories is explored further in the Discussion section (see also Table 3).

Managing metadata (data about data) is a key requirement of any repository system, though there are two distinct forms of metadata to consider. To promote interoperability and retain meaning within interpretation and analysis, shared data should be, as far as possible, structured, described and formatted using widely recognised data and metadata standards (e.g. Clinical Data Interchange Standards Consortium (CDISC), Core Outcome Measures in Effectiveness Trials (COMET), Medical Dictionary for Regulatory Activities (MedDRA))¹. The metadata in this context is descriptive, detailing the contents of the data. A repository should be able to check that such metadata is available, ideally in one of a range of specified formats, and support its inclusion with the data (see the details for criteria 1) but the responsibility for providing it rests with the data generators. But there is also a need for provenance and accessibility metadata, which is used to make up a repository's catalogue of content, and which describes, for example, the nature and source of the data, its date(s), the authors, and – especially important with sensitive data that is likely to be under managed access – how the data can be accessed, including the details of any application procedure. Providing such metadata is the responsibility of the repository itself, although ideally it is done in close collaboration with the data generators. This type of metadata is the subject of criterion 5.

In order to make the assessment of the criteria more operational and to distinguish features of the system (technical features) from measures around the system (e.g. policies and procedures), the criteria were split into specific requirements. This was performed by the group of authors. Table 1 provides a detailed breakdown of the eight criteria in terms of their associated ‘requirements’ – i.e. the features one would normally expect to see implemented. ‘System’ features (i.e. repository system and its technical features), are distinguished from ‘Procedures’ (i.e. function of the repository’s policies and procedures).

For example, to support ‘Guidelines for data upload and storage’, the requirements for the repository could include:

a), and b) are mainly aspects of the repository system and its technical features, whilst c) is more a function of the repository’s policies and procedures.

In the context of this study it is important to stress that only the requirements labelled as ‘system’ attributes in Table 1 were evaluated (19 of 29, or 66%). Each of these system features was assessed and its level of fulfilment within DSpace classified as:

The assessment of the requirements was performed by BT and based on publicly available information about DSpace (web pages, user manuals, Q&A pages, reports, etc.). DSpace was not contacted directly and but there was contact with the DSpace community. The Coordination Centre for Clinical Trials in Düsseldorf participated at a meeting of the German user community.

Guidelines for data upload and storage

DSpace exhibits a flexible approach to file storage by supporting a range of file types and metadata schemas (1a demonstrated). With a variety of tools available, along with detailed technical guidance, it also provides mechanisms for upload of files, including instructions (1b demonstrated).

De-identification practices before upload

The DSpace system has no published requirements or guidelines relating to the de-identification of uploaded data. It is the submitter’s responsibility to ensure that documents are consistent with current standards, guidelines and policies from official bodies and scientific organisations. The submitter is, however, able to use links to requirements, guidelines and/or tools, if these are established by the system’s administrator (2a partially demonstrated). As far as we could tell, neither the DSpace repository system nor the user community have implemented de-identification tools or programs, able to perform and document de-identification on an existing dataset (2b not demonstrated). Having said that it is worth noting that, should such support tools be created, DSpace does provide a task management system (known as the 'Curation System') in which such tools can be integrated and configured.

Control of quality of data

The control of the quality of data is more a question of procedures and workflow around a repository than technical features available in a particular system. Nevertheless, there are some technical features that could facilitate a quality control workflow. Some of these features are available within DSpace, usually as optional and configurable additions to the data upload process but they are limited to a predefined review workflow. This covers a single reviewer workflow, collection’s workflow steps and a score review workflow. This is certainly an important feature but does not correspond to a full quality-controlled process, which needs additional features like monitoring and tracking uploads, rejections, edits; reports about reviews in process and performed, etc. (3a partially demonstrated).

Formal contract regarding upload and storage

A formal data transfer contract signed by the data generator and the repository administrator should be a prerequisite for transferring clinical trial data to a repository, not least to clarify potential legal responsibilities under data protection legislation. At the end of the manual submission process in DSpace, the submitter (data generator) is asked to grant the repository service an appropriate distribution license (different licences can be made available to different user communities). The distribution license can be edited or customised, however, the platform does not provide a user interface to do this easily. Agreeing a distribution licence is not the same, however, as enforcing a data transfer agreement. Confirmation of the existence of a signed data transfer protocol can be required from the user, i.e. integrated within the distribution licence, if implemented. The demonstrator repository is not, however, able to provide support for constructing and editing such a document (4a partially demonstrated).

Application of a metadata schema to describe contents

DSpace can support multiple extended metadata schemas for describing an item. A qualified Dublin Core metadata is provided by default. Multiple schemas can be configured, and metadata fields selected from a mix of configured schemas to describe items (5a demonstrated). In addition, a new metadata schema can be created. In the demonstrator repository, the ECRIN Clinical Trial Metadata Schema was created¹¹ (5b demonstrated). DSpace has several tools to help data generators export content and metadata, ingest content and metadata tools and batch edit metadata (5c demonstrated). DSpace offers OAI-MPH, a standard protocol for metadata harvesting. Metadata are public in DSpace. Communities, Collections and Items are discoverable in the browse and search systems regardless of read authorisation. Therefore, everyone can access metadata of items openly (5d demonstrated).

Application of an identifier

DSpace uses the CNRI Handle System primarily to create a persistent identifier for every object (item, collection and community) stored in the system (6a demonstrated). DSpace also allows other persistent identifiers, such as a digital object identifier (DOI), to be applied to data sets to improve discoverability and to allow correct citation in DSpace. This is in parallel to the Handle System (6b demonstrated).

Flexibility of access

DSpace has sophisticated embargo management as well as full open access. Embargo settings allow submitters to define embargoes linked to specific dates, that by default are applied to all anonymous (non-administrator) access requests. Advanced embargo settings can be used to apply (or exclude) embargo policies for particular user groups (7a demonstrated). The DSpace system supports several common authentication systems, but web based self-attestation is not supported (7b not demonstrated). In this context the term 'self-attestation' refers to a registration like process where the user first has to provide information about themselves, including their contact details, and give details of the purpose for which they intend to use the data, together with any other information required by the data managers. Email details would then normally be verified (by clicking on a validation link sent to the address provided) before access would be granted.

Resources can be made available only to certain "privileged" users, and this functionality allows access through group membership to be implemented (7c demonstrated). The ‘request a copy’ functionality exists in DSpace to facilitate access in cases when uploaded content is not openly shared. With this feature, the data submitter or owner interacts directly with the requester on a case-by-case basis. More complex request evaluation processes, for example involving a data access committee, are not directly supported in DSpace, though could in theory be integrated into any dialog between the requestor and the data submitter (7d demonstrated). The DSpace administrator can assign permissions to a privileged user at the item, community and collections level, allowing granular access to different parts of datasets collections (7e demonstrated).

Long-term preservation and sustainability

These are two related issues, one dealing with the preservation of the data in the long term, the second with the sustainability of the repository itself. A repository’s longevity will mainly be dependent on resourcing and institutional commitment, and given the inevitable uncertainties around both of these a clear policy about what should happen to data if a repository is closed would clearly be a requirement for most potential users., At a technical level, however, provides some support for long term preservation mechanisms, e.g. checksums can be applied and verified on all items. It can also be integrated with the open source archiving system Archivematica, allowing the generation of system-independent Archival Information Packages (AIPs)¹² (8a, in so far it is a technical issue, demonstrated). DSpace also claims to have implemented a strategic plan for sustainability. Because it uses open technology, has a broad dissemination and usage, with a large user community and many diverse applications, the long-term availability and maintenance of the system is expected, if not guaranteed (8b demonstrated).

Assessment of the demonstrator repository

The performance in supporting sensitive personal data such as that from clinical trials was strong, with 14 requirements demonstrated (74%). This included strong support for different file types and metadata systems, a range of access control systems, including embargoes and granular access management, an integrated persistent identifier scheme plus support for other identifiers like DOIs, and good support for data management in the long-term.

Of the two areas that were not demonstrated at all, the first – the inability to incorporate de-identification tools in the submission workflow – is arguably an over ambitious requirement. Although general techniques certainly exist for de-identification this should normally be an exercise that is planned, documented and tested on a study-by-study basis, rather than an automatic process. Having links available to de-identification resources is probably a more realistic requirement.

The second missing requirement, the lack of a self-attestation system, is a feature that some data generators might want to use, as it requires much less administrative overhead then setting up access rights for groups and individuals. It would require an administrator to define the fields required for self-attestation and, like the current user registration process, it could be backed up by a system requiring confirmation of the email address given. Given the range of other access options available in DSpace it may not be a serious omission, but it is a missing feature that would be ‘nice to have’.

Of the three areas that were partially demonstrated, the need for repository managers to establish links to de-identification and other tools, rather than have them built-in to the system, may represent an additional task but it is one that should be relatively easy to do. It can also be argued that this approach is more flexible, and easier to keep up to date, than a set of links integrated into the system.

The second partially demonstrated area related to quality control. The submission workflow allows for up to three review stages, which is good, but few other elements of quality control and monitoring seemed to be built into the system. For repository managers handling sensitive data, it would be useful to have reports on upload and access or access request activity, and the ability to integrate checklists of required features or information (such as de-identification status, metadata completeness, access types allowed or identifiers applied), as might be applied during the review process, to tag on the data itself (i.e. within internal system metadata). This would allow the status of the data in the repository to be better monitored and potential issues with data quality and/or legal issues to be more quickly identified.

The third partially demonstrated issue related to data transfer agreements, governing the terms of data upload and storage. Sensitive data requires more than a simple upload to a repository because, unless the data is fully anonymised, there are likely to be legal issues that need to be clarified, for instance exactly which institution is acting as the Data Controller, as that term is defined in the General Data Protection Regulation (GDPR). (At the very least, the legal status of the data needs to be clear, i.e. does it fall under data protection legislation, and if so which, or is it exempt from such consideration because of the way it has been prepared.) In addition, there may be questions about who is responsible for versioning data if it is changed, for paying any associated costs, about the access management required, and who needs to review access requests if access is managed (etc.). These considerations go well beyond any general agreement whereby data generators simply grant the repository the right to make their data available under a selected licence – and for sensitive data they may need to be considered on a study by study basis.

It would therefore be very useful if – as a configured option – the system could enforce a clear check that such a data transfer agreement was in place, preferably with the date of its application. (At the moment that seems possible, but a rather complex workaround is required.) It would be even better if the system could also indicate where the data transfer agreement was stored and link to it, or even display its provisions within the system. Ideally, a mature system would even allow the agreement to be drafted and agreed within the system, as part of a private interchange between the data uploader and the repository.

Weaknesses of the study

A limitation of the study is that it is focusing only on the 8 repository features defined in Banzi et al.². Other quality features not considered here may also be very important, for example good data security. This study should therefore be seen as a starting point, which will need further extension, perhaps using alternate approaches and systems (see next section).

We focused on attributes that we thought were particularly important for clinical trial and similar data. Aspects of quality for data repositories that have been cited by other authors, but which have not been explicitly considered in our approach include:

Transparency and accountability have been referenced by Hrynaszkiewicz et al.⁷ and by Burton et al.⁸. Allowing access to data in a timely manner and including a proportionate review of the scientific rationale, without introducing unnecessary barriers has been formulated by 7. Science Europe supports the idea of a metadata repository, enabling referencing to related relevant information, such as other data and publications and asks for support of data versioning¹⁰. Effective audits are proposed by Burton et al.⁸. The ICSU World Data System requires that the repository has adequate funding and sufficient members of qualified staff managed through a clear system of governance to effectively carry out the mission and that the repository enables users to discover the data and refer to them in a persistent way through proper citation⁹. The ICSU World Data System⁹ requires that the repository functions on well-supported operating systems and other core infrastructural software and is using hardware and software technological appropriate to the services it provides to its designated community. In addition, the technical infrastructure of the repository should provide for protection of the facility and its data, product, services, and users⁸. The need to try and integrate these different approaches to assessing data repositories is discussed in the next section.

Another weakness of the study is that the assessment of the quality criteria is (necessarily) subjective – the criteria are not quantitative. In our approach, a rather simple scale based upon “demonstrated”, “partially demonstrated” and “not demonstrated” was used. The definition of the different categories may not have been precise enough to give an accurate representation of the repository’s functioning.

Finally, there may be an issue related to the sources and completeness of the information used. We only took publicly available information about DSpace into consideration (web pages, user manuals, Q&A pages, reports, etc.). We did not contact DSpace directly and were not in contact with their developers. We did, however, participate at a meeting of the German user community and had discussions with a DSpace user. It should be noted, however, that transparency has been formulated as one the main principles for trusted repositories: “In order to select the most appropriate repository for a particular use case, all potential users benefit from being able to easily find and access information on the scope, target user community, policies, and capabilities of the data repository.”¹³. As a consequence, publicly available information should be sufficient to basically assess a repository.

Approaches and systems for assessing the quality of repositories

There are overarching general principles that address aspects around data management and data repositories on a very high level. In the FAIR principles, it is formulated that data should be Findable, Accessible, Interoperable and Reusable¹⁴. The TRUST principles formulate guidance for digital repositories of research data with a focus on Transparency, Responsibility, User focus, Sustainability and Technology¹³. Concrete guidelines, recommendations and best practice for data sharing and for trusted repositories should follow these principles and should provide concrete help for implementation of these principles.

Different approaches have been used to assess the quality of repositories dedicated to data sharing, both of sensitive data and more generally, with different emphases laid upon different features. For instance, Hrynaszkiewicz at al.⁷ proposed additional features for data repositories to better accommodate non-public clinical datasets, including Data Use Agreements, whilst Burton et al.⁸ introduced the term “Data Safe Haven”, for sensitive data, and provided 12 criteria that characterised such a haven.

The Core Trustworthy Data Repositories Requirements⁹ are intended to reflect the characteristics of trustworthy repositories (for all types of data). All requirements are mandatory and are equally weighted, standalone items. Although some overlap is unavoidable, duplication of the evidence sought among requirements has been kept to a minimum where possible. The choices contained in the supplied checklists (e.g., repository type and curation level) are not considered to be comprehensive, and additional space is provided in all cases for the applicant to add ‘other’ (more idiosyncratic) information. This and any comments given may then be used to refine such lists in the future. The CoreTrustSeal Board offers all interested data repositories a core-level certification based on the DSA–WDS Core Trustworthy Data Repositories Requirements catalogue and procedures⁹.

One initiative of Science Europe¹⁰ was to develop a set of core requirements for data management plans (DMPs), as well as a list of criteria for the selection of trustworthy repositories where researchers can store their data for sharing. The different approaches are compared in Table 3. In light of the development of the European Open Science Cloud (EOSC) and the increasing pressure for data sharing, these requirements and criteria should help to harmonise rules on data management throughout Europe. This will aid researchers in complying with research data management requirements even when working with different research funders and research organisations.

In general, it may be necessary to better distinguish between criteria that are properties of the underlying infrastructure (e.g. staff preparation, physical security, logical security, appropriate technology) and those which are more tightly coupled to a specific repository system. In fact, we would suggest that there are three (overlapping) ‘layers’ of attributes that need to be considered – those associated with the underlying organisational infrastructure, those linked to the repository’s technical systems, and those derived from procedures and workflows. Future attempts to assess the quality of repositories should perhaps consider these layers more explicitly. In this study we were focused on the ‘system’ attributes, but a broader description and assessment of a demonstrator repository should examine all three aspects, perhaps across each of the three main functional areas of a data repository, i.e. data upload, data storage and data access.

None of the approaches described above is sufficient to classify the quality of repositories for clinical trial data, as pointed out by Banzi et al.². It may be that we need to differentiate criteria that should apply to all or most data repositories from those that only apply, or become more significant, in the context of particular types of data, like IPD. A general assessment, and especially a general ‘score’, of repositories may therefore be less meaningful than an assessment for particular types of data or data usage. Despite these difficulties we believe that it would be useful to try and achieve a consensus about what ‘quality’ means in terms of data repositories, in different contexts, both to support repository managers and to help guide and promote their use by researchers.