An analysis of characteristics of post-authorisation studies registered on the ENCePP EU PAS Register

Robert Carroll; Sreeram V. Ramagopalan; Javier Cid-Ruzafa; Dimitra Lambrelli; Laura McDonald

doi:10.12688/f1000research.12198.2

Home Browse An analysis of characteristics of post-authorisation studies registered...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Revised

An analysis of characteristics of post-authorisation studies registered on the ENCePP EU PAS Register

[version 2; peer review: 2 approved]

Robert Carroll¹, Sreeram V. Ramagopalan², Javier Cid-Ruzafa¹, Dimitra Lambrelli¹, Laura McDonald²

Robert Carroll¹, Sreeram V. Ramagopalan², [...] Javier Cid-Ruzafa¹, Dimitra Lambrelli¹, Laura McDonald²

PUBLISHED 10 Nov 2017

Author details Author details

¹ Real-World Evidence, Evidera, London, W6 8DL, UK
² Centre for Observational Research and Data Sciences, Bristol-Myers Squibb, Uxbridge, UB8 1DH, UK

Robert Carroll
Roles: Formal Analysis, Investigation, Writing – Review & Editing

Sreeram V. Ramagopalan
Roles: Supervision, Writing – Review & Editing

Javier Cid-Ruzafa
Roles: Writing – Review & Editing

Dimitra Lambrelli
Roles: Supervision, Writing – Review & Editing

Laura McDonald
Roles: Conceptualization, Formal Analysis, Investigation, Methodology, Project Administration, Writing – Original Draft Preparation

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Background: The objective of this study was to investigate the study design characteristics of Post-Authorisation Studies (PAS) requested by the European Medicines Agency which were recorded on the European Union (EU) PAS Register held by the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP).
Methods: We undertook a cross-sectional descriptive analysis of all studies registered on the EU PAS Register as of 18^thOctober 2016.
Results: We identified a total of 314 studies on the EU PAS Register, including 81 (26%) finalised, 160 (51%) ongoing and 73 (23%) planned. Of those studies identified, 205 (65%) included risk assessment in their scope, 133 (42%) included drug utilisation and 94 (30%) included effectiveness evaluation. Just over half of the studies (175; 56%) used primary data capture, 135 (43%) used secondary data and 4 (1%) used a hybrid design combining both approaches. Risk assessment and effectiveness studies were more likely to use primary data capture (60% and 85% respectively as compared to 39% and 14% respectively for secondary). The converse was true for drug utilisation studies where 59% were secondary vs. 39% for primary. For type 2 diabetes mellitus, database studies were more commonly used (80% vs 3% chart review, 3% hybrid and 13% primary data capture study designs) whereas for studies in oncology, primary data capture were more likely to be used (85% vs 4% chart review, and 11% database study designs).
Conclusions: Results of this analysis show that PAS design varies according to study objectives and therapeutic area.

Keywords

drug safety, pharmacovigilance, post-authorisation studies, study design

Corresponding author: Laura McDonald

Competing interests: RC, JC and DL are full-time employees of Evidera. SR and LM are full-time employees of Bristol-Myers Squibb.

Grant information: The author(s) declared that no grants were involved in supporting this work.

Copyright: © 2017 Carroll R et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

How to cite: Carroll R, Ramagopalan SV, Cid-Ruzafa J et al. An analysis of characteristics of post-authorisation studies registered on the ENCePP EU PAS Register [version 2; peer review: 2 approved]. F1000Research 2017, 6:1447 (https://doi.org/10.12688/f1000research.12198.2) First published: 14 Aug 2017, 6:1447 (https://doi.org/10.12688/f1000research.12198.1) Latest published: 10 Nov 2017, 6:1447 (https://doi.org/10.12688/f1000research.12198.2)

Revised Amendments from Version 1

We have made edits to the Introduction, to provide a more detailed background to post authorisation studies; the Methods, to more clearly specify what we did; and also to the Discussion, to better highlight limitations of the study as suggested by the reviewers.

See the authors' detailed response to the review by Xavier Kurz and Thomas Goedecke
See the authors' detailed response to the review by Lex Bouter

Introduction

Randomised trials are considered the gold standard in evaluating the efficacy of new healthcare interventions. However, despite their ability to potentially provide causal estimates of the efficacy of new treatments, their generalisability to patients in the real world is often unclear. Strict criteria excluding patients with comorbidities or those above a certain age may lead to participants in trials differing from the general clinical population^1,2. Observational studies offer a means of further characterising the safety and effectiveness of new healthcare interventions in real world clinical settings³.

Post-authorisation studies (PAS), also referred to as ‘postmarketing surveillance studies’ or ‘phase IV trials’, are an example of this type of research which aims to demonstrate the utilisation and safety profile of drugs following their regulatory approval. While these studies are predominantly non-interventional, PAS can include interventional study designs (as per the definition of post-authorisation safety study in GVP VIII). Historically, observational PAS have been criticized to be of poor quality and open to bias in order to increase sales at the cost of scientific rigor⁴. European pharmacovigilance legislation was introduced in an effort to increase transparency and ensure such studies are methodologically robust⁵.

Since July 2012 all PAS that are imposed as a condition of granting marketing authorisation are required to be published in a publicly available register. In Europe, the main register for PAS is the EU PAS Register and is held by the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). While one previous study has examined broad characteristics of studies registered on the EU PAS, the current study aimed to build on this work by further characterising studies requested by the EMA to understand trends in study design, data sources utilised, and the relationship to therapeutic area. As PAS are becoming increasingly mandated by the European Medicines Agency (EMA) and compliance is high⁶, approved PAS protocols can likely provide insights into the successful design of future studies.

Methods

Analysis focused on all PAS requested by the EMA which were recorded on the EU PAS Register as of 18^th October 2016. The EU PAS Register is accessible via the ENCePP website which is hosted and maintained by the European Medicines Agency (EMA). All studies in category 1 (imposed as a condition of marketing authorisation), category 2 (obligation of marketing authorisation), and category 3 (required by the risk management plans) on the EU PAS register were included. These studies currently (as at 6^th November 2017) represent 40% of studies recorded on the register. Our study results are therefore not generalisable to studies not requested by regulatory authorities.

Data from the Register on study status (finalised, ongoing or planned) were collected to allow comparison of studies across time. Information was also collected from the Register on medical condition to be studied (i.e. the disease area of interest listed by the study investigators), data sources used (if applicable), and study scope (effectiveness evaluation, drug utilisation or risk assessment studies). These data were collected from the publicly available information published on the PAS register (http://www.encepp.eu/encepp/studiesDatabase.jsp).

Studies were classified as either using primary data capture, secondary data capture, or a hybrid approach as used by Engel and colleagues⁷ who studied PAS protocols and assessments submitted from July 2012 to July 2015 to the EMA Pharmacovigilance Risk Assessment Committee (PRAC). Primary data capture refers to the collection of data specifically for the study. Secondary data is the use of data already collected for another purpose (e.g. administrative or claims databases; medical charts). Studies using a combination of primary and secondary approaches were classed as employing a hybrid approach. Studies using alternative approaches (e.g. interventional studies) to those described were not included, limiting our findings to the aforementioned study designs. It should be noted that study details on the register is not quality checked and therefore misclassification is possible.

Descriptive analysis was performed. Counts and percentages were used for categorical variables.

Results

As of 18^th October 2016, a total of 314 studies were identified on the EU PAS register. These studies included 81 (26%) finalised, 160 (51%) ongoing and 73 (23%) planned (Table 1).

Table 1. Type of EMA requested PAS registered in ENCePP by 18^th October 2016 by study status (finalized, ongoing or planned).

	Risk Assessment		Effectiveness Evaluation		Drug Utilization		Total
Finalized	42	20%	11	12%	47	35%	81	26%
Ongoing	112	55%	60	64%	59	44%	160	51%
Planned	51	25%	23	24%	27	20%	73	23%
Total	205	100%	94	100%	133	100%	314	100%

Study scope and design

Of the total studies identified, 205 (65%) included risk assessment in their scope, 133 (42%) included drug utilisation and 94 (30%) included effectiveness evaluation. The same study could cover more than one objective: 6 (2%) studies included drug utilisation and effectiveness evaluation, 36 (11%) drug utilisation and risk assessment, 32 (10%) risk assessment and effectiveness evaluation and 22 (7%) risk assessment, drug utilisation and effectiveness evaluation.

For the risk assessment studies, finalised studies were less common (42; 52% of all 81 finalised studies) than ongoing (112; 70% of all 160 ongoing studies) and planned (51; 70% of all 73 planned studies). For the effectiveness evaluation studies, again finalised studies were not as frequent (11; 14% of all 81 finalised studies) as ongoing (60; 38% of all 160 ongoing studies) or planned (23; 32% of all 73 planned studies). Finally, for the drug utilisation studies, finalised studies were more common (47; 58% of all 81 finalised studies) with 59 (37% of all 160 ongoing studies) ongoing and 27 (37% of all 73 planned studies) planned.

Just over half of the studies (175, 56%) used a primary data capture design, 135 (43%) used a secondary data capture design and 4 (1%) used a hybrid design combining both approaches.

More primary data capture studies were ongoing 102 (64% of all 160 ongoing studies) or planned 49 (67% of all 73 planned studies) than finalised 24 (30% of all of all 81 finalised studies). For secondary data capture, finalised studies were more frequent (57; 70%), as compared to 55 (34%) ongoing and 23 (32%) planned.

Risk assessment and effectiveness studies were more likely to use primary data capture (60% and 85% respectively for primary data capture as compared to 39% and 14% respectively for secondary data capture). The converse was true for drug utilisation studies, where 59% used secondary data capture vs. 39% for primary data capture.

Secondary data capture studies and data sources used

Of the secondary data capture studies, 117 (87%) used an existing claims or electronic medical record database (the remainder using a chart review approach). 93 (79%) studies used an existing real-world data source based in Europe alone, 9 (8%) studies used European and US data, 14 (12%) studies used US data alone and 1 (1%) study used Canadian data.

A single database was used in 58 (50%) studies, with the remainder using two or more. Where more than one data source was used, this was always from two or more countries. The most frequent established data source used was the United Kingdom’s (UK) Clinical Practice Research Datalink (CPRD, 31%), followed by Nordic (Denmark, Finland, Norway or Sweden) National registries (29%), and The Health Improvement Network (THIN, UK, 18%).

Disease area and study design

A total of 30 (10%) studies were in the field of type 2 diabetes mellitus, 29 (9%) in cardiovascular disease, 27 (9%) studies in oncology, 6 (2%) in chronic obstructive pulmonary disease (COPD) and 6 (2%) in multiple sclerosis (MS).

For type 2 diabetes mellitus, database studies were more commonly used (80% vs 3% chart review, 3% hybrid and 13% primary data capture). Similar patterns were seen for COPD (83% database vs 17% primary data capture) and cardiovascular disease (59% database vs 38% primary data capture and 3% hybrid design).

For studies in oncology, primary data capture was more likely to be used (85% vs 4% chart review, and 11% database study designs). A similar relationship was seen for MS (83% primary data capture vs 17% database study design).

Dataset 1.The full dataset used for analysis, extracted from the PAS register (http://www.encepp.eu/encepp/studiesDatabase.jsp) on the 18th of October 2016.

Discussion

Our analysis reveals a number of distinct characteristics of studies recorded on the register that can likely be used as a reference for future PAS designs.

Overall there appears to have been an increase in the number of risk assessment studies, as planned and ongoing studies were far more likely to include a risk assessment element than studies that had already been finalised.

Primary data capture study designs were the most commonly implemented study type, and again these were more frequently planned or ongoing, suggesting an increasing use or reflecting the longer nature in general to execute these types of studies. Furthermore, primary data capture were more commonly used for effectiveness and risk assessment studies. The predominance of primary data capture designs is similar to results observed by Engel et al., who studied 189 PAS protocols submitted to the EMA PRAC⁷.

Secondary data capture was more likely to be used for drug utilisation studies, and this was also described by Engel et al.⁷ Of the secondary studies, the CPRD in the UK was used in nearly a third of all database PAS. Registry data from the Nordic countries was only slightly less commonly used, with the remainder of existing data sources (eg. THIN, in the UK) being less frequently used.

Of the therapeutic areas investigated, type 2 diabetes mellitus, cardiovascular disease and COPD tended to use secondary data capture, whereas for oncology and MS, primary data capture studies were favoured.

Some limiting characteristics of current data sources in Europe are that they may only cover a specific type of patient care (e.g. primary or secondary) and not the full patient pathway. They may not collect data on everything that is needed for PAS objectives, for example in-hospital treatments, clinical endpoints or markers of disease severity. Given PAS studies are often requested on the grounds of exploring sub-groups of patients who may have been treated outside of these settings⁸, this is a major limitation of some data sources. Indeed, this may explain why primary data capture study designs are becoming more frequent, especially in the field of oncology. Furthermore, that a handful of data sources are very commonly used highlights the limited data landscape in Europe, and the need to improve coverage and access to routinely collected medical data for observational research. On the other hand, given some similarity in data elements captured in currently available datasets, studies using only a single database may have potentially missed a chance to validate any findings or increase statistical power to detect relevant effects. Additionally, some data sources (eg. CPRD) can be linked to other datasets or have an integrated data collection component to provide additional data. Nevertheless, the tendency for effectiveness and risk assessment studies to use primary data capture is also likely a reflection of the fact that this methodology can potentially overcome problems such as confounding, often associated with some secondary designs due to a lack of data availability on potential confounders.

In summary, we show here that using data from the EU PAS Register, study scope and therapeutic area are associated with the choice of study design agreed by health authorities and therapeutic drug makers for the conduct of PAS. While researchers should consider relevant guidelines (ENCePP Guide on Methodological Standards in Pharmacoepidemiology, Guidelines for Good Pharmacoepidemiology Practices of the International Society of Pharmacoepidemiology (ISPE GPP)) when designing PAS, the results of this study can contribute to the planning of future successful PAS.

Data availability

Dataset 1: The full dataset used for analysis, extracted from the PAS register (http://www.encepp.eu/encepp/studiesDatabase.jsp) on the 18th of October 2016.

DOI, 10.5256/f1000research.12198.d170864⁹

Competing interests

RC, JC and DL are full-time employees of Evidera. SR and LM are full-time employees of Bristol-Myers Squibb.

Grant information

The author(s) declared that no grants were involved in supporting this work.

F1000 recommended

References

1. Kalata P, Martus P, Zettl H, et al.: Differences between clinical trial participants and patients in a population-based registry: the German Rectal Cancer Study vs. the Rostock Cancer Registry. Dis Colon Rectum. 2009; 52(3): 425–437. PubMed Abstract | Publisher Full Text
2. Maasland L, Van Oostenbrugge RJ, Franke CF, et al.: Patients enrolled in large randomized clinical trials of antiplatelet treatment for prevention after transient ischemic attack or ischemic stroke are not representative of patients in clinical practice: the Netherlands Stroke Survey. Stroke. 2009; 40(8): 2662–2668. PubMed Abstract | Publisher Full Text
3. McDonald L, Lambrelli D, Wasiak R, et al.: Real-world data in the United Kingdom: opportunities and challenges. BMC Med. 2016; 14(1): 97. PubMed Abstract | Publisher Full Text | Free Full Text
4. Gale EA: Post-marketing studies of new insulins: sales or science? BMJ. 2012; 344: e3974. PubMed Abstract | Publisher Full Text
5. Borg JJ, Aislaitner G, Pirozynski M, et al.: Strengthening and rationalizing pharmacovigilance in the EU: where is Europe heading to? A review of the new EU legislation on pharmacovigilance. Drug Saf. 2011; 34(3): 187–197. PubMed Abstract | Publisher Full Text
6. Blake KV, Prilla S, Accadebled S, et al.: European Medicines Agency review of post-authorisation studies with implications for the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance. Pharmacoepidemiol Drug Saf. 2011; 20(10): 1021–9. PubMed Abstract | Publisher Full Text
7. Engel P, Almas MF, De Bruin ML, et al.: Lessons learned on the design and the conduct of Post-Authorization Safety Studies: review of 3 years of PRAC oversight. Br J Clin Pharmacol. 2017; 83(4): 884–893. PubMed Abstract | Publisher Full Text | Free Full Text
8. Hoekman J, Klamer TT, Mantel-Teeuwisse AK, et al.: Characteristics and follow-up of postmarketing studies of conditionally authorized medicines in the EU. Br J Clin Pharmacol. 2016; 82(1): 213–26. PubMed Abstract | Publisher Full Text | Free Full Text
9. Ramagopalan S, McDonald L, Carroll R, et al.: Dataset 1 in: An Analysis of Characteristics of Post-Authorisation Studies Registered on the ENCePP EU PAS Register. F1000Research. 2017. Data Source

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 14 Aug 2017

Author details Author details

¹ Real-World Evidence, Evidera, London, W6 8DL, UK
² Centre for Observational Research and Data Sciences, Bristol-Myers Squibb, Uxbridge, UB8 1DH, UK

Robert Carroll
Roles: Formal Analysis, Investigation, Writing – Review & Editing

Sreeram V. Ramagopalan
Roles: Supervision, Writing – Review & Editing

Javier Cid-Ruzafa
Roles: Writing – Review & Editing

Dimitra Lambrelli
Roles: Supervision, Writing – Review & Editing

Laura McDonald
Roles: Conceptualization, Formal Analysis, Investigation, Methodology, Project Administration, Writing – Original Draft Preparation

Competing interests

RC, JC and DL are full-time employees of Evidera. SR and LM are full-time employees of Bristol-Myers Squibb.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Article Versions (2)

version 2

Revised

Published: 10 Nov 2017, 6:1447

https://doi.org/10.12688/f1000research.12198.2

version 1

Published: 14 Aug 2017, 6:1447

https://doi.org/10.12688/f1000research.12198.1

© 2017 Carroll R et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Carroll R, Ramagopalan SV, Cid-Ruzafa J et al. An analysis of characteristics of post-authorisation studies registered on the ENCePP EU PAS Register [version 2; peer review: 2 approved]. F1000Research 2017, 6:1447 (https://doi.org/10.12688/f1000research.12198.2)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 2

VERSION 2

PUBLISHED 10 Nov 2017

Revised

Views

Reviewer Report 20 Nov 2017

Lex Bouter, VU University Medical Centre, Department of Epidemiology and Biostatistics, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

Approved

https://doi.org/10.5256/f1000research.14284.r27832

My comments and suggestions have been responded to adequately, but the author did not change the manuscript accordingly in all instances.

Specifically I recommend to add the following study limitations to the discussion section:

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 13 Nov 2017

Xavier Kurz, Pharmacovigilance and Epidemiology Department, European Medicines Agency, London, UK

Thomas Goedecke, Pharmacovigilance and Epidemiology Department, European Medicines Agency, London, UK

Approved

https://doi.org/10.5256/f1000research.14284.r27831

Most comments have been addressed.

Minor comments regarding the Introduction:

"GVP VIII": write in full and provide the link.
"as a condition for granting marketing authorisation": not only for granting, as

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 14 Aug 2017

Views

Reviewer Report 23 Oct 2017

Lex Bouter, VU University Medical Centre, Department of Epidemiology and Biostatistics, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

Approved with Reservations

https://doi.org/10.5256/f1000research.13205.r26965

The article contains a descriptive analysis of some characteristics of Post-Authorisation Studies registered in the ENCePP EU PAS Register. It’s clearly written and informative within the limited scope the authors set out to study. There are a number of issues ... Continue reading

The 314 included studies were on 18^th October 2016 contained in the ENCePP EU PAS Register AND concerned a PAS requested by EMA. That makes the reader wonder how many PAS requested by EMA (since a suitable date) were not contained in the Register. And also how many studies contained in the Register were not requested by EMA. In other words: how selective is the sample studied?
The conclusions come as a surprise because no explicit study questions or hypotheses were formulated. Furthermore, the conclusions are overstated in the sense that both the cross-sectional nature and the lack of a statistical (multivariable) data-analysis make a causal interpretation (‘influence’) unwarranted.
A serious study limitation is that no checks are done (for the completed studies) for non-publication, and for selective publication and other discrepancies between protocol and publication for the published studies.

Minor issues

Please explain better what is meant by PAS. Most readers will probably be more familiar with terms like ‘postmarketing surveillance’ and ‘phase IV trials’.
I assume that entries in the Registry need to be made before the start of data collection. And also that later changes (amendments) to the protocol cannot be made without leaving traces (log file). But this is not clearly explained in the current version of the article.
The Introduction section suggests that the European pharmacovigilance legislation was introduced to foster transparency and methodological quality of the Post-Authorization Studies. That may be correct, but the current study says very little on transparency and nothing about the methodological quality of PAS.
The first sentence of the Discussion section suggests that this article will ‘likely be used as reference for future PAS designs’. It completely escapes me why this would be so.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

No
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

No

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Epidemiology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 10 Nov 2017

Sreeram Ramagopalan, Centre for Observational Research and Data Sciences, Bristol-Myers Squibb, Uxbridge, UB8 1DH, UK

10 Nov 2017

Author Response

We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

Major issues
The 314 included studies were on 18^th October 2016 ... Continue reading We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

Major issues
The 314 included studies were on 18^th October 2016 contained in the ENCePP EU PAS Register AND concerned a PAS requested by EMA. That makes the reader wonder how many PAS requested by EMA (since a suitable date) were not contained in the Register. And also how many studies contained in the Register were not requested by EMA. In other words: how selective is the sample studied?

Response: Looking at recent figures on the PAS register (06/11/17) 40% (482/1191) fit these criteria. The lack of generalisability of our findings to studies that are not requested by the EMA is now highlighted in the results

The conclusions come as a surprise because no explicit study questions or hypotheses were formulated. Furthermore, the conclusions are overstated in the sense that both the cross-sectional nature and the lack of a statistical (multivariable) data-analysis make a causal interpretation (‘influence’) unwarranted.

Response: We had not intended to claim causal inference, but simply describe associations/trends observed. We have removed wording relating to ‘influence’.

A serious study limitation is that no checks are done (for the completed studies) for non-publication, and for selective publication and other discrepancies between protocol and publication for the published studies.

Response: The reviewer highlights an important point, but one that is not covered by the aim of our study. We describe the design of studies on the PAS register, we are not addressing the question of whether researchers comply with their study protocols when publishing results (although this would be an interesting analysis!)

Minor issues

Please explain better what is meant by PAS. Most readers will probably be more familiar with terms like ‘postmarketing surveillance’ and ‘phase IV trials’.

Response: These terms have been added to the manuscript where the concept of a PAS is first introduced

I assume that entries in the Registry need to be made before the start of data collection. And also that later changes (amendments) to the protocol cannot be made without leaving traces (log file). But this is not clearly explained in the current version of the article.

Response: We are not aware of a requirement for studies to have been entered on the register before data collection starts. Amendments can be made but no record of these amendments is made

The Introduction section suggests that the European pharmacovig)ilance legislation was introduced to foster transparency and methodological quality of the Post-Authorization Studies. That may be correct, but the current study says very little on transparency and nothing about the methodological quality of PAS.

Response: We agree, our study did not attempt to quantify transparency nor review the methodological rigor of studies, simply provide a overview of the study designs employed.

The first sentence of the Discussion section suggests that this article will ‘likely be used as reference for future PAS designs’. It completely escapes me why this would be so.

Response: The results presented on study objectives/study design could inform readers on designs most commonly employed to address certain research questions, and therefore potentially inform suitable designs for future studies.
We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

Major issues
The 314 included studies were on 18^th October 2016 contained in the ENCePP EU PAS Register AND concerned a PAS requested by EMA. That makes the reader wonder how many PAS requested by EMA (since a suitable date) were not contained in the Register. And also how many studies contained in the Register were not requested by EMA. In other words: how selective is the sample studied?

Response: Looking at recent figures on the PAS register (06/11/17) 40% (482/1191) fit these criteria. The lack of generalisability of our findings to studies that are not requested by the EMA is now highlighted in the results

The conclusions come as a surprise because no explicit study questions or hypotheses were formulated. Furthermore, the conclusions are overstated in the sense that both the cross-sectional nature and the lack of a statistical (multivariable) data-analysis make a causal interpretation (‘influence’) unwarranted.

Response: We had not intended to claim causal inference, but simply describe associations/trends observed. We have removed wording relating to ‘influence’.

A serious study limitation is that no checks are done (for the completed studies) for non-publication, and for selective publication and other discrepancies between protocol and publication for the published studies.

Response: The reviewer highlights an important point, but one that is not covered by the aim of our study. We describe the design of studies on the PAS register, we are not addressing the question of whether researchers comply with their study protocols when publishing results (although this would be an interesting analysis!)

Minor issues

Please explain better what is meant by PAS. Most readers will probably be more familiar with terms like ‘postmarketing surveillance’ and ‘phase IV trials’.

Response: These terms have been added to the manuscript where the concept of a PAS is first introduced

I assume that entries in the Registry need to be made before the start of data collection. And also that later changes (amendments) to the protocol cannot be made without leaving traces (log file). But this is not clearly explained in the current version of the article.

Response: We are not aware of a requirement for studies to have been entered on the register before data collection starts. Amendments can be made but no record of these amendments is made

The Introduction section suggests that the European pharmacovig)ilance legislation was introduced to foster transparency and methodological quality of the Post-Authorization Studies. That may be correct, but the current study says very little on transparency and nothing about the methodological quality of PAS.

Response: We agree, our study did not attempt to quantify transparency nor review the methodological rigor of studies, simply provide a overview of the study designs employed.

The first sentence of the Discussion section suggests that this article will ‘likely be used as reference for future PAS designs’. It completely escapes me why this would be so.

Response: The results presented on study objectives/study design could inform readers on designs most commonly employed to address certain research questions, and therefore potentially inform suitable designs for future studies.
Competing Interests: RC, JCR and DL are employees of Evidera Inc. SR and LM are employees of Bristol-Myers Squibb Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 10 Nov 2017

Sreeram Ramagopalan, Centre for Observational Research and Data Sciences, Bristol-Myers Squibb, Uxbridge, UB8 1DH, UK

10 Nov 2017

Author Response

We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

Major issues
The 314 included studies were on 18^th October 2016 ... Continue reading We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

Major issues
The 314 included studies were on 18^th October 2016 contained in the ENCePP EU PAS Register AND concerned a PAS requested by EMA. That makes the reader wonder how many PAS requested by EMA (since a suitable date) were not contained in the Register. And also how many studies contained in the Register were not requested by EMA. In other words: how selective is the sample studied?

Response: Looking at recent figures on the PAS register (06/11/17) 40% (482/1191) fit these criteria. The lack of generalisability of our findings to studies that are not requested by the EMA is now highlighted in the results

The conclusions come as a surprise because no explicit study questions or hypotheses were formulated. Furthermore, the conclusions are overstated in the sense that both the cross-sectional nature and the lack of a statistical (multivariable) data-analysis make a causal interpretation (‘influence’) unwarranted.

Response: We had not intended to claim causal inference, but simply describe associations/trends observed. We have removed wording relating to ‘influence’.

A serious study limitation is that no checks are done (for the completed studies) for non-publication, and for selective publication and other discrepancies between protocol and publication for the published studies.

Response: The reviewer highlights an important point, but one that is not covered by the aim of our study. We describe the design of studies on the PAS register, we are not addressing the question of whether researchers comply with their study protocols when publishing results (although this would be an interesting analysis!)

Minor issues

Please explain better what is meant by PAS. Most readers will probably be more familiar with terms like ‘postmarketing surveillance’ and ‘phase IV trials’.

Response: These terms have been added to the manuscript where the concept of a PAS is first introduced

I assume that entries in the Registry need to be made before the start of data collection. And also that later changes (amendments) to the protocol cannot be made without leaving traces (log file). But this is not clearly explained in the current version of the article.

Response: We are not aware of a requirement for studies to have been entered on the register before data collection starts. Amendments can be made but no record of these amendments is made

The Introduction section suggests that the European pharmacovig)ilance legislation was introduced to foster transparency and methodological quality of the Post-Authorization Studies. That may be correct, but the current study says very little on transparency and nothing about the methodological quality of PAS.

Response: We agree, our study did not attempt to quantify transparency nor review the methodological rigor of studies, simply provide a overview of the study designs employed.

The first sentence of the Discussion section suggests that this article will ‘likely be used as reference for future PAS designs’. It completely escapes me why this would be so.

Response: The results presented on study objectives/study design could inform readers on designs most commonly employed to address certain research questions, and therefore potentially inform suitable designs for future studies.
We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

Major issues
The 314 included studies were on 18^th October 2016 contained in the ENCePP EU PAS Register AND concerned a PAS requested by EMA. That makes the reader wonder how many PAS requested by EMA (since a suitable date) were not contained in the Register. And also how many studies contained in the Register were not requested by EMA. In other words: how selective is the sample studied?

Response: Looking at recent figures on the PAS register (06/11/17) 40% (482/1191) fit these criteria. The lack of generalisability of our findings to studies that are not requested by the EMA is now highlighted in the results

The conclusions come as a surprise because no explicit study questions or hypotheses were formulated. Furthermore, the conclusions are overstated in the sense that both the cross-sectional nature and the lack of a statistical (multivariable) data-analysis make a causal interpretation (‘influence’) unwarranted.

Response: We had not intended to claim causal inference, but simply describe associations/trends observed. We have removed wording relating to ‘influence’.

A serious study limitation is that no checks are done (for the completed studies) for non-publication, and for selective publication and other discrepancies between protocol and publication for the published studies.

Response: The reviewer highlights an important point, but one that is not covered by the aim of our study. We describe the design of studies on the PAS register, we are not addressing the question of whether researchers comply with their study protocols when publishing results (although this would be an interesting analysis!)

Minor issues

Please explain better what is meant by PAS. Most readers will probably be more familiar with terms like ‘postmarketing surveillance’ and ‘phase IV trials’.

Response: These terms have been added to the manuscript where the concept of a PAS is first introduced

I assume that entries in the Registry need to be made before the start of data collection. And also that later changes (amendments) to the protocol cannot be made without leaving traces (log file). But this is not clearly explained in the current version of the article.

Response: We are not aware of a requirement for studies to have been entered on the register before data collection starts. Amendments can be made but no record of these amendments is made

The Introduction section suggests that the European pharmacovig)ilance legislation was introduced to foster transparency and methodological quality of the Post-Authorization Studies. That may be correct, but the current study says very little on transparency and nothing about the methodological quality of PAS.

Response: We agree, our study did not attempt to quantify transparency nor review the methodological rigor of studies, simply provide a overview of the study designs employed.

The first sentence of the Discussion section suggests that this article will ‘likely be used as reference for future PAS designs’. It completely escapes me why this would be so.

Response: The results presented on study objectives/study design could inform readers on designs most commonly employed to address certain research questions, and therefore potentially inform suitable designs for future studies.
Competing Interests: RC, JCR and DL are employees of Evidera Inc. SR and LM are employees of Bristol-Myers Squibb Close
Report a concern

Views

Reviewer Report 28 Sep 2017

Xavier Kurz, Pharmacovigilance and Epidemiology Department, European Medicines Agency, London, UK

Thomas Goedecke, Pharmacovigilance and Epidemiology Department, European Medicines Agency, London, UK

Approved with Reservations

https://doi.org/10.5256/f1000research.13205.r26352

The article is well written and provides a descriptive analysis of the EU PAS Register. This type of research is welcome as an evaluation of the usefulness of the EU PAS Register. The article would benefit from a couple of clarifications:
... Continue reading

The EU PAS Register is accessible via the ENCePP website which is hosted and maintained by the European Medicines Agency (EMA).
The term post-authorisation study (PAS) is not exclusive to observational research and may include interventional study designs (e.g. see definition of post-authorisation safety study in GVP VIII), this should be made clear as the introduction refers to historical ‘criticism’ raised mainly in context of non-interventional research. The majority of studies in the EU PAS Register are non-interventional but also a few RCTs fall under the inclusion criteria applied by the authors for their descriptive study.
The EU pharmacovigilance legislation and the requirement to publish PAS imposed as a condition of marketing authorisation in a public register came into force in July 2012, not in 2010 (the new legislation was however published in December 2010).
The authors should explain what they mean by ‘approved PAS’. Regulatory approval of PAS is not a guarantee for successful conduct of a study. The EU pharmacovigilance legislation provides for a regulatory procedure to approve protocols of post-authorisation safety studies (PASS) imposed as a condition of marketing authorisation (EU RMP category 1 and 2 studies) before study start to ensure the objectives of the PASS are met and the protocol follows the format requirements. However, this needs to be distinguished from regulatory supervision of EU RMP category 3 studies where prior protocol approval is not required by law (but may be requested to the marketing authorisation holder voluntarily). The general statement above should be balanced, also in the discussion. Like for any other study, relevant scientific guidance should be considered by marketing authorisation holders and investigators for the development of study protocols, the conduct of studies and the writing of study reports (e.g. ENCePP Guide on Methodological Standards in Pharmacoepidemiology, Guidelines for Good Pharmacoepidemiology Practices of the International Society of Pharmacoepidemiology (ISPE GPP) etc.).

Methods

Clarify if only studies requested by EMA or also by other EU or non-EU regulators have been included in the descriptive analysis. This is important for the discussion of the results on data sources.
The authors have chosen 3 distinct study scopes as inclusion criteria, but there might be other studies requested by a regulator which provide insight in trends in study design, data sources and therapeutic areas (a query performed on 27.09.2017 found more than 170 studies requested by a regulator with other scopes). The authors should explain this limitation.

Results

The presentation of the results on study status, data sources and disease areas may benefit from a breakdown by EU RMP category to show any potential association of with the timelines for conducting studies.
The authors do not specify how the disease areas were defined (by ATC code?).

Discussion

The statement that planned and ongoing studies were far more likely to include a safety element is not clear from the presented results; any study in an EU RMP is linked to a safety concern, only with varying scope i.e. effectiveness of measures implemented to manage risks (e.g. educational materials), changes in drug utilisation (e.g. restricted indication) or risk quantification (e.g. frequency of adverse reactions)?
A limitation of the study that may be worth explaining is that the registration of studies is voluntary (except for imposed PASS), and the entries are not verified as to their accuracy. For examples, it is not checked by ENCePP or EMA whether the RMP category entered is correct and corresponds to the outcome of the regulatory procedure. Therefore, there may be some misclassification. For PASS imposed by regulators, the RMP category field may have been incorrectly populated by the company.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: The reviewers are employees of the European Medicines Agency, which hosts the EU PAS Register.

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

CITE

Report a concern

Author Response 10 Nov 2017

Sreeram Ramagopalan, Centre for Observational Research and Data Sciences, Bristol-Myers Squibb, Uxbridge, UB8 1DH, UK

10 Nov 2017

Author Response
We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:
- The EU PAS Register is accessible via
... Continue reading
We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

The EU PAS Register is accessible via the ENCePP website which is hosted and maintained by the European Medicines Agency (EMA).

Response: This has been clarified in the methods.

The term post-authorisation study (PAS) is not exclusive to observational research and may include interventional study designs (e.g. see definition of post-authorisation safety study in GVP VIII), this should be made clear as the introduction refers to historical ‘criticism’ raised mainly in context of non-interventional research. The majority of studies in the EU PAS Register are non-interventional but also a few RCTs fall under the inclusion criteria applied by the authors for their descriptive study.

Response: This point has been clarified in the introduction

The EU pharmacovigilance legislation and the requirement to publish PAS imposed as a condition of marketing authorisation in a public register came into force in July 2012, not in 2010 (the new legislation was however published in December 2010).

Response: Text corrected, thank you.

The authors should explain what they mean by ‘approved PAS’. Regulatory approval of PAS is not a guarantee for successful conduct of a study. The EU pharmacovigilance legislation provides for a regulatory procedure to approve protocols of post-authorisation safety studies (PASS) imposed as a condition of marketing authorisation (EU RMP category 1 and 2 studies) before study start to ensure the objectives of the PASS are met and the protocol follows the format requirements. However, this needs to be distinguished from regulatory supervision of EU RMP category 3 studies where prior protocol approval is not required by law (but may be requested to the marketing authorisation holder voluntarily). The general statement above should be balanced, also in the discussion. Like for any other study, relevant scientific guidance should be considered by marketing authorisation holders and investigators for the development of study protocols, the conduct of studies and the writing of study reports (e.g. ENCePP Guide on Methodological Standards in Pharmacoepidemiology, Guidelines for Good Pharmacoepidemiology Practices of the International Society of Pharmacoepidemiology (ISPE GPP) etc.)

Response: We have added text in the discussion to acknowledge this.

Clarify if only studies requested by EMA or also by other EU or non-EU regulators have been included in the descriptive analysis. This is important for the discussion of the results on data sources.

Response: This was just EMA only.

The authors have chosen 3 distinct study scopes as inclusion criteria, but there might be other studies requested by a regulator which provide insight in trends in study design, data sources and therapeutic areas (a query performed on 27.09.2017 found more than 170 studies requested by a regulator with other scopes). The authors should explain this limitation.

Reponse: This limitation has been noted

Results

The presentation of the results on study status, data sources and disease areas may benefit from a breakdown by EU RMP category to show any potential association of with the timelines for conducting studies.

Response: Unfortunately we did not collect this data in our analysis but agree it would be of interest for future work.

The authors do not specify how the disease areas were defined (by ATC code?).

Response: The disease areas are listed in the EU PAS register. This has been clarified

Discussion

The statement that planned and ongoing studies were far more likely to include a safety element is not clear from the presented results; any study in an EU RMP is linked to a safety concern, only with varying scope i.e. effectiveness of measures implemented to manage risks (e.g. educational materials), changes in drug utilisation (e.g. restricted indication) or risk quantification (e.g. frequency of adverse reactions)?

Response: Risk assessment is meant here (as opposed to drug utilisation, for instance). The text has been updated.

A limitation of the study that may be worth explaining is that the registration of studies is voluntary (except for imposed PASS), and the entries are not verified as to their accuracy. For examples, it is not checked by ENCePP or EMA whether the RMP category entered is correct and corresponds to the outcome of the regulatory procedure. Therefore, there may be some misclassification. For PASS imposed by regulators, the RMP category field may have been incorrectly populated by the company.

Response: This limitation has been noted
We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

The EU PAS Register is accessible via the ENCePP website which is hosted and maintained by the European Medicines Agency (EMA).

Response: This has been clarified in the methods.

The term post-authorisation study (PAS) is not exclusive to observational research and may include interventional study designs (e.g. see definition of post-authorisation safety study in GVP VIII), this should be made clear as the introduction refers to historical ‘criticism’ raised mainly in context of non-interventional research. The majority of studies in the EU PAS Register are non-interventional but also a few RCTs fall under the inclusion criteria applied by the authors for their descriptive study.

Response: This point has been clarified in the introduction

The EU pharmacovigilance legislation and the requirement to publish PAS imposed as a condition of marketing authorisation in a public register came into force in July 2012, not in 2010 (the new legislation was however published in December 2010).

Response: Text corrected, thank you.

The authors should explain what they mean by ‘approved PAS’. Regulatory approval of PAS is not a guarantee for successful conduct of a study. The EU pharmacovigilance legislation provides for a regulatory procedure to approve protocols of post-authorisation safety studies (PASS) imposed as a condition of marketing authorisation (EU RMP category 1 and 2 studies) before study start to ensure the objectives of the PASS are met and the protocol follows the format requirements. However, this needs to be distinguished from regulatory supervision of EU RMP category 3 studies where prior protocol approval is not required by law (but may be requested to the marketing authorisation holder voluntarily). The general statement above should be balanced, also in the discussion. Like for any other study, relevant scientific guidance should be considered by marketing authorisation holders and investigators for the development of study protocols, the conduct of studies and the writing of study reports (e.g. ENCePP Guide on Methodological Standards in Pharmacoepidemiology, Guidelines for Good Pharmacoepidemiology Practices of the International Society of Pharmacoepidemiology (ISPE GPP) etc.)

Response: We have added text in the discussion to acknowledge this.

Clarify if only studies requested by EMA or also by other EU or non-EU regulators have been included in the descriptive analysis. This is important for the discussion of the results on data sources.

Response: This was just EMA only.

The authors have chosen 3 distinct study scopes as inclusion criteria, but there might be other studies requested by a regulator which provide insight in trends in study design, data sources and therapeutic areas (a query performed on 27.09.2017 found more than 170 studies requested by a regulator with other scopes). The authors should explain this limitation.

Reponse: This limitation has been noted

Results

The presentation of the results on study status, data sources and disease areas may benefit from a breakdown by EU RMP category to show any potential association of with the timelines for conducting studies.

Response: Unfortunately we did not collect this data in our analysis but agree it would be of interest for future work.

The authors do not specify how the disease areas were defined (by ATC code?).

Response: The disease areas are listed in the EU PAS register. This has been clarified

Discussion

The statement that planned and ongoing studies were far more likely to include a safety element is not clear from the presented results; any study in an EU RMP is linked to a safety concern, only with varying scope i.e. effectiveness of measures implemented to manage risks (e.g. educational materials), changes in drug utilisation (e.g. restricted indication) or risk quantification (e.g. frequency of adverse reactions)?

Response: Risk assessment is meant here (as opposed to drug utilisation, for instance). The text has been updated.

A limitation of the study that may be worth explaining is that the registration of studies is voluntary (except for imposed PASS), and the entries are not verified as to their accuracy. For examples, it is not checked by ENCePP or EMA whether the RMP category entered is correct and corresponds to the outcome of the regulatory procedure. Therefore, there may be some misclassification. For PASS imposed by regulators, the RMP category field may have been incorrectly populated by the company.

Response: This limitation has been noted
Competing Interests: RC, JCR and DL are employees of Evidera Inc. SR and LM are employees of Bristol-Myers Squibb Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 10 Nov 2017

Sreeram Ramagopalan, Centre for Observational Research and Data Sciences, Bristol-Myers Squibb, Uxbridge, UB8 1DH, UK

10 Nov 2017

Author Response
We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:
- The EU PAS Register is accessible via
... Continue reading
We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

The EU PAS Register is accessible via the ENCePP website which is hosted and maintained by the European Medicines Agency (EMA).

Response: This has been clarified in the methods.

The term post-authorisation study (PAS) is not exclusive to observational research and may include interventional study designs (e.g. see definition of post-authorisation safety study in GVP VIII), this should be made clear as the introduction refers to historical ‘criticism’ raised mainly in context of non-interventional research. The majority of studies in the EU PAS Register are non-interventional but also a few RCTs fall under the inclusion criteria applied by the authors for their descriptive study.

Response: This point has been clarified in the introduction

The EU pharmacovigilance legislation and the requirement to publish PAS imposed as a condition of marketing authorisation in a public register came into force in July 2012, not in 2010 (the new legislation was however published in December 2010).

Response: Text corrected, thank you.

The authors should explain what they mean by ‘approved PAS’. Regulatory approval of PAS is not a guarantee for successful conduct of a study. The EU pharmacovigilance legislation provides for a regulatory procedure to approve protocols of post-authorisation safety studies (PASS) imposed as a condition of marketing authorisation (EU RMP category 1 and 2 studies) before study start to ensure the objectives of the PASS are met and the protocol follows the format requirements. However, this needs to be distinguished from regulatory supervision of EU RMP category 3 studies where prior protocol approval is not required by law (but may be requested to the marketing authorisation holder voluntarily). The general statement above should be balanced, also in the discussion. Like for any other study, relevant scientific guidance should be considered by marketing authorisation holders and investigators for the development of study protocols, the conduct of studies and the writing of study reports (e.g. ENCePP Guide on Methodological Standards in Pharmacoepidemiology, Guidelines for Good Pharmacoepidemiology Practices of the International Society of Pharmacoepidemiology (ISPE GPP) etc.)

Response: We have added text in the discussion to acknowledge this.

Clarify if only studies requested by EMA or also by other EU or non-EU regulators have been included in the descriptive analysis. This is important for the discussion of the results on data sources.

Response: This was just EMA only.

The authors have chosen 3 distinct study scopes as inclusion criteria, but there might be other studies requested by a regulator which provide insight in trends in study design, data sources and therapeutic areas (a query performed on 27.09.2017 found more than 170 studies requested by a regulator with other scopes). The authors should explain this limitation.

Reponse: This limitation has been noted

Results

The presentation of the results on study status, data sources and disease areas may benefit from a breakdown by EU RMP category to show any potential association of with the timelines for conducting studies.

Response: Unfortunately we did not collect this data in our analysis but agree it would be of interest for future work.

The authors do not specify how the disease areas were defined (by ATC code?).

Response: The disease areas are listed in the EU PAS register. This has been clarified

Discussion

The statement that planned and ongoing studies were far more likely to include a safety element is not clear from the presented results; any study in an EU RMP is linked to a safety concern, only with varying scope i.e. effectiveness of measures implemented to manage risks (e.g. educational materials), changes in drug utilisation (e.g. restricted indication) or risk quantification (e.g. frequency of adverse reactions)?

Response: Risk assessment is meant here (as opposed to drug utilisation, for instance). The text has been updated.

A limitation of the study that may be worth explaining is that the registration of studies is voluntary (except for imposed PASS), and the entries are not verified as to their accuracy. For examples, it is not checked by ENCePP or EMA whether the RMP category entered is correct and corresponds to the outcome of the regulatory procedure. Therefore, there may be some misclassification. For PASS imposed by regulators, the RMP category field may have been incorrectly populated by the company.

Response: This limitation has been noted
We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

The EU PAS Register is accessible via the ENCePP website which is hosted and maintained by the European Medicines Agency (EMA).

Response: This has been clarified in the methods.

The term post-authorisation study (PAS) is not exclusive to observational research and may include interventional study designs (e.g. see definition of post-authorisation safety study in GVP VIII), this should be made clear as the introduction refers to historical ‘criticism’ raised mainly in context of non-interventional research. The majority of studies in the EU PAS Register are non-interventional but also a few RCTs fall under the inclusion criteria applied by the authors for their descriptive study.

Response: This point has been clarified in the introduction

The EU pharmacovigilance legislation and the requirement to publish PAS imposed as a condition of marketing authorisation in a public register came into force in July 2012, not in 2010 (the new legislation was however published in December 2010).

Response: Text corrected, thank you.

The authors should explain what they mean by ‘approved PAS’. Regulatory approval of PAS is not a guarantee for successful conduct of a study. The EU pharmacovigilance legislation provides for a regulatory procedure to approve protocols of post-authorisation safety studies (PASS) imposed as a condition of marketing authorisation (EU RMP category 1 and 2 studies) before study start to ensure the objectives of the PASS are met and the protocol follows the format requirements. However, this needs to be distinguished from regulatory supervision of EU RMP category 3 studies where prior protocol approval is not required by law (but may be requested to the marketing authorisation holder voluntarily). The general statement above should be balanced, also in the discussion. Like for any other study, relevant scientific guidance should be considered by marketing authorisation holders and investigators for the development of study protocols, the conduct of studies and the writing of study reports (e.g. ENCePP Guide on Methodological Standards in Pharmacoepidemiology, Guidelines for Good Pharmacoepidemiology Practices of the International Society of Pharmacoepidemiology (ISPE GPP) etc.)

Response: We have added text in the discussion to acknowledge this.

Clarify if only studies requested by EMA or also by other EU or non-EU regulators have been included in the descriptive analysis. This is important for the discussion of the results on data sources.

Response: This was just EMA only.

The authors have chosen 3 distinct study scopes as inclusion criteria, but there might be other studies requested by a regulator which provide insight in trends in study design, data sources and therapeutic areas (a query performed on 27.09.2017 found more than 170 studies requested by a regulator with other scopes). The authors should explain this limitation.

Reponse: This limitation has been noted

Results

The presentation of the results on study status, data sources and disease areas may benefit from a breakdown by EU RMP category to show any potential association of with the timelines for conducting studies.

Response: Unfortunately we did not collect this data in our analysis but agree it would be of interest for future work.

The authors do not specify how the disease areas were defined (by ATC code?).

Response: The disease areas are listed in the EU PAS register. This has been clarified

Discussion

The statement that planned and ongoing studies were far more likely to include a safety element is not clear from the presented results; any study in an EU RMP is linked to a safety concern, only with varying scope i.e. effectiveness of measures implemented to manage risks (e.g. educational materials), changes in drug utilisation (e.g. restricted indication) or risk quantification (e.g. frequency of adverse reactions)?

Response: Risk assessment is meant here (as opposed to drug utilisation, for instance). The text has been updated.

A limitation of the study that may be worth explaining is that the registration of studies is voluntary (except for imposed PASS), and the entries are not verified as to their accuracy. For examples, it is not checked by ENCePP or EMA whether the RMP category entered is correct and corresponds to the outcome of the regulatory procedure. Therefore, there may be some misclassification. For PASS imposed by regulators, the RMP category field may have been incorrectly populated by the company.

Response: This limitation has been noted
Competing Interests: RC, JCR and DL are employees of Evidera Inc. SR and LM are employees of Bristol-Myers Squibb Close
Report a concern

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 14 Aug 2017

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 2 (revision) 10 Nov 17	read	read
Version 1 14 Aug 17	read	read

Xavier Kurz, European Medicines Agency, London, UK

Thomas Goedecke, European Medicines Agency, London, UK
Lex Bouter, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

5 Views

20 Nov 2017 | for Version 2

Lex Bouter, VU University Medical Centre, Department of Epidemiology and Biostatistics, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

5 Views Cite this report Responses(0)

Approved

No checks are done for non‐publication, selective publication, and for other discrepancies between protocol and publication for the published studies.
There is no requirement for studies to have been entered on the register before data collection starts. Similarly later changes to the entry in the register are not identifyable as such.
The study did not attempt to quantify transparency nor review the methodological rigor of studies, but simply provides an overview of the study designs employed.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Epidemiology, methodology, research integrity

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

4 Views

13 Nov 2017 | for Version 2

Xavier Kurz, Pharmacovigilance and Epidemiology Department, European Medicines Agency, London, UK

Thomas Goedecke, Pharmacovigilance and Epidemiology Department, European Medicines Agency, London, UK

4 Views Cite this report Responses(0)

Approved

Most comments have been addressed.

Minor comments regarding the Introduction:

"GVP VIII": write in full and provide the link.
"as a condition for granting marketing authorisation": not only for granting, as PASS can be imposed post-authorisation; write: "as a condition of the marketing authorisation".
The EU PAS Register is hosted by ENCePP

Competing Interests

No competing interests were disclosed.

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

15 Views

23 Oct 2017 | for Version 1

Lex Bouter, VU University Medical Centre, Department of Epidemiology and Biostatistics, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

15 Views Cite this report Responses(1)

Approved With Reservations

The 314 included studies were on 18^th October 2016 contained in the ENCePP EU PAS Register AND concerned a PAS requested by EMA. That makes the reader wonder how many PAS requested by EMA (since a suitable date) were not contained in the Register. And also how many studies contained in the Register were not requested by EMA. In other words: how selective is the sample studied?
The conclusions come as a surprise because no explicit study questions or hypotheses were formulated. Furthermore, the conclusions are overstated in the sense that both the cross-sectional nature and the lack of a statistical (multivariable) data-analysis make a causal interpretation (‘influence’) unwarranted.
A serious study limitation is that no checks are done (for the completed studies) for non-publication, and for selective publication and other discrepancies between protocol and publication for the published studies.

Minor issues

Please explain better what is meant by PAS. Most readers will probably be more familiar with terms like ‘postmarketing surveillance’ and ‘phase IV trials’.
I assume that entries in the Registry need to be made before the start of data collection. And also that later changes (amendments) to the protocol cannot be made without leaving traces (log file). But this is not clearly explained in the current version of the article.
The Introduction section suggests that the European pharmacovigilance legislation was introduced to foster transparency and methodological quality of the Post-Authorization Studies. That may be correct, but the current study says very little on transparency and nothing about the methodological quality of PAS.
The first sentence of the Discussion section suggests that this article will ‘likely be used as reference for future PAS designs’. It completely escapes me why this would be so.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

No
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

No

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Epidemiology

Respond to this report

Responses (1)

Author Response

10 Nov 2017

Sreeram Ramagopalan, Centre for Observational Research and Data Sciences, Bristol-Myers Squibb, Uxbridge, UB8 1DH, UK

We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

Major issues
The 314 included studies were on 18^th October 2016 contained in the ENCePP EU PAS Register AND concerned a PAS requested by EMA. That makes the reader wonder how many PAS requested by EMA (since a suitable date) were not contained in the Register. And also how many studies contained in the Register were not requested by EMA. In other words: how selective is the sample studied?

Response: Looking at recent figures on the PAS register (06/11/17) 40% (482/1191) fit these criteria. The lack of generalisability of our findings to studies that are not requested by the EMA is now highlighted in the results

The conclusions come as a surprise because no explicit study questions or hypotheses were formulated. Furthermore, the conclusions are overstated in the sense that both the cross-sectional nature and the lack of a statistical (multivariable) data-analysis make a causal interpretation (‘influence’) unwarranted.

Response: We had not intended to claim causal inference, but simply describe associations/trends observed. We have removed wording relating to ‘influence’.

A serious study limitation is that no checks are done (for the completed studies) for non-publication, and for selective publication and other discrepancies between protocol and publication for the published studies.

Response: The reviewer highlights an important point, but one that is not covered by the aim of our study. We describe the design of studies on the PAS register, we are not addressing the question of whether researchers comply with their study protocols when publishing results (although this would be an interesting analysis!)

Minor issues

Please explain better what is meant by PAS. Most readers will probably be more familiar with terms like ‘postmarketing surveillance’ and ‘phase IV trials’.

Response: These terms have been added to the manuscript where the concept of a PAS is first introduced

I assume that entries in the Registry need to be made before the start of data collection. And also that later changes (amendments) to the protocol cannot be made without leaving traces (log file). But this is not clearly explained in the current version of the article.

Response: We are not aware of a requirement for studies to have been entered on the register before data collection starts. Amendments can be made but no record of these amendments is made

The Introduction section suggests that the European pharmacovig)ilance legislation was introduced to foster transparency and methodological quality of the Post-Authorization Studies. That may be correct, but the current study says very little on transparency and nothing about the methodological quality of PAS.

Response: We agree, our study did not attempt to quantify transparency nor review the methodological rigor of studies, simply provide a overview of the study designs employed.

The first sentence of the Discussion section suggests that this article will ‘likely be used as reference for future PAS designs’. It completely escapes me why this would be so.

Response: The results presented on study objectives/study design could inform readers on designs most commonly employed to address certain research questions, and therefore potentially inform suitable designs for future studies.

View more View less

Competing Interests

RC, JCR and DL are employees of Evidera Inc. SR and LM are employees of Bristol-Myers Squibb

Back to all reports

Reviewer Report

14 Views

28 Sep 2017 | for Version 1

Xavier Kurz, Pharmacovigilance and Epidemiology Department, European Medicines Agency, London, UK

Thomas Goedecke, Pharmacovigilance and Epidemiology Department, European Medicines Agency, London, UK

14 Views Cite this report Responses(1)

Approved With Reservations

The EU PAS Register is accessible via the ENCePP website which is hosted and maintained by the European Medicines Agency (EMA).
The term post-authorisation study (PAS) is not exclusive to observational research and may include interventional study designs (e.g. see definition of post-authorisation safety study in GVP VIII), this should be made clear as the introduction refers to historical ‘criticism’ raised mainly in context of non-interventional research. The majority of studies in the EU PAS Register are non-interventional but also a few RCTs fall under the inclusion criteria applied by the authors for their descriptive study.
The EU pharmacovigilance legislation and the requirement to publish PAS imposed as a condition of marketing authorisation in a public register came into force in July 2012, not in 2010 (the new legislation was however published in December 2010).
The authors should explain what they mean by ‘approved PAS’. Regulatory approval of PAS is not a guarantee for successful conduct of a study. The EU pharmacovigilance legislation provides for a regulatory procedure to approve protocols of post-authorisation safety studies (PASS) imposed as a condition of marketing authorisation (EU RMP category 1 and 2 studies) before study start to ensure the objectives of the PASS are met and the protocol follows the format requirements. However, this needs to be distinguished from regulatory supervision of EU RMP category 3 studies where prior protocol approval is not required by law (but may be requested to the marketing authorisation holder voluntarily). The general statement above should be balanced, also in the discussion. Like for any other study, relevant scientific guidance should be considered by marketing authorisation holders and investigators for the development of study protocols, the conduct of studies and the writing of study reports (e.g. ENCePP Guide on Methodological Standards in Pharmacoepidemiology, Guidelines for Good Pharmacoepidemiology Practices of the International Society of Pharmacoepidemiology (ISPE GPP) etc.).

Methods

Clarify if only studies requested by EMA or also by other EU or non-EU regulators have been included in the descriptive analysis. This is important for the discussion of the results on data sources.
The authors have chosen 3 distinct study scopes as inclusion criteria, but there might be other studies requested by a regulator which provide insight in trends in study design, data sources and therapeutic areas (a query performed on 27.09.2017 found more than 170 studies requested by a regulator with other scopes). The authors should explain this limitation.

Results

The presentation of the results on study status, data sources and disease areas may benefit from a breakdown by EU RMP category to show any potential association of with the timelines for conducting studies.
The authors do not specify how the disease areas were defined (by ATC code?).

Discussion

The statement that planned and ongoing studies were far more likely to include a safety element is not clear from the presented results; any study in an EU RMP is linked to a safety concern, only with varying scope i.e. effectiveness of measures implemented to manage risks (e.g. educational materials), changes in drug utilisation (e.g. restricted indication) or risk quantification (e.g. frequency of adverse reactions)?
A limitation of the study that may be worth explaining is that the registration of studies is voluntary (except for imposed PASS), and the entries are not verified as to their accuracy. For examples, it is not checked by ENCePP or EMA whether the RMP category entered is correct and corresponds to the outcome of the regulatory procedure. Therefore, there may be some misclassification. For PASS imposed by regulators, the RMP category field may have been incorrectly populated by the company.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

The reviewers are employees of the European Medicines Agency, which hosts the EU PAS Register.

Respond to this report

Responses (1)

Author Response

10 Nov 2017

Sreeram Ramagopalan, Centre for Observational Research and Data Sciences, Bristol-Myers Squibb, Uxbridge, UB8 1DH, UK

We thank the reviewers for their comments. We respond specifically to their points below and hope they find our changes acceptable:

The EU PAS Register is accessible via the ENCePP website which is hosted and maintained by the European Medicines Agency (EMA).

Response: This has been clarified in the methods.

The term post-authorisation study (PAS) is not exclusive to observational research and may include interventional study designs (e.g. see definition of post-authorisation safety study in GVP VIII), this should be made clear as the introduction refers to historical ‘criticism’ raised mainly in context of non-interventional research. The majority of studies in the EU PAS Register are non-interventional but also a few RCTs fall under the inclusion criteria applied by the authors for their descriptive study.

Response: This point has been clarified in the introduction

The EU pharmacovigilance legislation and the requirement to publish PAS imposed as a condition of marketing authorisation in a public register came into force in July 2012, not in 2010 (the new legislation was however published in December 2010).

Response: Text corrected, thank you.

The authors should explain what they mean by ‘approved PAS’. Regulatory approval of PAS is not a guarantee for successful conduct of a study. The EU pharmacovigilance legislation provides for a regulatory procedure to approve protocols of post-authorisation safety studies (PASS) imposed as a condition of marketing authorisation (EU RMP category 1 and 2 studies) before study start to ensure the objectives of the PASS are met and the protocol follows the format requirements. However, this needs to be distinguished from regulatory supervision of EU RMP category 3 studies where prior protocol approval is not required by law (but may be requested to the marketing authorisation holder voluntarily). The general statement above should be balanced, also in the discussion. Like for any other study, relevant scientific guidance should be considered by marketing authorisation holders and investigators for the development of study protocols, the conduct of studies and the writing of study reports (e.g. ENCePP Guide on Methodological Standards in Pharmacoepidemiology, Guidelines for Good Pharmacoepidemiology Practices of the International Society of Pharmacoepidemiology (ISPE GPP) etc.)

Response: We have added text in the discussion to acknowledge this.

Clarify if only studies requested by EMA or also by other EU or non-EU regulators have been included in the descriptive analysis. This is important for the discussion of the results on data sources.

Response: This was just EMA only.

The authors have chosen 3 distinct study scopes as inclusion criteria, but there might be other studies requested by a regulator which provide insight in trends in study design, data sources and therapeutic areas (a query performed on 27.09.2017 found more than 170 studies requested by a regulator with other scopes). The authors should explain this limitation.

Reponse: This limitation has been noted

Results

The presentation of the results on study status, data sources and disease areas may benefit from a breakdown by EU RMP category to show any potential association of with the timelines for conducting studies.

Response: Unfortunately we did not collect this data in our analysis but agree it would be of interest for future work.

The authors do not specify how the disease areas were defined (by ATC code?).

Response: The disease areas are listed in the EU PAS register. This has been clarified

Discussion

The statement that planned and ongoing studies were far more likely to include a safety element is not clear from the presented results; any study in an EU RMP is linked to a safety concern, only with varying scope i.e. effectiveness of measures implemented to manage risks (e.g. educational materials), changes in drug utilisation (e.g. restricted indication) or risk quantification (e.g. frequency of adverse reactions)?

Response: Risk assessment is meant here (as opposed to drug utilisation, for instance). The text has been updated.

A limitation of the study that may be worth explaining is that the registration of studies is voluntary (except for imposed PASS), and the entries are not verified as to their accuracy. For examples, it is not checked by ENCePP or EMA whether the RMP category entered is correct and corresponds to the outcome of the regulatory procedure. Therefore, there may be some misclassification. For PASS imposed by regulators, the RMP category field may have been incorrectly populated by the company.

Response: This limitation has been noted

View more View less

Competing Interests

RC, JCR and DL are employees of Evidera Inc. SR and LM are employees of Bristol-Myers Squibb

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

Click here to access the data.

Downloaded data do not display as expected? Download the data

[1] 1. Kalata P, Martus P, Zettl H, et al.: Differences between clinical trial participants and patients in a population-based registry: the German Rectal Cancer Study vs. the Rostock Cancer Registry. Dis Colon Rectum. 2009; 52(3): 425–437. PubMed Abstract | Publisher Full Text

[2] 2. Maasland L, Van Oostenbrugge RJ, Franke CF, et al.: Patients enrolled in large randomized clinical trials of antiplatelet treatment for prevention after transient ischemic attack or ischemic stroke are not representative of patients in clinical practice: the Netherlands Stroke Survey. Stroke. 2009; 40(8): 2662–2668. PubMed Abstract | Publisher Full Text

[3] 3. McDonald L, Lambrelli D, Wasiak R, et al.: Real-world data in the United Kingdom: opportunities and challenges. BMC Med. 2016; 14(1): 97. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Gale EA: Post-marketing studies of new insulins: sales or science? BMJ. 2012; 344: e3974. PubMed Abstract | Publisher Full Text

[5] 5. Borg JJ, Aislaitner G, Pirozynski M, et al.: Strengthening and rationalizing pharmacovigilance in the EU: where is Europe heading to? A review of the new EU legislation on pharmacovigilance. Drug Saf. 2011; 34(3): 187–197. PubMed Abstract | Publisher Full Text

[6] 6. Blake KV, Prilla S, Accadebled S, et al.: European Medicines Agency review of post-authorisation studies with implications for the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance. Pharmacoepidemiol Drug Saf. 2011; 20(10): 1021–9. PubMed Abstract | Publisher Full Text

[7] 7. Engel P, Almas MF, De Bruin ML, et al.: Lessons learned on the design and the conduct of Post-Authorization Safety Studies: review of 3 years of PRAC oversight. Br J Clin Pharmacol. 2017; 83(4): 884–893. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Hoekman J, Klamer TT, Mantel-Teeuwisse AK, et al.: Characteristics and follow-up of postmarketing studies of conditionally authorized medicines in the EU. Br J Clin Pharmacol. 2016; 82(1): 213–26. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Ramagopalan S, McDonald L, Carroll R, et al.: Dataset 1 in: An Analysis of Characteristics of Post-Authorisation Studies Registered on the ENCePP EU PAS Register. F1000Research. 2017. Data Source

An analysis of characteristics of post-authorisation studies registered on the ENCePP EU PAS Register

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Methods

Results

Table 1. Type of EMA requested PAS registered in ENCePP by 18th October 2016 by study status (finalized, ongoing or planned).

Study scope and design

Secondary data capture studies and data sources used

Disease area and study design

Discussion

Data availability

Competing interests

Grant information

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

The problem

How to fix it

Competing Interests Policy

Stay Updated

Table 1. Type of EMA requested PAS registered in ENCePP by 18^th October 2016 by study status (finalized, ongoing or planned).