Comments about the appraisal of systematic reviews in restorative dentistry

Adequate adoption of evidence-based practice is deeply rooted in accessing methodological quality and completeness of systematic reviews and meta-analyses reporting. Nonetheless, this assumption might be flawed if the methodological quality assessment has not been properly conducted. Taking the former statement into consideration, this correspondence article encourages the improvement of future tertiary manuscripts, especially in the field of restorative dentistry. Thus, this article addresses an overview of reviews in restorative dentistry as an example of evaluating tertiary evidence for increasing the awareness of reviewers, authors, and readers.


Amendments from Version 2
The text has been revised to make it sound more neutral. Also, some orthographic errors have been amended.
We have included mention of Embase and Medline to support this correspondence article.
The reference list has been updated to show the breadth and depth of the subject and the expertise of the authors in the topic related to research synthesis, which includes systematic review and umbrella review methods.
Dr. Musab Hamed Saeed, Associate Professor at Ajman University, UAE, contributed to the Funding Acquisition, Project Administration, Resources, and Writing -Review & Editing of this new version. Thus, he has been added to the author list; the article text has been modified to acknowledge the plurality of the authors throughout this new version.
"Vicerrectorado de Investigación, Universidad Católica San Antonio de Murcia, Murcia, 30107, Spain" has been removed from the affiliations because the work conducted during this study was not associated with this institute.

REVISED
Comments about the appraisal of systematic reviews in restorative dentistry

Dear respectable Advisory Editors and readers:
The publication by Sarkis-Onofre et al. 1 "Systematic reviews in restorative dentistry: discussing relevant aspects," in the Journal of Esthetic and Restorative Dentistry in the May 2019 issue was read with great and particular interest. This well-written overview of reviews or systematic review of systematic reviews stated that "This study was not registered in PROSPERO" since PROSPERO indicates that "Reviews of methodological issues need to contain at least one outcome of direct patient or clinical relevance in order to be included in PROSPERO." Interestingly, despite the fact that the above referenced tertiary study falls in the PROSPERO's review of reviews category, it was neglected to being classified as such. Additionally, if PROSPERO was not an option, the a priori protocol could have been easily registered in another repository. Therefore, the authors' arguments not to register their protocol in PROSPERO are not valid.
Moreover, the authors 1 mentioned that the previous version of their review 2 has a protocol available upon request. However, their first paper 2 , which is in a Brazilian University Magazine printed in Portuguese language, does not support the updated version of the review properly since their first version does not consider any protocol in the text.
Their critical appraisal using AMSTAR-2 appears in Table 2.
Five out of the 16 included review studies in their review of reviews, had between one to four AMSTAR-2 items referred as "Authors reported different information by e-mail however, it was not found in the article." This reporting method may not be the most scholarly or safest to present their findings, especially when the authors of their included review studies kindly accepted to provide further clarification about their methodology.
Particularly, the first author of Afrashtehfar et al. 3  Additionally, the review hand searched eight journals and also screened manually in the reference list of all identified relevant primary studies and related secondary studies 3 . Next, the list of excluded primary studies and justifications (item 7) were provided in Supplemental Table 6 (i.e., full-text excluded articles and reasons for exclusion). Regarding any potential sources of conflict (item 16), it is well-stated on the first page of the review that this study was "Supported in part by a Knowledge Transfer Grant from the Network for Oral and Bone Health Research." Additionally, there is a section at the end of the paper for Acknowledgements where librarians and statisticians were thanked for their consultancy 3 .
Moreover, the search and eligibility criteria for Sarkis-Onofre et al. 1 were systematic reviews that met PRISMA-P, including adults over 18 years of age with direct composite resin restoration in posterior teeth compared with other materials or techniques used in posterior teeth regardless of the outcome up to October 15 th , 2017. However, some articles that fully suited their inclusion criteria were not included. For example, Afrashtehfar et al. 5 "Failure of single-unit restorations on root filled posterior teeth: a systematic review" was not included despite being available from November 21 st , 2016. Therefore, their search strategy and their search conduct (including the elimination of duplicates) 6 , as well as screening 7 , raise some serious methodological concerns 8 .
Their overview of reviews has a collaboration with well-known evidence-based medicine experts from Canada, Tricco and Moher 9 , which usually rely on the talent from their research team for screening and assessing the literature.
After a brief analysis, this letter encourages the improvement of future synthesis of studies including their quality assessment to: ▪ Address clarification with authors of potentially included studies safely and respectfully to avoid accusation, especially if there is no consensus on the matter from different experts (i.e., two experts as a minimum) 10-15 .
▪ Take the time and effort necessary to assess the review paper of interventions according to AMSTAR-2 4 . At least two experts in the field should also determine this instead of two research trainees.
▪ Spend sufficient time with expert librarians to develop an adequate search strategy in multiple databases 3,5 .
▪ Use a reference manager and do not rely on removing the duplicates by selecting only one category (i.e., authors' names). Thus, the available categories should be combined to avoid removing records that may share the same publication journal, year, or authors.
▪ A PRISMA checklist 16,17 should be submitted, reporting compliance with each item by indicating the paragraph and page where they can be identified in the review/ appraisal. All the required reporting should be included in the quality assessment to ensure transparency and validity.

Data availability
Underlying data No data are associated with this article.

Spyridon N. Papageorgiou
Clinic of Orthodontics and Pediatric Dentistry, Center of Dental Medicine, University of Zurich, Zurich, Switzerland I read with interest the abovementioned submission. However, I must say that I am somewhat confused since the author used this piece to comment on inadequacies, as deemed by the author, of a JERD-published overview of reviews that also included one of his SRs. In essence, this piece is not thought of as a standalone paper but is rather aimed at the already published overview like a correspondence with points to be addressed by the authors of the original overview in JERD. Therefore, I believe this would be more appropriate to be submitted as such in JERD and not here (not to mention that it is significantly more probable that it will be read and will be replied upon by the overview authors). Thanks for letting me see this.
Bear in mind that methodological quality is different from the risk of bias. I'm not sure which of the two you are implying here, but it makes sense that it is the latter.

2.
"The improvement of future methodological quality assessment manuscripts": this needs to be reformulated more succinctly. increasing the awareness of reviewers, authors, and readers." Instead of relaying what this piece does, better relay its' scope/aim and its final recommendations.
Text. The cited paper is neither "methodological quality assessment of systematic reviews" (methodological quality assessment is a procedure) nor a "systematic review of systematic reviews"-a systematic review addresses a clinically-based research question, ideally based on PICO. The cited paper is an overview of systematic reviews.

5.
The point about lack of registration in PROSPERO is a valid one, according to Sarkis-Onofre et al., since it complies with the instructions of PROSPERO. However, all studies that have an a priori protocol can register it (and benefit from this) and many different appropriate repositories exist. Likewise, I'm not sure what is meant with the 3 rd paragraph. Information provided through e-mails might not necessarily be regarding the same way as published information, but can still be used in a scientific publication, if deemed appropriate.

8.
The paragraph about the Sarkis-Onofre et al. authors wrongly judging some AMSTAR 2 fields of your SR seem to be valid, however.

9.
Likewise, the comment about potentially missed SRs to include seems to be valid. 10.
The last bullet point might not necessarily be correct, since PRISMA is meant for systematic review-it is not a reporting guideline for overviews of review (though some fields of it might be adopted with modifications).

Is the conclusion balanced and justified on the basis of the presented arguments? No
Competing Interests: No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.
Author Response 16 Nov 2021 Kelvin Afrashtehfar, Ajman University, Ajman, United Arab Emirates Dr. Papageorgiou is thanked for his contribution after the correspondence author suggested him as a reviewer.
The correspondence author is glad to answer his comments and clear any confusion Dr. Papageorgiou has referred to.
The overview authors will be invited to reply to the correspondence author comment upon completion. As the reviewer may know, some dental journals do not accept letters to the Editor. JERD does not accept replica or letters to the Editor, whereas F1000Research provides the opportunity to have this correspondence open to the public free of charge. Thus, the latter is a different option and could be considered, by some researchers, an ideal journal to allocate the correspondence comment.
Reply to the main specific comments: The correspondence author agrees with Dr. Papageorgiou that methodological quality assessment is not the same as the risk of bias. However, the correspondence author must say that an "overview of systematic reviews" is the same as a "systematic review of systematic reviews." Kindly refer to Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter V: Overviews of Reviews. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from www.training.cochrane.org/handbook. The correspondence author does understand that PROSPERO is probably the most popular repository among many other available options. Indeed, the correspondence author did refer to PROSPERO as the cited paper did. In fact, the correspondence author quoted their statement. The correspondence author appreciates that Dr. Papageorgiou found valid the point concerning the lack of PROSPERO registration.
The point of mentioning PROSPERO is supported by AMSTAR-2, which considers "protocol registered before start of review" as a critical domain. Please refer to Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or nonrandomised studies of healthcare interventions, or both. BMJ 2017; 358: j4008. Thus, one would expect that an overview of reviews (or systematic review of systematic reviews) should avoid missing such critical domain that they have assessed in their study where they used systematic reviews as the analytic unit.
The correspondence author thanks Dr. Papageorgiou for all the other comments that he found valid. For instance, the overview is missing to include SRs, review authors wrongly judging some AMSTAR-2 fields of a previous SR by the correspondence author, and lack of repository registry.
The criticisms that are posed in the article, we believe are valid and well structured. The bibliography that underlies them is thick and published in high-level journals. We, therefore, consider this article suitable for indexing, sure of the fact that it can be an added value for the quality of this type of research.

Is the rationale for commenting on the previous publication clearly described? Yes
Are any opinions stated well-argued, clear and cogent? Yes This comment from Afrashtehfar refers to an overview by Sarkis-Onofre et al., 2019 1 , published in Esthet Restor Dent, which discusses key aspects of systematic reviews in restorative dentistry, focusing on the improvement of the conduct and reporting of these reviews. The author of the comment identifies specific shortcomings of the Sarkis-Onofre et al. overview and formulates some suggestions to improve future methodological quality assessment manuscripts.
As a general thought, this comment would surely be better suited to be published in the original journal, in order to reach the same audience that read the original paper.
The author of this comment is also an author of a review included in Sarkis-Onofre et al. and illustrates his criticisms of their paper with the assessments made to his review. Given this clear conflict of interest, he should be careful in formulating subjective assessments such as "Perhaps Sarkis-Onofre et al. should have dedicated more time to conduct an adequate assessment", or using the terms 'safe' or 'respectful'.
The author criticises the methods applied by Sarkis-Onofre et al., particularly that a single researcher conducted the study selection and AMSTAR-2 assessment. These criticisms are based on perceived errors in the AMSTAR-2 assessments made to his own review included in Sarkis-Onofre et al., and on the failure to identify and include in the overview another work he authored (Afrashtehfar et al., 2016 2 ). While these are undeniable facts and the methods applied in the overview were subpar, it's not clear whether the errors identified are the exception or the norm, nor their impact on the overview conclusions.
He also points out the need for more transparency in the reporting of AMSTAR-2 assessments whenever these challenge personal communications with the original study authors. I fully support the idea to present in an appendix the clarifications about methodology provided in personal communications and the reasons for Sarkis-Onofre et al. to accept or disregard them in each case.
However, there are two instances of unjustified statements made in the comment. First, the author challenges the justification of Sarkis-Onofre et al. for not registering their methodological overview in PROSPERO, arguing that as an overview it could have been registered. However, this fails to acknowledge that methodological overviews that provide no data on patient-relevant outcomes don't verify PROSPERO registration criteria for methodological reviews.
Second, the author assumes that the overview, which updates a previously published overview from different authors, should follow the original overview protocol. But, while any review must have a pre-specified protocol, it's not necessary for an update to follow the protocol of the original review, as that protocol may be inexistent or outdated. What is important is that Sarkis-Onofre et al. do have a protocol available at request.

The benefits of publishing with F1000Research:
Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com