F1000Research F1000Research 2046-1402 F1000 Research Limited London, UK 10.12688/f1000research.125531.1 Correspondence Articles Challenges in specifying parameter values for COVID-19 simulation models

[version 1; peer review: 1 approved, 1 approved with reservations]

 Endo Akira Conceptualization Investigation Writing – Original Draft Preparation https://orcid.org/0000-0001-6377-7296 a 1 2 3 Nishi Akihiro Investigation Supervision Writing – Review & Editing 4 Department of Infectious Disease Epidemiology, London School of Hygiene & Tropical Medicine, London, WC1E 7HT, UK The Centre for Mathematical Modelling of Infectious Diseases, London School of Hygiene & Tropical Medicine, London, WC1E 7HT, UK School of Tropical Medicine and Global Health, Nagasaki University, Nagasaki, 852-8523, Japan Department of Epidemiology, Fielding School of Public Health, University of California, Los Angeles, Los Angeles, CA, 90095, USA aendo@nus.edu.sg

Competing interests: AE received a research grant from Taisho Pharmaceutical Co., Ltd. AN is a consultant to Vacan, Inc.

 21 9 2022 2022 11 1076 14 9 2022 Copyright: © 2022 Endo A and Nishi A 2022 This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

A recent modelling paper on the coronavirus disease 2019 (COVID-19) epidemic in the US (Bartsch et al.) suggested that maintaining face mask use until a high vaccine coverage (70–90%) is achieved is generally cost-effective or even cost-saving in many of the scenarios considered. Their conclusion was based on the assumed effectiveness of continued face mask use, cited from a study that reported an 18% reduction in the effective reproduction number associated with the introduction of state-level mask mandate policies in the US in the summer of 2020. However, using this value implicitly assumes that the effect of face mask use in 2021 through 2022 is the same as that of summer 2020, when stringent nonpharmaceutical interventions were in place. The effectiveness of universal mask wearing in 2021–2022 is probably more uncertain than considered in Bartsch et al. and rigorous sensitivity analysis on this parameter is warranted.

 COVID-19 mathematical models simulation mask mandates cost effectiveness National Institutes of Health K01AI166347 Japan Society for the Promotion of Science KAKENHI22K17329;JSPSOverseasResearchFellowships AE is supported by JSPS KAKENHI (22K17329) and JSPS Overseas Research Fellowships. AN and AE are supported by the National Institutes of Health under award number K01AI166347. The content is solely the responsibility of the authors and does not necessarily represent the official views of the JSPS or the NIH. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. <p>In a recent paper in <italic toggle="yes">Lancet Public Health</italic>, Bartsch <italic toggle="yes">et al.</italic> <sup> <xref ref-type="bibr" rid="ref1">1</xref> </sup> used an age-stratified transmission model to simulate the coronavirus disease 2019 (COVID-19) epidemic in the US and predicted the cost-effectiveness of maintaining face mask use until a high vaccine coverage (70–90%) is achieved. Their simulations showed that continued face mask use is generally cost-effective and even cost-saving in many of the scenarios considered. Such model-based economic analyses along with epidemiological evidence have the potential to guide policymakers in a timely manner.</p> <p>One of the biggest challenges in modelling studies is how to reliably choose parameter inputs as their misspecifications can substantially affect the conclusions. <sup> <xref ref-type="bibr" rid="ref2">2</xref> </sup> Bartsch <italic toggle="yes">et al.</italic> <sup> <xref ref-type="bibr" rid="ref1">1</xref> </sup> chose over 80 parameter inputs in their model, one of which represented the effectiveness of continued face mask use. The authors referred to a study that analysed temporal changes in the effective reproduction number ( <italic toggle="yes">R <sub>t</sub> </italic>) around the introduction of state-level mask mandate policies in the US in the summer of 2020 to find an 18% reduction in <italic toggle="yes">R <sub>t</sub> </italic> associated with the policies. <sup> <xref ref-type="bibr" rid="ref3">3</xref> </sup> Bartsch <italic toggle="yes">et al.</italic> chose this 18% for their effectiveness parameter; however, we need to be careful because this choice implicitly produces an assumption: the effect of face mask use in 2021 through 2022 is the same as that in summer 2020, when stringent interventions including a stay-at-home order and school closure were in place.</p> <p>This assumption may need to be revisited. COVID-19 frequently spreads over social contacts in settings where people do not wear masks, e.g. dining at restaurants, drinking at bars and social gathering with friends and relatives, <sup> <xref ref-type="bibr" rid="ref4">4</xref> </sup> and the stringent interventions in 2020 aimed to restrict contacts in these settings for outbreak control. <sup> <xref ref-type="bibr" rid="ref5">5</xref> </sup> With an increased proportion of contacts in these settings after the lifting of restrictions, public space mask mandates alone may not be able to easily achieve an equivalent <italic toggle="yes">R <sub>t</sub> </italic> reduction of 18%.</p> <p>Conversely, there are also changes from summer 2020 that likely favour the effect of facial mask use, e.g. <ext-link ext-link-type="uri" xlink:href="https://www.reuters.com/world/us/us-make-400-million-n95-masks-available-free-fight-covid-19-pandemic-official-2022-01-19/">improved supply of better-quality masks</ext-link>. <sup> <xref ref-type="bibr" rid="ref6">6</xref> </sup> In sum, the mask effectiveness in 2021–2022 is more uncertain than considered in Bartsch <italic toggle="yes">et al</italic>. Rigorous sensitivity analysis on this parameter (e.g. between 5% and 50%) is warranted to provide a balanced view on this important policy question.</p> </sec> <sec id="sec1"> <title>Data availability

No data are associated with this article.

 References Bartsch SM O’Shea KJ Chin KL : Maintaining face mask use before and after achieving different COVID-19 vaccination coverage levels: a modelling study. Lancet Public Health. 2022;7:e356–e365. 10.1016/S2468-2667(22)00040-8 Eriksson H Morin M Ekberg J : Assumptions Management in Simulation of Infectious Disease Outbreaks. AMIA Annu. Symp. Proc. 2009;2009:173–177. Yang B Huang AT Garcia-Carreras B : Effect of specific non-pharmaceutical intervention policies on SARS-CoV-2 transmission in the counties of the United States. Nat. Commun. 2021;12:3560. 34117244 10.1038/s41467-021-23865-8 Leclerc QJ Fuller NM Knight LE : What settings have been linked to SARS-CoV-2 transmission clusters? Wellcome Open Res. 2020;5:83. 32656368 10.12688/wellcomeopenres.15889.2 Smith M Ponce-de-Leon M Valencia A : Evaluating the policy of closing bars and restaurants in Cataluña and its effects on mobility and COVID19 incidence. Sci. Rep. 2022;12:9132. 35672332 10.1038/s41598-022-11531-y Mason J Baertlein L : U.S. to distribute 400 million free N95 masks at CVS, Walgreens in COVID fight. Reuters;2022; published online Jan 19. (accessed April 5, 2022). Reference Source 10.5256/f1000research.137847.r191697 Reviewer response for version 1 Herrera-Diestra José L 1 Referee The University of Texas at Austin, Austin, Texas, USA

Competing interests: No competing interests were disclosed.

 7 9 2023 Copyright: © 2023 Herrera-Diestra JL 2023 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve

I consider that the case made by the authors in this correspondence are valid and important. Changes in the conditions that lead to the 18% reduction of Rt are certainly a combination of all measures implemented in 2020, and may not be directly applicable in 2021-2022. I agree that a "rigorous sensitivity analysis" might be a good starting point. However, besides this sensitivity analysis, more elaborated methods need to be developed to assess more accurately the influence of the different interventions that were in play in the summer of 2020, and which of these interventions could be reasonably extrapolated to 2021-2022.

Are arguments sufficiently supported by evidence from the published literature or by new data and results?

Partly

Is the conclusion balanced and justified on the basis of the presented arguments?

Yes

Is the rationale for commenting on the previous publication clearly described?

Yes

Are any opinions stated well-argued, clear and cogent?

Yes

Reviewer Expertise:

Epidemiology, surveillance strategies, contact network epidemiology, complex systems, complex networks.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

 10.5256/f1000research.137847.r191712 Reviewer response for version 1 Gurbaxani Brian M 1 2 Referee Departments of Electrical and Computer Engineering and Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, USA NCIRD, Centers for Disease Control and Prevention, Atlanta, GA, DeKalb, USA

Competing interests: No competing interests were disclosed.

 23 8 2023 Copyright: © 2023 Gurbaxani BM 2023 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

The authors take issue with the fixed, 18% efficacy figure for face masks in the economic evaluation of masks usage post-vaccination paper by Bartsch et al., and of course they are correct: the efficacy isn’t fixed, and it depends on a lot of factors. So the question is: if face mask impact on Rt is a function of 1) social behaviour (e.g. contact rates), 2) quality and quantity of face mask usage, and 3) intrinsic properties of the viral variant circulating (R0) 1 , and you’re trying to quantify the economic impact of maintaining facemask use during and after a vaccine campaign using a calibration of facemask impact on Rt from an earlier time when all 3 of those factors might be different, then couldn’t your economic impact assessment be off? Yes, it could.

I’m not sure that the author’s suggestion of simply widening the uncertainty in the parameter value from 5 to 50% and doing a sensitivity analysis is going to do much good, however, because it won’t answer the policy questions people have, and will leave everyone more uncertain. I think it is possible, through modeling, to recalibrate the impact of facemasks on Rt for more recent times, when better quality masks are more widely available, but the variants are more easily transmissible as well, and society has less of a pandemic, lockdown mentality 1 , 2 . One could then present the results of different time periods corresponding to the spread of different variants, but with more certainty, and let the reader decide which scenario is more likely.

Are arguments sufficiently supported by evidence from the published literature or by new data and results?

Partly

Is the conclusion balanced and justified on the basis of the presented arguments?

Yes

Is the rationale for commenting on the previous publication clearly described?

Yes

Are any opinions stated well-argued, clear and cogent?

Yes

Reviewer Expertise:

Computational biology, statistics, modeling, bioinformatics, systems engineering and operations research, immunology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

References : Evaluation of different types of face masks to limit the spread of SARS-CoV-2: a modeling study. Sci Rep .2022;12(1) : 10.1038/s41598-022-11934-x 8630 35606393 10.1038/s41598-022-11934-x : Risk estimation of lifted mask mandates and emerging variants using mathematical model. Heliyon .2023;9(6) : 10.1016/j.heliyon.2023.e16841 e16841 37303548 10.1016/j.heliyon.2023.e16841