Drug effectiveness for COVID-19 inpatients inferred from Japanese medical claim data using propensity score matching

Shingo Mitsushima; Hiromasa Horiguchi; Kiyosu Taniguchi

doi:10.12688/f1000research.131102.1

Home Browse Drug effectiveness for COVID-19 inpatients inferred from Japanese...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Drug effectiveness for COVID-19 inpatients inferred from Japanese medical claim data using propensity score matching

[version 1; peer review: 1 approved with reservations]

Shingo Mitsushima ¹, Hiromasa Horiguchi², Kiyosu Taniguchi^3,4

PUBLISHED 13 Apr 2023

Author details Author details

¹ Center for Field Epidemic Intelligence, Research and Professional Development, National Institute of Infectious Diseases, Shinjuku-ku, Tokyo, 1620052, Japan
² Department of Clinical Data Management and Research, Clinical Research Center, National Hospital Organization, Meguro-ku, Tokyo, 1528621, Japan
³ General Director, National Hospital Organization Mie National Hospital, Tsu, Mie, 5140125, Japan
⁴ Research Director, The Tokyo Foundation for Policy Research, Minato-ku, Tokyo, 1066234, Japan

Shingo Mitsushima
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Hiromasa Horiguchi
Roles: Conceptualization, Data Curation, Funding Acquisition, Resources, Software, Supervision, Validation, Writing – Review & Editing

Kiyosu Taniguchi
Roles: Conceptualization, Data Curation, Funding Acquisition, Methodology, Project Administration, Resources, Supervision, Validation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Emerging Diseases and Outbreaks gateway.

Abstract

Background: Earlier studies and clinical trials have shown that the drugs such as antiviral drugs, antibody cocktails, and steroids and anti-inflammatory drugs are expected to prevent severe coronavirus 2019 (COVID-19) outcomes and death.

Methods: We used observational data for Japan to assess the effectiveness of these drugs for COVID-19. We applied propensity scoring, which can treat the choice of administered drug as a random assignment to inpatients, to the Medical Information Analysis Databank operated by National Hospital Organization in Japan. The outcome was defined as mortality. Subjects were all inpatients, inpatients with oxygen administration, and inpatients using respiratory ventilators, classified by three age classes: all ages, 65 years old or older, and younger than 65 years old. Information about demographical characteristics, underlying disease, administered drug, the proportion of Alpha, Beta and Omicron variant strains, and vaccine coverage were used as explanatory variable in logistic regression.

Results: Estimated results indicated that only an antibody cocktail (sotrovimab, casirivimab and imdevimab) raised the probability to save life consistently. By contrast, other drugs might reduce the probability of saving life. The results indicated that an antiviral drug (remdesivir), a steroid (dexamethasone), and an anti-inflammatory drug (baricitinib and tocilizumab) might not contribute to saving life even at the pseudo-situation of random assignment. However, this logistic regression at the first step might have only insufficient explanatory power.

Conclusions: We found a high likelihood that antibody cocktails were consistently effective to raise the probability of saving life, though a lesser likelihood in other drugs for older patients with mild to severe severity and all age patients with moderate severity.

Keywords

COVID-19, mutated strain, underlying diseases, antibody cocktail, antiviral drug, mortality, steroid and anti-inflammatory drug

Corresponding author: Shingo Mitsushima

Competing interests: No competing interests were disclosed.

Grant information: This work was supported by the Ministry of Health, Labour, and Welfare [grant number 20HA1005].
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2023 Mitsushima S et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Mitsushima S, Horiguchi H and Taniguchi K. Drug effectiveness for COVID-19 inpatients inferred from Japanese medical claim data using propensity score matching [version 1; peer review: 1 approved with reservations]. F1000Research 2023, 12:398 (https://doi.org/10.12688/f1000research.131102.1) First published: 13 Apr 2023, 12:398 (https://doi.org/10.12688/f1000research.131102.1) Latest published: 22 Jan 2024, 12:398 (https://doi.org/10.12688/f1000research.131102.2)

Introduction

Coronavirus disease 2019 (COVID-19) is caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). COVID-19 caused pandemic throughout the world from December 2019. In Japan, the first case of COVID-19 was detected in January 2020, and then COVID-19 spread throughout Japan. As characteristics of COVID-19 were revealed and treatments for COVID-19 developed, the mortality rate due to COVID-19 in Japan has declined.¹

Earlier studies and clinical trials have shown that drugs used against COVID-19, such as antiviral drugs (remdesivir),² antibody cocktails (casirivimab/imdevimab and sotrovimab),³^,⁴ steroids (dexamethasone),⁵ and anti-inflammatory drugs (baricitinib and tocilizumab),⁶^–⁸ statins,⁹^,¹⁰ RNA-dependent RNA polymerase inhibitor (molnupiravir),¹¹^,¹² protease inhibitor (nirmatrelvir/ritonavir),¹¹^,¹³ are expected to prevent severe COVID-19 outcomes and death. For this study, we considered the real-world effectiveness of drugs against COVID-19.

The National Hospital Organization (NHO) in Japan, an organization of regional core hospitals accounting for about 3.4% of all beds in Japan,¹⁴ provides a database of medical claims from 60 representative NHO hospitals: the Medical Information Analysis Databank (MIA). NHO is one of the largest organizations in Japan and each prefecture has one or more NHO hospitals. MIA collects the data on medical insurance claims for outpatients and inpatients. It includes inpatients’ demographical characteristics, underlying diseases, medical interventions including oxygen administration, the use of respiratory ventilators, administration of drugs, and outcomes such as discharge or death.¹⁵ We used MIA data of inpatients to verify drug effectiveness. MIA does not have the data on vaccination history or causative strain.

In the real world, whether a drug is administered or not depends on the patient condition. In other words, administration of drugs is probably not a random assignment. Generally speaking, patients who have a higher likelihood of developing an illness have a higher probability of being administered drugs, which implies, because of selection bias in the observational data, a negative association between drug administration and outcomes. Therefore, non-random choice of drug administration should be taken into account to estimate the drug effectiveness. Propensity score matching might resolve this problem statistically but not experimentally.¹⁶ This procedure predicts the likelihood of receiving drugs initially. Then outcomes are compared between a drug-received group and a drug not-administered group, the members of which have an almost identical likelihood of having been administered some drugs. We then examined drug effectiveness using this method.

Propensity score matching is widely used in social sciences for the evaluation of programs: participants choose to join a program spontaneously, such as a job training program or an unemployment payment program. However, in the natural sciences, a researcher can perform experiments and thereafter delete selection bias in the choice of subjects. Nevertheless, such experiments are expensive and require a longer period. Moreover, the number of subjects should be constrained; outcomes tend to be evaluated under less severe condition.¹⁷^,¹⁸

In medicine, random assignment experiments after launching are not so popular. Therefore, situations change, mutated strains emerge, vaccine coverage and/or developing treatment might affect drug effectiveness, making it impossible to evaluate. In particular, mortality tends not to be used as an outcome for evaluation, even though it should be the endpoint of greatest concern. Therefore, experiments in medicine for updated situations and mortality might be difficult. In other words, propensity score matching methods might be useful even for medicine, particularly for orthopaedic surgery and cardiovascular research.¹⁷^,¹⁸

Method

Data sources

This study used MIA for confirmed inpatients including their age, sex, underlying diseases, hospitalization date, administered drug, outcome and whether they received oxygen therapy and/or ventilation.

We used data for vaccine administration published by the Cabinet Secretariat. Moreover, prevalence in Alpha, Delta and Omicron variant strains were referred from a monitoring meeting in Tokyo because MIA included no information about the patient’s vaccine status and sublineage in SARS-CoV-2.

The study period was January 2020 through March 2022, using data recorded as of May 2022. This study area was the entirety of Japan.

Definitions of variables

We defined demographical conditions as age and sex, underlying diseases as cancer (C00–C90 in ICD10), asthma (J45), chronic obstructive pulmonary disease (COPD) (J44), hypertension (HT) (I10), heart failure (HF) (I50), and diabetes mellitus (DM) (E10). We examined the effects of an antiviral drug (remdesivir),² an antibody cocktail (sotrovimab and casirivimab/imdevimab),³^,⁴ a steroid (dexamethasone),⁵ and anti-inflammatory drugs (baricitinib and tocilizumab),⁶^–⁸ which were proven effective against COVID-19 by previous studies. Because the number of cases with antibody cocktails was insufficiently large, we did not divide them by the name of drug, as sotrovimab, casirivimab, and imdevimab. We considered the overall effects of antibody cocktails. These drugs were the five most common drugs against COVID-19. We assumed that vaccination coverage was the rate of the second dose of vaccine received two weeks prior by age class, as younger than 65 years old, and 65 years old or older. Mutated strains, Alpha, Delta and Omicron variant strains, were measured by percentage at one week before admission. The Alpha, Delta and Omicron variant strains were defined as the proportion of Alpha, Delta and Omicron variant strains. Omicron included BA.2 or a later sublineage. Alternatively, we used a dummy variable for the period of the 4–6th wave instead of the proportion of the mutated strains as an explanatory variable, to check robustness. By this specification, the Alpha variant strain emerged and then dominated in the 4th wave, defined as from 1 March through 20 June 2021. Similarly, the Delta variant strain emerged and then dominated in the 5th wave, defined as 21 June through 21 November 2021. Omicron BA.1 strain emerged and then dominated from the 6th wave, defined as 22 November 2021 to the end of the study period. Death during hospitalization was defined as the outcome.

Subjects

Subjects were all inpatients confirmed as having SARS-CoV-2. However, aside from pure medical criteria, the criteria of hospitalization for asymptomatic patients or patients with mild symptoms who did not require oxygen therapy were probably affected strongly by medical resource scarcity or social situations such as support for their staying at home and recuperation at home. For that reason, we also limited our study to subjects who were expected to be inpatients with oxygen therapy or respiratory ventilation.

Statistical analysis

We conducted estimates separately by the type of drug: remdesivir, antibody cocktail (sotrovimab, casirivimab and imdevimab), dexamethasone, baricitinib, and tocilizumab.

The first step to assess whether the patient was administered a type of drug or not was performed through logistic regression on their age, sex, underlying diseases, pharmaceutical therapy, vaccine coverage, and prevalence in the mutated strains as explanatory variables. The second step was comparison of outcomes among administered patients and unadministered patients with almost identical likelihood as the prediction done in the first step.

All statistical analyses were conducted using Stata software (SE 17.0; Stata Corp.). We adopted 5% as the significance level.

Ethical considerations

Individual informed consent was not required to conduct this study because the dataset was provided as anonymized data by National Hospital Organization.¹⁵ This study was approved by the Ethics Committee of Mie Hospital (Approval No. 2020-89) on 23 October 2020. Specific permission to use MIA data was obtained from the NHO (Registration No. 1201003) on 1 December 2020.

Results

A supplementary table (see Extended data¹⁹), presents a summary of the estimation results obtained for the first step logistic regression for drug administration. Table 1 presents the estimation results obtained for all inpatients, inpatients with oxygen therapy, and inpatients who used a respiratory ventilator in three age classes. Results show that 24 significance estimators of 90 estimators in all, but 20 estimators were positive, which indicates that drug administration reduced the probability of saving life.

Table 1. Estimation results of propensity score matching method.

Age class	All		65 years old and older		Younger than 65 years old
Age class	difference	p-value	difference	p-value	difference	p-value
Remdesivir
Proportion
All	0.034	0.000	0.064	0.000	0.019	0.068
Oxygen administration	0.015	0.315	0.045	0.056	0.013	0.311
Respiratory ventilator used	0.012	0.768	0.127	0.080	N.A.	N.A.
Period
All	0.031	0.000	0.067	0.004	0.010	0.126
Oxygen administration	0.012	0.348	0.044	0.087	0.011	0.337
Respiratory ventilator used	-0.006	0.893	0.047	0.477	N.A.	N.A.
Dexamethasone
Proportion
All	0.027	0.000	0.079	0.000	0.003	0.253
Oxygen administration	0.020	0.013	0.052	0.000	0.006	0.234
Respiratory ventilator used	-0.021	0.548	-0.055	0.296	0.044	0.319
Period
All	0.029	0.000	0.076	0.000	0.004	0.119
Oxygen administration	0.014	0.090	0.038	0.011	0.006	0.131
Vent	-0.051	0.158	-0.083	0.125	0.008	0.857
Tocilizumab
Proportion
All	0.091	0.000	0.214	0.000	0.019	0.057
Oxygen administration	0.079	0.001	0.126	0.021	N.A.	N.A.
Respiratory ventilator used	0.117	0.227	N.A.	N.A.	-0.077	0.030
Period
All	0.103	0.000	0.223	0.000	0.025	0.030
Oxygen administration	0.086	0.001	0.126	0.020	N.A.	N.A.
Respiratory ventilator used	0.090	0.375	N.A.	N.A.	-0.062	0.079
Baricitinib
Proportion
All	0.023	0.100	0.411	0.114	N.A.	N.A.
Oxygen administration	-0.016	0.220	-0.033	0.900	-0.008	0.023
Respiratory ventilator used	-0.149	0.586	0.384	0.000	N.A.	N.A.
Period
All	N.A.	N.A.	N.A.	N.A.	N.A.	N.A.
Oxygen administration	N.A.	N.A.	N.A.	N.A.	N.A.	N.A.
Respiratory ventilator used	N.A.	N.A.	N.A.	N.A.	N.A.	N.A.
Sotrovimab or casirivimab/imdevimab
Proportion
All	N.A.	N.A.	-0.105	0.000	N.A.	N.A.
Oxygen administration	-0.085	0.000	N.A.	N.A.	N.A.	N.A.
Respiratory ventilator used	N.A.	N.A.	N.A.	N.A.	N.A.	N.A.
Period	N.A.	N.A.	N.A.	N.A.	N.A.	N.A.
All	N.A.	N.A.	N.A.	N.A.	N.A.	N.A.
Oxygen administration	N.A.	N.A.	N.A.	N.A.	N.A.	N.A.
Respiratory ventilator used	N.A.	N.A.	N.A.	N.A.	N.A.	N.A.

Four estimators indicating that a drug probably contributes to saving life were the following: tocilizumab among younger inpatients with ventilators; baricitinib among younger inpatients with oxygen therapy; and antibody cocktails among all older inpatients and among all inpatients with oxygen therapy. Of three drugs, only the antibody cocktail did not have a positive difference, which suggests an increase in the probability of saving life. No other treatment effect estimator of such drugs was available. Some treatment effects in tocilizumab and baricitinib were positive, especially for all older inpatients or older inpatients with moderate severity. Therefore, the results obtained for these two drugs were mixed.

Discussion

Results showed that antibody cocktails might contribute to saving life. The estimated results for tocilizumab and baricitinib were mixed, but these drugs were presumed to have some probability of saving life among younger but severe patients. However, we were unable to find any evidence that remdesivir and dexamethasone contribute to saving life at all.

These counterintuitive findings, which are inconsistent with results of earlier studies of the effectiveness of remdesivir and dexamethasone, might result from worse matching at the first step.²^,⁵ Also, endogeneity might not be controlled well in decisions to use drugs. Drugs to be administered against COVID-19 depend on the severity of illness of the patients. For instance, an earlier study showed dexamethasone as effective for moderate or severe COVID-19, but ineffective for mild COVID-19.⁵ Actually, more than antibody cocktails,³^,⁴ remdesivir² and dexamethasone⁵ tend to be administered more for severer patients. To resolve this difficulty, more information indicating severity is needed. Moreover, we did not use test results such as blood pressure, body mass index, and oxygen saturation. We must include such information in the first step. Inclusion of such data remains a challenge for future research.

The results for antibody cocktails included many “not available” entries, although drug effectiveness was confirmed in well matched cases. That finding might be attributable to the small sample of antibody cocktail use for patients. In Japan, remdesivir, baricitinib, casirivimab/imdevimab, sotrovimab, and tocilizumab were approved for treatment of COVID-19, respectively, in May 2020, April 2021, July 2021, September 2021, and January 2022.²⁰ More recently approved drugs tend to be associated with smaller samples. Data accumulation might resolve that shortcoming.

Limitations

First, we estimated the effectiveness of drugs separately. However, the choice of drug was not actually independent. Therefore, intercorrelation among drugs should be incorporated into the estimation model.

Second, because MIA is a database of medical claims, data of the prior few weeks might change during a few months. For this study, we collected and analyzed data from January 2020 through the end of March 2022, recorded as of May 2022. Data and therefore estimation results might change over time if the study period were extended. In addition to this, we examined using Japanese vaccine coverage instead of the vaccination history of the patients. We also used data on prevalence of mutated strains in Tokyo as causative strains. If the vaccine history and causative strains of the patients themselves were available, the results might be different.

Third, although we compared mortality among an administrated group and an unadministered group based on the estimated propensity score at the first step, we also control demographical condition and/or underlying diseases, in addition to epidemiological situations for mortality. That remains as a challenge for our future research.

Fourth, the drugs considered for this study were not always administered to the most severely ill patients. For example, antibody cocktails were administered to patients with less–severe illness. However, to simplify the analysis used for this study, we did not incorporate the combination of drugs or change from one considered drug to another considered drug. That choice might bias the results to some degree. Such a process or pattern of drug administration might be important to evaluate drug effectiveness.

Conclusion

The obtained results showed that antibody cocktails for all older patients and all inpatients with moderate severity, tocilizumab for severely ill younger inpatients, and baricitinib for younger inpatients with moderate illness severity might contribute to saving life. In the latter two drugs, tocilizumab and baricitinib, some treatment effects suggested a reduction of saving life for all or older inpatients. We infer that the effects of these drugs were mixed. In fact, remdesivir and dexamethasone might not contribute to saving life. However, more information, including test results, is needed for better matching and achievement of definitive conclusions.

Data availability

Underlying data

The data in this study was provided from the National Hospital Organization (NHO), but the availability of these data was restricted because of privacy policy of NHO. This data was used under license for this study and were therefore not available to the public. Authors were permitted to use this data by the Ethical Committee of Mie Hospital and NHO Committee.

Extended data

Figshare: Drug effectiveness for COVID-19 inpatients inferred from Japanese medical claim data using propensity score matching_supplement data.csv https://doi.org/10.6084/m9.figshare.22102016.v1.¹⁹

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

References

1. Miyashita K, Hozumi H, Furuhashi K, et al.: Changes in the characteristics and outcomes of COVID-19 patients from the early pandemic to the delta variant epidemic: a nationwide population-based study. Emerg. Microbes. Infect. 2023; 12(1): 2155250. PubMed Abstract | Publisher Full Text | Free Full Text
2. Mozaffari E, Chandak A, Zhang Z, et al.: Remdesivir Treatment in Hospitalized Patients With Coronavirus Disease 2019 (COVID-19): A Comparative Analysis of In-hospital All-cause Mortality in a Large Multicenter Observational Cohort. Clin. Infect. Dis. 2022; 75(1): e450–e458. PubMed Abstract | Publisher Full Text | Free Full Text
3. Weinreich DM, Sivapalasingam S, Norton T, et al.: REGEN-COV Antibody Combination and Outcomes in Outpatients with Covid-19. N. Engl. J. Med. 2021; 385(23): e81. Epub 20210929. PubMed Abstract | Publisher Full Text | Free Full Text
4. Gupta A, Gonzalez-Rojas Y, Juarez E, et al.: Early Treatment for Covid-19 with SARS-CoV-2 Neutralizing Antibody Sotrovimab. N. Engl. J. Med. 2021; 385(21): 1941–1950. PubMed Abstract | Publisher Full Text
5. Horby P, Lim WS, Emberson JR, et al.: Dexamethasone in Hospitalized Patients with Covid-19. N. Engl. J. Med. 2021; 384(8): 693–704. PubMed Abstract | Publisher Full Text
6. Marconi VC, Ramanan AV, de Bono S , et al.: Efficacy and safety of baricitinib for the treatment of hospitalised adults with COVID-19 (COV-BARRIER): a randomised, double-blind, parallel-group, placebo-controlled phase 3 trial. Lancet Respir. Med. 2021; 9(12): 1407–1418. PubMed Abstract | Publisher Full Text | Free Full Text
7. Salama C, Han J, Yau L, et al.: Tocilizumab in Patients Hospitalized with Covid-19 Pneumonia. N. Engl. J. Med. 2021; 384(1): 20–30. PubMed Abstract | Publisher Full Text | Free Full Text
8. Gordon AC, Mouncey PR, Al-Beidh F, et al.: Interleukin-6 Receptor Antagonists in Critically Ill Patients with Covid-19. N. Engl. J. Med. 2021; 384(16): 1491–1502. PubMed Abstract | Publisher Full Text
9. Umakanthan S, Senthil S, John S, et al.: The Effect of Statins on Clinical Outcome Among Hospitalized Patients With COVID-19: A Multi-Centric Cohort Study. Front. Pharmacol. 2022; 13: 742273. Epub 20220705. PubMed Abstract | Publisher Full Text | Free Full Text
10. Bergqvist R, Ahlqvist VH, Lundberg M, et al.: HMG-CoA reductase inhibitors and COVID-19 mortality in Stockholm, Sweden: A registry-based cohort study. PLoS Med. 2021; 18(10): e1003820. Epub 20211014. PubMed Abstract | Publisher Full Text | Free Full Text
11. Wong CKH, Au ICH, Lau KTK, et al.: Real-world effectiveness of molnupiravir and nirmatrelvir plus ritonavir against mortality, hospitalisation, and in-hospital outcomes among community-dwelling, ambulatory patients with confirmed SARS-CoV-2 infection during the omicron wave in Hong Kong: an observational study. Lancet. 2022; 400(10359): 1213–1222. Publisher Full Text
12. Jayk Bernal A, Gomes da Silva MM, Musungaie DB, et al.: Molnupiravir for Oral Treatment of Covid-19 in Nonhospitalized Patients. N. Engl. J. Med. 2022; 386(6): 509–520. PubMed Abstract | Publisher Full Text | Free Full Text
13. Hammond J, Leister-Tebbe H, Gardner A, et al.: Oral Nirmatrelvir for High-Risk, Nonhospitalized Adults with Covid-19. N. Engl. J. Med. 2022; 386(15): 1397–1408. PubMed Abstract | Publisher Full Text | Free Full Text
14. 令和2 (2020) 年医療施設 (静態・動態) 調査 [Survey of Medical Institutions. 2020]. Tokyo: Ministry of Health, Labour, and Welfare; 2020 [cited 2022 Nov 10]. Reference Source
15. Kanazawa N, Tani T, Imai S, et al.: Existing Data Sources for Clinical Epidemiology: Database of the National Hospital Organization in Japan. Clin. Epidemiol. 2022; 14: 689–698. Publisher Full Text
16. Heckman JJ, Ichimura H, Todd PE: Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme. Rev. Econ. Stud. 1997; 64(4): 605–654. Publisher Full Text
17. Kane LT, Fang T, Galetta MS, et al.: Propensity Score Matching: A Statistical Method. Clin. Spine Surg. 2020; 33(3): 120–122. Publisher Full Text
18. Deb S, Austin PC, Tu JV, et al.: A Review of Propensity-Score Methods and Their Use in Cardiovascular Research. Can. J. Cardiol. 2016; 32(2): 259–265. PubMed Abstract | Publisher Full Text
19. Mitsushima S, Horiguchi H, Taniguchi K: Drug effectiveness for COVID-19 inpatients inferred from Japanese medical claim data using propensity score matching_supplement data.csv. Dataset. figshare. Publisher Full Text
20. Ministry of Health, Labour, and Welfare: Clinical Management of Patients with COVID-19 ver. 8.1. Tokyo: Ministry of Health, Labour, and Welfare (in Japanese); [cited 2022 Nov 8]. Reference Source

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 13 Apr 2023

Author details Author details

Hiromasa Horiguchi
Roles: Conceptualization, Data Curation, Funding Acquisition, Resources, Software, Supervision, Validation, Writing – Review & Editing

Kiyosu Taniguchi
Roles: Conceptualization, Data Curation, Funding Acquisition, Methodology, Project Administration, Resources, Supervision, Validation, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

This work was supported by the Ministry of Health, Labour, and Welfare [grant number 20HA1005].
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (2)

version 2

Revised

Published: 22 Jan 2024, 12:398

https://doi.org/10.12688/f1000research.131102.2

version 1

Published: 13 Apr 2023, 12:398

https://doi.org/10.12688/f1000research.131102.1

© 2023 Mitsushima S et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Mitsushima S, Horiguchi H and Taniguchi K. Drug effectiveness for COVID-19 inpatients inferred from Japanese medical claim data using propensity score matching [version 1; peer review: 1 approved with reservations]. F1000Research 2023, 12:398 (https://doi.org/10.12688/f1000research.131102.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 13 Apr 2023

Views

Reviewer Report 10 Nov 2023

Ramesh Kandimalla, Biochemistry, Kakatiya Medical College, Warangal, Telengana, India

Approved with Reservations

https://doi.org/10.5256/f1000research.143911.r214688

The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This could help readers better understand the study design.

The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This could help readers better understand the study design.
It would be helpful to provide more information about the reliability and representativeness of the observational data from Japan. Mentioning any limitations or potential biases in the data source would enhance the transparency of the study.
The abstract implies causal relationships between drug administration and outcomes, but the observational nature of the study might only establish correlations. It's essential to acknowledge the limitations in inferring causation from observational data.
The abstract mentions that the logistic regression might have "insufficient explanatory power." It would be beneficial to elaborate on this limitation and discuss potential sources of bias or confounding that could affect the study's validity.
When discussing the drugs, providing a brief rationale for why certain drugs might be effective or ineffective could enhance the comprehensibility of the results.
The mention of the Omicron variant is timely, but given the evolving nature of the pandemic, it's important to note the potential for changes in outcomes as new variants emerge. This could affect the generalizability of the findings.
The abstract could include a brief section discussing the limitations of the study, such as potential confounders not accounted for in the analysis or any challenges in accurately measuring certain variables.
The use of phrases like "in Japan" and "against COVID-19" is repeated frequently. Adding variety to the language could enhance the readability of the introduction.
The timeline of events related to COVID-19 in Japan could be presented more clearly. For example, mentioning the initial surge in cases, the development of treatments, and the subsequent decline in mortality rate in a more chronological order could improve the flow.
The introduction references earlier studies and clinical trials but doesn't provide specific citations for the mentioned data or findings. Including citations within the text could enhance the credibility of the information presented.
While the introduction mentions the decline in the mortality rate due to COVID-19 in Japan, it might be beneficial to include specific statistics or percentages to quantify this change and emphasize the significance.
The term "real-world effectiveness" is used, but a brief explanation or definition could be included to clarify its meaning for readers who may not be familiar with the term.
The limitations of the MIA data, such as the absence of vaccination history or causative strain information, are mentioned but could be discussed more explicitly. Acknowledging these limitations upfront adds transparency to the study.
The transition from providing background information to discussing the methods is somewhat abrupt. Adding a sentence or two to smoothly transition into the methods section could improve the overall flow of the introduction.
While the introduction mentions propensity score matching, a brief explanation of how it works and its purpose in addressing selection bias could enhance reader understanding.
The introduction could touch upon the external validity of the study, acknowledging that findings may not be universally applicable due to variations in healthcare systems, populations, or treatment protocols.
The introduction could explicitly state the research question or hypothesis being addressed by the study to guide readers in understanding the study's purpose from the outset.
The text repeats certain points, such as the mention of "propensity score" and "saving life," multiple times. Reducing redundancy could improve the overall clarity and conciseness of the text.
Some sentences are quite complex, which might make it challenging for readers to follow the arguments. Simplifying the language without compromising precision could enhance comprehension.
The text occasionally uses language that implies causation, such as "might contribute to saving life." It's important to be cautious about making causal claims based on observational data. Acknowledging the limitations in establishing causation would strengthen the study's validity.
The text mentions "not available" entries in the results for antibody cocktails, attributing it to a small sample size. Providing a more explicit explanation of why certain data is missing and its potential impact on the results would enhance transparency.
The limitations section is placed towards the end of the text. Moving it closer to the beginning or integrating limitations as they become relevant in the discussion could improve the flow and help readers contextualize the findings.
The use of terms like "severely ill," "moderate severity," and "less–severe illness" could benefit from clearer operational definitions or references to specific criteria used for classification.
The discussion section covers a range of topics, including limitations and future research, without clear organization. Structuring the discussion into subsections could make it more reader-friendly.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: COVID-19, Diabetes, Alzheimer's, Mitochondria, Carbon dots, Nanospheres

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 22 Jan 2024

Shingo Mitsushima, Center for Field Epidemic Intelligence, Research and Professional Development, National Institute of Infectious Diseases, Shinjuku-ku, 1620052, Japan

22 Jan 2024

Author Response

1. The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This ... Continue reading 1. The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This could help readers better understand the study design.

Response: We added the following sentence to Introduction.
“Moreover, after approval, experiments with random assignment are expected to be difficult to conduct in the real world because of ethical considerations: some patients might be unable to receive a potentially effective drug. Therefore, pseudo-experimental method should be applied to evaluate drug effectiveness in the real world.”

2. It would be helpful to provide more information about the reliability and representativeness of the observational data from Japan. Mentioning any limitations or potential biases in the data source would enhance the transparency of the study.

Response: We added the following sentence to Introduction.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation.”

3. The abstract implies causal relationships between drug administration and outcomes, but the observational nature of the study might only establish correlations. It's essential to acknowledge the limitations in inferring causation from observational data.

Response: We revised sentences in Abstract as below.
OLD: “raised the probability to save life” “contribute to saving life”
NEW: “was associated with raising the probability of survival” “were related to reduced probability of survival”
In addition, we added the following sentence to Discussion.
“this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

4. The abstract mentions that the logistic regression might have "insufficient explanatory power." It would be beneficial to elaborate on this limitation and discuss potential sources of bias or confounding that could affect the study's validity.

Response: We added the following sentence to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

5. When discussing the drugs, providing a brief rationale for why certain drugs might be effective or ineffective could enhance the comprehensibility of the results.

Response: We added the following sentences to Discussion.
“An earlier study indicated that casirivimab/imdevimab, an antibody cocktail, lowered the viral load. Typically, these antibody cocktails were given to patients with mild severity, which means that they were administered before a marked increase in viral load. This pattern of administration might be one reason why antibody cocktails were effective for survival.”

6. The mention of the Omicron variant is timely, but given the evolving nature of the pandemic, it's important to note the potential for changes in outcomes as new variants emerge. This could affect the generalizability of the findings.

Response: We added the following sentences to Discussion.
“If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

7. The abstract could include a brief section discussing the limitations of the study, such as potential confounders not accounted for in the analysis or any challenges in accurately measuring certain variables.

Response:
We added the following sentences to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

8. The use of phrases like "in Japan" and "against COVID-19" is repeated frequently. Adding variety to the language could enhance the readability of the introduction.

Response: We replaced some words “Japan” and “COVID-19” with alternative words.

9. The timeline of events related to COVID-19 in Japan could be presented more clearly. For example, mentioning the initial surge in cases, the development of treatments, and the subsequent decline in mortality rate in a more chronological order could improve the flow.

Response: We added Figure 1 and Figure 2 to show the timeline of COVID-19 events. We also added the following sentences to Result.
“Table 1 shows the number of COVID-19 patients from National data and the numbers of inpatients, inpatients with oxygen therapy, and inpatients with respiratory ventilation in MIA data. These data show similar trends. Most notably, several surges occurred from January 2020 through March 2022, as shown in Figure 1. Figure 2 presents the COVID-19 fatality rate in National data and the rate of deaths among COVID-19 inpatients in MIA data. These data were less than 10%. They followed a similar trend.”

10. The introduction references earlier studies and clinical trials but doesn't provide specific citations for the mentioned data or findings. Including citations within the text could enhance the credibility of the information presented.

Response: We added the following sentence to Introduction.
“In earlier studies, all these drugs were associated with improved survival or reduced the risk of disease progression, but antibody cocktails were effective only for mildly ill patients; dexamethasone was only effective for moderately to severely ill patients.”

11. While the introduction mentions the decline in the mortality rate due to COVID-19 in Japan, it might be beneficial to include specific statistics or percentages to quantify this change and emphasize the significance.

Response: We drew Figure 2, which shows fatality rates in National data and rate of deaths among COVID-19 inpatients with specific percentage.

12. The term "real-world effectiveness" is used, but a brief explanation or definition could be included to clarify its meaning for readers who may not be familiar with the term.

Response: We added the following sentences to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.” “Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols.”

13. The limitations of the MIA data, such as the absence of vaccination history or causative strain information, are mentioned but could be discussed more explicitly. Acknowledging these limitations upfront adds transparency to the study.

Response: We added the following sentences about the limitation of vaccination history and causative strain to Discussion.
“Second, we examined the data, particularly using Japanese vaccine coverage instead of the vaccination history of the patients. We also used data of the prevalence of mutated strains in Tokyo as causative strains. If the vaccine history and causative strains of the patients themselves were available, then the results might be different.”
We also added the sentences about the limitation of MIA data to Discussion.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation. Moreover, because MIA is a database of medical claims, data of the prior few weeks might change during a few months. For this study, we collected and analyzed data from January 2020 through the end of March 2022, recorded as of May 2022. If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

14. The transition from providing background information to discussing the methods is somewhat abrupt. Adding a sentence or two to smoothly transition into the methods section could improve the overall flow of the introduction.

Response: We added the following sentence to Introduction.
“An average treatment effect model including propensity matching was applied to create these situations.”

15. While the introduction mentions propensity score matching, a brief explanation of how it works and its purpose in addressing selection bias could enhance reader understanding.

Response: We added the following sentence to Abstract.
“We applied the average treatment effect model, particularly propensity scoring, which can treat the choice of administered drug as if administration were randomly assigned to inpatients.”

16. The introduction could touch upon the external validity of the study, acknowledging that findings may not be universally applicable due to variations in healthcare systems, populations, or treatment protocols.

Response: We added the following sentences to Introduction.
““Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols. Therefore, the results of this study might not be applicable to other countries.”

17. The introduction could explicitly state the research question or hypothesis being addressed by the study to guide readers in understanding the study's purpose from the outset.

Response: We added the following sentence to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.””

18. The text repeats certain points, such as the mention of "propensity score" and "saving life," multiple times. Reducing redundancy could improve the overall clarity and conciseness of the text.

Response: We did the English proofreading.

19. Some sentences are quite complex, which might make it challenging for readers to follow the arguments. Simplifying the language without compromising precision could enhance comprehension.

Response: We did the English proofreading.

20. The text occasionally uses language that implies causation, such as "might contribute to saving life." It's important to be cautious about making causal claims based on observational data. Acknowledging the limitations in establishing causation would strengthen the study's validity.

Response: We added the following sentence to Discussion.
“Third, this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

21. The text mentions "not available" entries in the results for antibody cocktails, attributing it to a small sample size. Providing a more explicit explanation of why certain data is missing and its potential impact on the results would enhance transparency.

Response: The approval of antibody cocktails was relatively late, and the number of prescriptions was simply low, so there is no potential impact. We added the following sentence in Discussion.
“The antibody cocktail results included many “not available” entries, although drug effectiveness was confirmed in well-matched cases. The approval of antibody cocktails was late. The number of prescriptions was simply low, which might affect many cases for which “not available” was reported.”

22. The limitations section is placed towards the end of the text. Moving it closer to the beginning or integrating limitations as they become relevant in the discussion could improve the flow and help readers contextualize the findings.

Response: We integrated limitations to Discussion and changed each limitation to understand easily for the reader.

23. The use of terms like "severely ill," "moderate severity," and "less–severe illness" could benefit from clearer operational definitions or references to specific criteria used for classification.

Response: We added the following sentence to Method.
“For this study, mild symptoms were defined as a patient who did not require medical intervention such as oxygen therapy or respiratory ventilation, moderate symptoms were defined as a patient who required oxygen administration, and severe symptoms were defined as a patient who required respiratory ventilation.”

24. The discussion section covers a range of topics, including limitations and future research, without clear organization. Structuring the discussion into subsections could make it more reader-friendly.

Response: We have organized discussion structure by issue.
1. The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This could help readers better understand the study design.

Response: We added the following sentence to Introduction.
“Moreover, after approval, experiments with random assignment are expected to be difficult to conduct in the real world because of ethical considerations: some patients might be unable to receive a potentially effective drug. Therefore, pseudo-experimental method should be applied to evaluate drug effectiveness in the real world.”

2. It would be helpful to provide more information about the reliability and representativeness of the observational data from Japan. Mentioning any limitations or potential biases in the data source would enhance the transparency of the study.

Response: We added the following sentence to Introduction.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation.”

3. The abstract implies causal relationships between drug administration and outcomes, but the observational nature of the study might only establish correlations. It's essential to acknowledge the limitations in inferring causation from observational data.

Response: We revised sentences in Abstract as below.
OLD: “raised the probability to save life” “contribute to saving life”
NEW: “was associated with raising the probability of survival” “were related to reduced probability of survival”
In addition, we added the following sentence to Discussion.
“this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

4. The abstract mentions that the logistic regression might have "insufficient explanatory power." It would be beneficial to elaborate on this limitation and discuss potential sources of bias or confounding that could affect the study's validity.

Response: We added the following sentence to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

5. When discussing the drugs, providing a brief rationale for why certain drugs might be effective or ineffective could enhance the comprehensibility of the results.

Response: We added the following sentences to Discussion.
“An earlier study indicated that casirivimab/imdevimab, an antibody cocktail, lowered the viral load. Typically, these antibody cocktails were given to patients with mild severity, which means that they were administered before a marked increase in viral load. This pattern of administration might be one reason why antibody cocktails were effective for survival.”

6. The mention of the Omicron variant is timely, but given the evolving nature of the pandemic, it's important to note the potential for changes in outcomes as new variants emerge. This could affect the generalizability of the findings.

Response: We added the following sentences to Discussion.
“If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

7. The abstract could include a brief section discussing the limitations of the study, such as potential confounders not accounted for in the analysis or any challenges in accurately measuring certain variables.

Response:
We added the following sentences to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

8. The use of phrases like "in Japan" and "against COVID-19" is repeated frequently. Adding variety to the language could enhance the readability of the introduction.

Response: We replaced some words “Japan” and “COVID-19” with alternative words.

9. The timeline of events related to COVID-19 in Japan could be presented more clearly. For example, mentioning the initial surge in cases, the development of treatments, and the subsequent decline in mortality rate in a more chronological order could improve the flow.

Response: We added Figure 1 and Figure 2 to show the timeline of COVID-19 events. We also added the following sentences to Result.
“Table 1 shows the number of COVID-19 patients from National data and the numbers of inpatients, inpatients with oxygen therapy, and inpatients with respiratory ventilation in MIA data. These data show similar trends. Most notably, several surges occurred from January 2020 through March 2022, as shown in Figure 1. Figure 2 presents the COVID-19 fatality rate in National data and the rate of deaths among COVID-19 inpatients in MIA data. These data were less than 10%. They followed a similar trend.”

10. The introduction references earlier studies and clinical trials but doesn't provide specific citations for the mentioned data or findings. Including citations within the text could enhance the credibility of the information presented.

Response: We added the following sentence to Introduction.
“In earlier studies, all these drugs were associated with improved survival or reduced the risk of disease progression, but antibody cocktails were effective only for mildly ill patients; dexamethasone was only effective for moderately to severely ill patients.”

11. While the introduction mentions the decline in the mortality rate due to COVID-19 in Japan, it might be beneficial to include specific statistics or percentages to quantify this change and emphasize the significance.

Response: We drew Figure 2, which shows fatality rates in National data and rate of deaths among COVID-19 inpatients with specific percentage.

12. The term "real-world effectiveness" is used, but a brief explanation or definition could be included to clarify its meaning for readers who may not be familiar with the term.

Response: We added the following sentences to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.” “Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols.”

13. The limitations of the MIA data, such as the absence of vaccination history or causative strain information, are mentioned but could be discussed more explicitly. Acknowledging these limitations upfront adds transparency to the study.

Response: We added the following sentences about the limitation of vaccination history and causative strain to Discussion.
“Second, we examined the data, particularly using Japanese vaccine coverage instead of the vaccination history of the patients. We also used data of the prevalence of mutated strains in Tokyo as causative strains. If the vaccine history and causative strains of the patients themselves were available, then the results might be different.”
We also added the sentences about the limitation of MIA data to Discussion.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation. Moreover, because MIA is a database of medical claims, data of the prior few weeks might change during a few months. For this study, we collected and analyzed data from January 2020 through the end of March 2022, recorded as of May 2022. If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

14. The transition from providing background information to discussing the methods is somewhat abrupt. Adding a sentence or two to smoothly transition into the methods section could improve the overall flow of the introduction.

Response: We added the following sentence to Introduction.
“An average treatment effect model including propensity matching was applied to create these situations.”

15. While the introduction mentions propensity score matching, a brief explanation of how it works and its purpose in addressing selection bias could enhance reader understanding.

Response: We added the following sentence to Abstract.
“We applied the average treatment effect model, particularly propensity scoring, which can treat the choice of administered drug as if administration were randomly assigned to inpatients.”

16. The introduction could touch upon the external validity of the study, acknowledging that findings may not be universally applicable due to variations in healthcare systems, populations, or treatment protocols.

Response: We added the following sentences to Introduction.
““Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols. Therefore, the results of this study might not be applicable to other countries.”

17. The introduction could explicitly state the research question or hypothesis being addressed by the study to guide readers in understanding the study's purpose from the outset.

Response: We added the following sentence to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.””

18. The text repeats certain points, such as the mention of "propensity score" and "saving life," multiple times. Reducing redundancy could improve the overall clarity and conciseness of the text.

Response: We did the English proofreading.

19. Some sentences are quite complex, which might make it challenging for readers to follow the arguments. Simplifying the language without compromising precision could enhance comprehension.

Response: We did the English proofreading.

20. The text occasionally uses language that implies causation, such as "might contribute to saving life." It's important to be cautious about making causal claims based on observational data. Acknowledging the limitations in establishing causation would strengthen the study's validity.

Response: We added the following sentence to Discussion.
“Third, this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

21. The text mentions "not available" entries in the results for antibody cocktails, attributing it to a small sample size. Providing a more explicit explanation of why certain data is missing and its potential impact on the results would enhance transparency.

Response: The approval of antibody cocktails was relatively late, and the number of prescriptions was simply low, so there is no potential impact. We added the following sentence in Discussion.
“The antibody cocktail results included many “not available” entries, although drug effectiveness was confirmed in well-matched cases. The approval of antibody cocktails was late. The number of prescriptions was simply low, which might affect many cases for which “not available” was reported.”

22. The limitations section is placed towards the end of the text. Moving it closer to the beginning or integrating limitations as they become relevant in the discussion could improve the flow and help readers contextualize the findings.

Response: We integrated limitations to Discussion and changed each limitation to understand easily for the reader.

23. The use of terms like "severely ill," "moderate severity," and "less–severe illness" could benefit from clearer operational definitions or references to specific criteria used for classification.

Response: We added the following sentence to Method.
“For this study, mild symptoms were defined as a patient who did not require medical intervention such as oxygen therapy or respiratory ventilation, moderate symptoms were defined as a patient who required oxygen administration, and severe symptoms were defined as a patient who required respiratory ventilation.”

24. The discussion section covers a range of topics, including limitations and future research, without clear organization. Structuring the discussion into subsections could make it more reader-friendly.

Response: We have organized discussion structure by issue.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 22 Jan 2024

Shingo Mitsushima, Center for Field Epidemic Intelligence, Research and Professional Development, National Institute of Infectious Diseases, Shinjuku-ku, 1620052, Japan

22 Jan 2024

Author Response

1. The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This ... Continue reading 1. The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This could help readers better understand the study design.

Response: We added the following sentence to Introduction.
“Moreover, after approval, experiments with random assignment are expected to be difficult to conduct in the real world because of ethical considerations: some patients might be unable to receive a potentially effective drug. Therefore, pseudo-experimental method should be applied to evaluate drug effectiveness in the real world.”

2. It would be helpful to provide more information about the reliability and representativeness of the observational data from Japan. Mentioning any limitations or potential biases in the data source would enhance the transparency of the study.

Response: We added the following sentence to Introduction.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation.”

3. The abstract implies causal relationships between drug administration and outcomes, but the observational nature of the study might only establish correlations. It's essential to acknowledge the limitations in inferring causation from observational data.

Response: We revised sentences in Abstract as below.
OLD: “raised the probability to save life” “contribute to saving life”
NEW: “was associated with raising the probability of survival” “were related to reduced probability of survival”
In addition, we added the following sentence to Discussion.
“this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

4. The abstract mentions that the logistic regression might have "insufficient explanatory power." It would be beneficial to elaborate on this limitation and discuss potential sources of bias or confounding that could affect the study's validity.

Response: We added the following sentence to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

5. When discussing the drugs, providing a brief rationale for why certain drugs might be effective or ineffective could enhance the comprehensibility of the results.

Response: We added the following sentences to Discussion.
“An earlier study indicated that casirivimab/imdevimab, an antibody cocktail, lowered the viral load. Typically, these antibody cocktails were given to patients with mild severity, which means that they were administered before a marked increase in viral load. This pattern of administration might be one reason why antibody cocktails were effective for survival.”

6. The mention of the Omicron variant is timely, but given the evolving nature of the pandemic, it's important to note the potential for changes in outcomes as new variants emerge. This could affect the generalizability of the findings.

Response: We added the following sentences to Discussion.
“If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

7. The abstract could include a brief section discussing the limitations of the study, such as potential confounders not accounted for in the analysis or any challenges in accurately measuring certain variables.

Response:
We added the following sentences to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

8. The use of phrases like "in Japan" and "against COVID-19" is repeated frequently. Adding variety to the language could enhance the readability of the introduction.

Response: We replaced some words “Japan” and “COVID-19” with alternative words.

9. The timeline of events related to COVID-19 in Japan could be presented more clearly. For example, mentioning the initial surge in cases, the development of treatments, and the subsequent decline in mortality rate in a more chronological order could improve the flow.

Response: We added Figure 1 and Figure 2 to show the timeline of COVID-19 events. We also added the following sentences to Result.
“Table 1 shows the number of COVID-19 patients from National data and the numbers of inpatients, inpatients with oxygen therapy, and inpatients with respiratory ventilation in MIA data. These data show similar trends. Most notably, several surges occurred from January 2020 through March 2022, as shown in Figure 1. Figure 2 presents the COVID-19 fatality rate in National data and the rate of deaths among COVID-19 inpatients in MIA data. These data were less than 10%. They followed a similar trend.”

10. The introduction references earlier studies and clinical trials but doesn't provide specific citations for the mentioned data or findings. Including citations within the text could enhance the credibility of the information presented.

Response: We added the following sentence to Introduction.
“In earlier studies, all these drugs were associated with improved survival or reduced the risk of disease progression, but antibody cocktails were effective only for mildly ill patients; dexamethasone was only effective for moderately to severely ill patients.”

11. While the introduction mentions the decline in the mortality rate due to COVID-19 in Japan, it might be beneficial to include specific statistics or percentages to quantify this change and emphasize the significance.

Response: We drew Figure 2, which shows fatality rates in National data and rate of deaths among COVID-19 inpatients with specific percentage.

12. The term "real-world effectiveness" is used, but a brief explanation or definition could be included to clarify its meaning for readers who may not be familiar with the term.

Response: We added the following sentences to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.” “Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols.”

13. The limitations of the MIA data, such as the absence of vaccination history or causative strain information, are mentioned but could be discussed more explicitly. Acknowledging these limitations upfront adds transparency to the study.

Response: We added the following sentences about the limitation of vaccination history and causative strain to Discussion.
“Second, we examined the data, particularly using Japanese vaccine coverage instead of the vaccination history of the patients. We also used data of the prevalence of mutated strains in Tokyo as causative strains. If the vaccine history and causative strains of the patients themselves were available, then the results might be different.”
We also added the sentences about the limitation of MIA data to Discussion.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation. Moreover, because MIA is a database of medical claims, data of the prior few weeks might change during a few months. For this study, we collected and analyzed data from January 2020 through the end of March 2022, recorded as of May 2022. If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

14. The transition from providing background information to discussing the methods is somewhat abrupt. Adding a sentence or two to smoothly transition into the methods section could improve the overall flow of the introduction.

Response: We added the following sentence to Introduction.
“An average treatment effect model including propensity matching was applied to create these situations.”

15. While the introduction mentions propensity score matching, a brief explanation of how it works and its purpose in addressing selection bias could enhance reader understanding.

Response: We added the following sentence to Abstract.
“We applied the average treatment effect model, particularly propensity scoring, which can treat the choice of administered drug as if administration were randomly assigned to inpatients.”

16. The introduction could touch upon the external validity of the study, acknowledging that findings may not be universally applicable due to variations in healthcare systems, populations, or treatment protocols.

Response: We added the following sentences to Introduction.
““Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols. Therefore, the results of this study might not be applicable to other countries.”

17. The introduction could explicitly state the research question or hypothesis being addressed by the study to guide readers in understanding the study's purpose from the outset.

Response: We added the following sentence to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.””

18. The text repeats certain points, such as the mention of "propensity score" and "saving life," multiple times. Reducing redundancy could improve the overall clarity and conciseness of the text.

Response: We did the English proofreading.

19. Some sentences are quite complex, which might make it challenging for readers to follow the arguments. Simplifying the language without compromising precision could enhance comprehension.

Response: We did the English proofreading.

20. The text occasionally uses language that implies causation, such as "might contribute to saving life." It's important to be cautious about making causal claims based on observational data. Acknowledging the limitations in establishing causation would strengthen the study's validity.

Response: We added the following sentence to Discussion.
“Third, this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

21. The text mentions "not available" entries in the results for antibody cocktails, attributing it to a small sample size. Providing a more explicit explanation of why certain data is missing and its potential impact on the results would enhance transparency.

Response: The approval of antibody cocktails was relatively late, and the number of prescriptions was simply low, so there is no potential impact. We added the following sentence in Discussion.
“The antibody cocktail results included many “not available” entries, although drug effectiveness was confirmed in well-matched cases. The approval of antibody cocktails was late. The number of prescriptions was simply low, which might affect many cases for which “not available” was reported.”

22. The limitations section is placed towards the end of the text. Moving it closer to the beginning or integrating limitations as they become relevant in the discussion could improve the flow and help readers contextualize the findings.

Response: We integrated limitations to Discussion and changed each limitation to understand easily for the reader.

23. The use of terms like "severely ill," "moderate severity," and "less–severe illness" could benefit from clearer operational definitions or references to specific criteria used for classification.

Response: We added the following sentence to Method.
“For this study, mild symptoms were defined as a patient who did not require medical intervention such as oxygen therapy or respiratory ventilation, moderate symptoms were defined as a patient who required oxygen administration, and severe symptoms were defined as a patient who required respiratory ventilation.”

24. The discussion section covers a range of topics, including limitations and future research, without clear organization. Structuring the discussion into subsections could make it more reader-friendly.

Response: We have organized discussion structure by issue.
1. The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This could help readers better understand the study design.

Response: We added the following sentence to Introduction.
“Moreover, after approval, experiments with random assignment are expected to be difficult to conduct in the real world because of ethical considerations: some patients might be unable to receive a potentially effective drug. Therefore, pseudo-experimental method should be applied to evaluate drug effectiveness in the real world.”

2. It would be helpful to provide more information about the reliability and representativeness of the observational data from Japan. Mentioning any limitations or potential biases in the data source would enhance the transparency of the study.

Response: We added the following sentence to Introduction.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation.”

3. The abstract implies causal relationships between drug administration and outcomes, but the observational nature of the study might only establish correlations. It's essential to acknowledge the limitations in inferring causation from observational data.

Response: We revised sentences in Abstract as below.
OLD: “raised the probability to save life” “contribute to saving life”
NEW: “was associated with raising the probability of survival” “were related to reduced probability of survival”
In addition, we added the following sentence to Discussion.
“this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

4. The abstract mentions that the logistic regression might have "insufficient explanatory power." It would be beneficial to elaborate on this limitation and discuss potential sources of bias or confounding that could affect the study's validity.

Response: We added the following sentence to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

5. When discussing the drugs, providing a brief rationale for why certain drugs might be effective or ineffective could enhance the comprehensibility of the results.

Response: We added the following sentences to Discussion.
“An earlier study indicated that casirivimab/imdevimab, an antibody cocktail, lowered the viral load. Typically, these antibody cocktails were given to patients with mild severity, which means that they were administered before a marked increase in viral load. This pattern of administration might be one reason why antibody cocktails were effective for survival.”

6. The mention of the Omicron variant is timely, but given the evolving nature of the pandemic, it's important to note the potential for changes in outcomes as new variants emerge. This could affect the generalizability of the findings.

Response: We added the following sentences to Discussion.
“If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

7. The abstract could include a brief section discussing the limitations of the study, such as potential confounders not accounted for in the analysis or any challenges in accurately measuring certain variables.

Response:
We added the following sentences to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

8. The use of phrases like "in Japan" and "against COVID-19" is repeated frequently. Adding variety to the language could enhance the readability of the introduction.

Response: We replaced some words “Japan” and “COVID-19” with alternative words.

9. The timeline of events related to COVID-19 in Japan could be presented more clearly. For example, mentioning the initial surge in cases, the development of treatments, and the subsequent decline in mortality rate in a more chronological order could improve the flow.

Response: We added Figure 1 and Figure 2 to show the timeline of COVID-19 events. We also added the following sentences to Result.
“Table 1 shows the number of COVID-19 patients from National data and the numbers of inpatients, inpatients with oxygen therapy, and inpatients with respiratory ventilation in MIA data. These data show similar trends. Most notably, several surges occurred from January 2020 through March 2022, as shown in Figure 1. Figure 2 presents the COVID-19 fatality rate in National data and the rate of deaths among COVID-19 inpatients in MIA data. These data were less than 10%. They followed a similar trend.”

10. The introduction references earlier studies and clinical trials but doesn't provide specific citations for the mentioned data or findings. Including citations within the text could enhance the credibility of the information presented.

Response: We added the following sentence to Introduction.
“In earlier studies, all these drugs were associated with improved survival or reduced the risk of disease progression, but antibody cocktails were effective only for mildly ill patients; dexamethasone was only effective for moderately to severely ill patients.”

11. While the introduction mentions the decline in the mortality rate due to COVID-19 in Japan, it might be beneficial to include specific statistics or percentages to quantify this change and emphasize the significance.

Response: We drew Figure 2, which shows fatality rates in National data and rate of deaths among COVID-19 inpatients with specific percentage.

12. The term "real-world effectiveness" is used, but a brief explanation or definition could be included to clarify its meaning for readers who may not be familiar with the term.

Response: We added the following sentences to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.” “Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols.”

13. The limitations of the MIA data, such as the absence of vaccination history or causative strain information, are mentioned but could be discussed more explicitly. Acknowledging these limitations upfront adds transparency to the study.

Response: We added the following sentences about the limitation of vaccination history and causative strain to Discussion.
“Second, we examined the data, particularly using Japanese vaccine coverage instead of the vaccination history of the patients. We also used data of the prevalence of mutated strains in Tokyo as causative strains. If the vaccine history and causative strains of the patients themselves were available, then the results might be different.”
We also added the sentences about the limitation of MIA data to Discussion.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation. Moreover, because MIA is a database of medical claims, data of the prior few weeks might change during a few months. For this study, we collected and analyzed data from January 2020 through the end of March 2022, recorded as of May 2022. If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

14. The transition from providing background information to discussing the methods is somewhat abrupt. Adding a sentence or two to smoothly transition into the methods section could improve the overall flow of the introduction.

Response: We added the following sentence to Introduction.
“An average treatment effect model including propensity matching was applied to create these situations.”

15. While the introduction mentions propensity score matching, a brief explanation of how it works and its purpose in addressing selection bias could enhance reader understanding.

Response: We added the following sentence to Abstract.
“We applied the average treatment effect model, particularly propensity scoring, which can treat the choice of administered drug as if administration were randomly assigned to inpatients.”

16. The introduction could touch upon the external validity of the study, acknowledging that findings may not be universally applicable due to variations in healthcare systems, populations, or treatment protocols.

Response: We added the following sentences to Introduction.
““Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols. Therefore, the results of this study might not be applicable to other countries.”

17. The introduction could explicitly state the research question or hypothesis being addressed by the study to guide readers in understanding the study's purpose from the outset.

Response: We added the following sentence to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.””

18. The text repeats certain points, such as the mention of "propensity score" and "saving life," multiple times. Reducing redundancy could improve the overall clarity and conciseness of the text.

Response: We did the English proofreading.

19. Some sentences are quite complex, which might make it challenging for readers to follow the arguments. Simplifying the language without compromising precision could enhance comprehension.

Response: We did the English proofreading.

20. The text occasionally uses language that implies causation, such as "might contribute to saving life." It's important to be cautious about making causal claims based on observational data. Acknowledging the limitations in establishing causation would strengthen the study's validity.

Response: We added the following sentence to Discussion.
“Third, this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

21. The text mentions "not available" entries in the results for antibody cocktails, attributing it to a small sample size. Providing a more explicit explanation of why certain data is missing and its potential impact on the results would enhance transparency.

Response: The approval of antibody cocktails was relatively late, and the number of prescriptions was simply low, so there is no potential impact. We added the following sentence in Discussion.
“The antibody cocktail results included many “not available” entries, although drug effectiveness was confirmed in well-matched cases. The approval of antibody cocktails was late. The number of prescriptions was simply low, which might affect many cases for which “not available” was reported.”

22. The limitations section is placed towards the end of the text. Moving it closer to the beginning or integrating limitations as they become relevant in the discussion could improve the flow and help readers contextualize the findings.

Response: We integrated limitations to Discussion and changed each limitation to understand easily for the reader.

23. The use of terms like "severely ill," "moderate severity," and "less–severe illness" could benefit from clearer operational definitions or references to specific criteria used for classification.

Response: We added the following sentence to Method.
“For this study, mild symptoms were defined as a patient who did not require medical intervention such as oxygen therapy or respiratory ventilation, moderate symptoms were defined as a patient who required oxygen administration, and severe symptoms were defined as a patient who required respiratory ventilation.”

24. The discussion section covers a range of topics, including limitations and future research, without clear organization. Structuring the discussion into subsections could make it more reader-friendly.

Response: We have organized discussion structure by issue.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 13 Apr 2023

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 2 (revision) 22 Jan 24	read	read
Version 1 13 Apr 23	read

Ramesh Kandimalla, Kakatiya Medical College, Warangal, India
Junko Kurita, Daitobunka University, Higashimatsuyama-shi, Japan

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

4 Views

01 Aug 2024 | for Version 2

Junko Kurita, Department of Nursing, Faculty of Sports & Health Science, Daitobunka University, Higashimatsuyama-shi, Japan

4 Views Cite this report Responses(0)

Approved

"Drug effectiveness for COVID-19 inpatients inferred from Japanese medical claim data using propensity score matching" was an excellent study investigating drug effectiveness in the real world. Even though it was approved through rct, effectiveness in the real world might be different from rct. So as to realize an experimental situation in the observational clinical data, they used propensity score matching procedure.
I congratulate the author's clear analysis in this field. However, I have some minor points to revise.

Minor points:

1) Figure 2 shows that there were COVID-19 patients and died in the first or second week, 2020, in MIA dara. I understand that the initial COVID-19 patient was confirmed on January 15, 2020 and thus the third or fourth week of 2020. Was the initial patient in MIA true ? Or had these patients administered in the first week of 2020 before COVID-19 infection, but been infected in COVID-19 in the hospital much later and died ?

2) Because the study period was from January 2020, was the period before approve mentor recommendation in guideline for COVID-19 included in the study period ? If so, it might cause some bias to the results. It might be just a comparison before and after approvement.

3) Though the yellow marks were very clear, but it might not be the usual presentation.
I recommended to remove yellow marks.

4) "Proportion" or "period" in Table 1 were unclear. Could you add words in Table and/or add more explanation about them in legend.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

statistical and/or mathematical analysis for medical data

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

9 Views

11 Apr 2024 | for Version 2

Ramesh Kandimalla, Biochemistry, Kakatiya Medical College, Warangal, Telengana, India

9 Views Cite this report Responses(0)

Approved

All the comments were addressed appropriately. No more additional comments for me

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

COVID-19, Diabetes, Alzheimer's, Mitochondria, Carbon dots, Nanospheres

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

37 Views

10 Nov 2023 | for Version 1

Ramesh Kandimalla, Biochemistry, Kakatiya Medical College, Warangal, Telengana, India

37 Views Cite this report Responses(1)

Approved With Reservations

The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This could help readers better understand the study design.
It would be helpful to provide more information about the reliability and representativeness of the observational data from Japan. Mentioning any limitations or potential biases in the data source would enhance the transparency of the study.
The abstract implies causal relationships between drug administration and outcomes, but the observational nature of the study might only establish correlations. It's essential to acknowledge the limitations in inferring causation from observational data.
The abstract mentions that the logistic regression might have "insufficient explanatory power." It would be beneficial to elaborate on this limitation and discuss potential sources of bias or confounding that could affect the study's validity.
When discussing the drugs, providing a brief rationale for why certain drugs might be effective or ineffective could enhance the comprehensibility of the results.
The mention of the Omicron variant is timely, but given the evolving nature of the pandemic, it's important to note the potential for changes in outcomes as new variants emerge. This could affect the generalizability of the findings.
The abstract could include a brief section discussing the limitations of the study, such as potential confounders not accounted for in the analysis or any challenges in accurately measuring certain variables.
The use of phrases like "in Japan" and "against COVID-19" is repeated frequently. Adding variety to the language could enhance the readability of the introduction.
The timeline of events related to COVID-19 in Japan could be presented more clearly. For example, mentioning the initial surge in cases, the development of treatments, and the subsequent decline in mortality rate in a more chronological order could improve the flow.
The introduction references earlier studies and clinical trials but doesn't provide specific citations for the mentioned data or findings. Including citations within the text could enhance the credibility of the information presented.
While the introduction mentions the decline in the mortality rate due to COVID-19 in Japan, it might be beneficial to include specific statistics or percentages to quantify this change and emphasize the significance.
The term "real-world effectiveness" is used, but a brief explanation or definition could be included to clarify its meaning for readers who may not be familiar with the term.
The limitations of the MIA data, such as the absence of vaccination history or causative strain information, are mentioned but could be discussed more explicitly. Acknowledging these limitations upfront adds transparency to the study.
The transition from providing background information to discussing the methods is somewhat abrupt. Adding a sentence or two to smoothly transition into the methods section could improve the overall flow of the introduction.
While the introduction mentions propensity score matching, a brief explanation of how it works and its purpose in addressing selection bias could enhance reader understanding.
The introduction could touch upon the external validity of the study, acknowledging that findings may not be universally applicable due to variations in healthcare systems, populations, or treatment protocols.
The introduction could explicitly state the research question or hypothesis being addressed by the study to guide readers in understanding the study's purpose from the outset.
The text repeats certain points, such as the mention of "propensity score" and "saving life," multiple times. Reducing redundancy could improve the overall clarity and conciseness of the text.
Some sentences are quite complex, which might make it challenging for readers to follow the arguments. Simplifying the language without compromising precision could enhance comprehension.
The text occasionally uses language that implies causation, such as "might contribute to saving life." It's important to be cautious about making causal claims based on observational data. Acknowledging the limitations in establishing causation would strengthen the study's validity.
The text mentions "not available" entries in the results for antibody cocktails, attributing it to a small sample size. Providing a more explicit explanation of why certain data is missing and its potential impact on the results would enhance transparency.
The limitations section is placed towards the end of the text. Moving it closer to the beginning or integrating limitations as they become relevant in the discussion could improve the flow and help readers contextualize the findings.
The use of terms like "severely ill," "moderate severity," and "less–severe illness" could benefit from clearer operational definitions or references to specific criteria used for classification.
The discussion section covers a range of topics, including limitations and future research, without clear organization. Structuring the discussion into subsections could make it more reader-friendly.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

COVID-19, Diabetes, Alzheimer's, Mitochondria, Carbon dots, Nanospheres

Respond to this report

Responses (1)

Author Response

22 Jan 2024

Shingo Mitsushima, Center for Field Epidemic Intelligence, Research and Professional Development, National Institute of Infectious Diseases, Shinjuku-ku, 1620052, Japan

1. The methods section is detailed, but it could benefit from additional clarity in explaining the rationale behind using propensity scoring and the choice of variables in logistic regression. This could help readers better understand the study design.

Response: We added the following sentence to Introduction.
“Moreover, after approval, experiments with random assignment are expected to be difficult to conduct in the real world because of ethical considerations: some patients might be unable to receive a potentially effective drug. Therefore, pseudo-experimental method should be applied to evaluate drug effectiveness in the real world.”

2. It would be helpful to provide more information about the reliability and representativeness of the observational data from Japan. Mentioning any limitations or potential biases in the data source would enhance the transparency of the study.

Response: We added the following sentence to Introduction.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation.”

3. The abstract implies causal relationships between drug administration and outcomes, but the observational nature of the study might only establish correlations. It's essential to acknowledge the limitations in inferring causation from observational data.

Response: We revised sentences in Abstract as below.
OLD: “raised the probability to save life” “contribute to saving life”
NEW: “was associated with raising the probability of survival” “were related to reduced probability of survival”
In addition, we added the following sentence to Discussion.
“this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

4. The abstract mentions that the logistic regression might have "insufficient explanatory power." It would be beneficial to elaborate on this limitation and discuss potential sources of bias or confounding that could affect the study's validity.

Response: We added the following sentence to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

5. When discussing the drugs, providing a brief rationale for why certain drugs might be effective or ineffective could enhance the comprehensibility of the results.

Response: We added the following sentences to Discussion.
“An earlier study indicated that casirivimab/imdevimab, an antibody cocktail, lowered the viral load. Typically, these antibody cocktails were given to patients with mild severity, which means that they were administered before a marked increase in viral load. This pattern of administration might be one reason why antibody cocktails were effective for survival.”

6. The mention of the Omicron variant is timely, but given the evolving nature of the pandemic, it's important to note the potential for changes in outcomes as new variants emerge. This could affect the generalizability of the findings.

Response: We added the following sentences to Discussion.
“If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

7. The abstract could include a brief section discussing the limitations of the study, such as potential confounders not accounted for in the analysis or any challenges in accurately measuring certain variables.

Response:
We added the following sentences to Abstract.
“However, propensity score matching method might engender biased results because of a lack of data such as detailed information related to intervention and potential confounders. Therefore, the effectiveness of some drugs might not be evaluated properly in this study.”

8. The use of phrases like "in Japan" and "against COVID-19" is repeated frequently. Adding variety to the language could enhance the readability of the introduction.

Response: We replaced some words “Japan” and “COVID-19” with alternative words.

9. The timeline of events related to COVID-19 in Japan could be presented more clearly. For example, mentioning the initial surge in cases, the development of treatments, and the subsequent decline in mortality rate in a more chronological order could improve the flow.

Response: We added Figure 1 and Figure 2 to show the timeline of COVID-19 events. We also added the following sentences to Result.
“Table 1 shows the number of COVID-19 patients from National data and the numbers of inpatients, inpatients with oxygen therapy, and inpatients with respiratory ventilation in MIA data. These data show similar trends. Most notably, several surges occurred from January 2020 through March 2022, as shown in Figure 1. Figure 2 presents the COVID-19 fatality rate in National data and the rate of deaths among COVID-19 inpatients in MIA data. These data were less than 10%. They followed a similar trend.”

10. The introduction references earlier studies and clinical trials but doesn't provide specific citations for the mentioned data or findings. Including citations within the text could enhance the credibility of the information presented.

Response: We added the following sentence to Introduction.
“In earlier studies, all these drugs were associated with improved survival or reduced the risk of disease progression, but antibody cocktails were effective only for mildly ill patients; dexamethasone was only effective for moderately to severely ill patients.”

11. While the introduction mentions the decline in the mortality rate due to COVID-19 in Japan, it might be beneficial to include specific statistics or percentages to quantify this change and emphasize the significance.

Response: We drew Figure 2, which shows fatality rates in National data and rate of deaths among COVID-19 inpatients with specific percentage.

12. The term "real-world effectiveness" is used, but a brief explanation or definition could be included to clarify its meaning for readers who may not be familiar with the term.

Response: We added the following sentences to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.” “Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols.”

13. The limitations of the MIA data, such as the absence of vaccination history or causative strain information, are mentioned but could be discussed more explicitly. Acknowledging these limitations upfront adds transparency to the study.

Response: We added the following sentences about the limitation of vaccination history and causative strain to Discussion.
“Second, we examined the data, particularly using Japanese vaccine coverage instead of the vaccination history of the patients. We also used data of the prevalence of mutated strains in Tokyo as causative strains. If the vaccine history and causative strains of the patients themselves were available, then the results might be different.”
We also added the sentences about the limitation of MIA data to Discussion.
“In our earlier study, although we confirmed the representativeness of fatality cases using MIA data, the data on the numbers of patients with oxygen therapy and respiratory ventilation in Japan have not been published, yet. We have not verified the representativeness of patients with oxygen therapy and respiratory ventilation. Moreover, because MIA is a database of medical claims, data of the prior few weeks might change during a few months. For this study, we collected and analyzed data from January 2020 through the end of March 2022, recorded as of May 2022. If the study period was extended, the estimation results might change over time because the accumulated data engender more accurate results. Emerging new mutated strains, immunization, and development of new therapies might affect the results.”

14. The transition from providing background information to discussing the methods is somewhat abrupt. Adding a sentence or two to smoothly transition into the methods section could improve the overall flow of the introduction.

Response: We added the following sentence to Introduction.
“An average treatment effect model including propensity matching was applied to create these situations.”

15. While the introduction mentions propensity score matching, a brief explanation of how it works and its purpose in addressing selection bias could enhance reader understanding.

Response: We added the following sentence to Abstract.
“We applied the average treatment effect model, particularly propensity scoring, which can treat the choice of administered drug as if administration were randomly assigned to inpatients.”

16. The introduction could touch upon the external validity of the study, acknowledging that findings may not be universally applicable due to variations in healthcare systems, populations, or treatment protocols.

Response: We added the following sentences to Introduction.
““Real-world effectiveness of drugs” means the effectiveness of drugs considering several factors including population characteristics, examinations, vaccinations, and healthcare system such as hospitalization and treatment protocols. Therefore, the results of this study might not be applicable to other countries.”

17. The introduction could explicitly state the research question or hypothesis being addressed by the study to guide readers in understanding the study's purpose from the outset.

Response: We added the following sentence to Introduction.
“The objective of this study was evaluating the “real-world effectiveness of drugs against COVID-19 in Japan.””

18. The text repeats certain points, such as the mention of "propensity score" and "saving life," multiple times. Reducing redundancy could improve the overall clarity and conciseness of the text.

Response: We did the English proofreading.

19. Some sentences are quite complex, which might make it challenging for readers to follow the arguments. Simplifying the language without compromising precision could enhance comprehension.

Response: We did the English proofreading.

20. The text occasionally uses language that implies causation, such as "might contribute to saving life." It's important to be cautious about making causal claims based on observational data. Acknowledging the limitations in establishing causation would strengthen the study's validity.

Response: We added the following sentence to Discussion.
“Third, this study was an observational study that was able to show only association between considered drugs and death, not necessarily a causal association.”

21. The text mentions "not available" entries in the results for antibody cocktails, attributing it to a small sample size. Providing a more explicit explanation of why certain data is missing and its potential impact on the results would enhance transparency.

Response: The approval of antibody cocktails was relatively late, and the number of prescriptions was simply low, so there is no potential impact. We added the following sentence in Discussion.
“The antibody cocktail results included many “not available” entries, although drug effectiveness was confirmed in well-matched cases. The approval of antibody cocktails was late. The number of prescriptions was simply low, which might affect many cases for which “not available” was reported.”

22. The limitations section is placed towards the end of the text. Moving it closer to the beginning or integrating limitations as they become relevant in the discussion could improve the flow and help readers contextualize the findings.

Response: We integrated limitations to Discussion and changed each limitation to understand easily for the reader.

23. The use of terms like "severely ill," "moderate severity," and "less–severe illness" could benefit from clearer operational definitions or references to specific criteria used for classification.

Response: We added the following sentence to Method.
“For this study, mild symptoms were defined as a patient who did not require medical intervention such as oxygen therapy or respiratory ventilation, moderate symptoms were defined as a patient who required oxygen administration, and severe symptoms were defined as a patient who required respiratory ventilation.”

24. The discussion section covers a range of topics, including limitations and future research, without clear organization. Structuring the discussion into subsections could make it more reader-friendly.

Response: We have organized discussion structure by issue.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Miyashita K, Hozumi H, Furuhashi K, et al.: Changes in the characteristics and outcomes of COVID-19 patients from the early pandemic to the delta variant epidemic: a nationwide population-based study. Emerg. Microbes. Infect. 2023; 12(1): 2155250. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Mozaffari E, Chandak A, Zhang Z, et al.: Remdesivir Treatment in Hospitalized Patients With Coronavirus Disease 2019 (COVID-19): A Comparative Analysis of In-hospital All-cause Mortality in a Large Multicenter Observational Cohort. Clin. Infect. Dis. 2022; 75(1): e450–e458. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Weinreich DM, Sivapalasingam S, Norton T, et al.: REGEN-COV Antibody Combination and Outcomes in Outpatients with Covid-19. N. Engl. J. Med. 2021; 385(23): e81. Epub 20210929. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Gupta A, Gonzalez-Rojas Y, Juarez E, et al.: Early Treatment for Covid-19 with SARS-CoV-2 Neutralizing Antibody Sotrovimab. N. Engl. J. Med. 2021; 385(21): 1941–1950. PubMed Abstract | Publisher Full Text

[5] 5. Horby P, Lim WS, Emberson JR, et al.: Dexamethasone in Hospitalized Patients with Covid-19. N. Engl. J. Med. 2021; 384(8): 693–704. PubMed Abstract | Publisher Full Text

[6] 6. Marconi VC, Ramanan AV, de Bono S , et al.: Efficacy and safety of baricitinib for the treatment of hospitalised adults with COVID-19 (COV-BARRIER): a randomised, double-blind, parallel-group, placebo-controlled phase 3 trial. Lancet Respir. Med. 2021; 9(12): 1407–1418. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Salama C, Han J, Yau L, et al.: Tocilizumab in Patients Hospitalized with Covid-19 Pneumonia. N. Engl. J. Med. 2021; 384(1): 20–30. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Gordon AC, Mouncey PR, Al-Beidh F, et al.: Interleukin-6 Receptor Antagonists in Critically Ill Patients with Covid-19. N. Engl. J. Med. 2021; 384(16): 1491–1502. PubMed Abstract | Publisher Full Text

[9] 9. Umakanthan S, Senthil S, John S, et al.: The Effect of Statins on Clinical Outcome Among Hospitalized Patients With COVID-19: A Multi-Centric Cohort Study. Front. Pharmacol. 2022; 13: 742273. Epub 20220705. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Bergqvist R, Ahlqvist VH, Lundberg M, et al.: HMG-CoA reductase inhibitors and COVID-19 mortality in Stockholm, Sweden: A registry-based cohort study. PLoS Med. 2021; 18(10): e1003820. Epub 20211014. PubMed Abstract | Publisher Full Text | Free Full Text

[11] 11. Wong CKH, Au ICH, Lau KTK, et al.: Real-world effectiveness of molnupiravir and nirmatrelvir plus ritonavir against mortality, hospitalisation, and in-hospital outcomes among community-dwelling, ambulatory patients with confirmed SARS-CoV-2 infection during the omicron wave in Hong Kong: an observational study. Lancet. 2022; 400(10359): 1213–1222. Publisher Full Text

[12] 12. Jayk Bernal A, Gomes da Silva MM, Musungaie DB, et al.: Molnupiravir for Oral Treatment of Covid-19 in Nonhospitalized Patients. N. Engl. J. Med. 2022; 386(6): 509–520. PubMed Abstract | Publisher Full Text | Free Full Text

[13] 13. Hammond J, Leister-Tebbe H, Gardner A, et al.: Oral Nirmatrelvir for High-Risk, Nonhospitalized Adults with Covid-19. N. Engl. J. Med. 2022; 386(15): 1397–1408. PubMed Abstract | Publisher Full Text | Free Full Text

[14] 14. 令和2 (2020) 年医療施設 (静態・動態) 調査 [Survey of Medical Institutions. 2020]. Tokyo: Ministry of Health, Labour, and Welfare; 2020 [cited 2022 Nov 10]. Reference Source

[15] 15. Kanazawa N, Tani T, Imai S, et al.: Existing Data Sources for Clinical Epidemiology: Database of the National Hospital Organization in Japan. Clin. Epidemiol. 2022; 14: 689–698. Publisher Full Text

[16] 16. Heckman JJ, Ichimura H, Todd PE: Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme. Rev. Econ. Stud. 1997; 64(4): 605–654. Publisher Full Text

[17] 17. Kane LT, Fang T, Galetta MS, et al.: Propensity Score Matching: A Statistical Method. Clin. Spine Surg. 2020; 33(3): 120–122. Publisher Full Text

[18] 18. Deb S, Austin PC, Tu JV, et al.: A Review of Propensity-Score Methods and Their Use in Cardiovascular Research. Can. J. Cardiol. 2016; 32(2): 259–265. PubMed Abstract | Publisher Full Text

[19] 19. Mitsushima S, Horiguchi H, Taniguchi K: Drug effectiveness for COVID-19 inpatients inferred from Japanese medical claim data using propensity score matching_supplement data.csv. Dataset. figshare. Publisher Full Text

[20] 20. Ministry of Health, Labour, and Welfare: Clinical Management of Patients with COVID-19 ver. 8.1. Tokyo: Ministry of Health, Labour, and Welfare (in Japanese); [cited 2022 Nov 8]. Reference Source

Drug effectiveness for COVID-19 inpatients inferred from Japanese medical claim data using propensity score matching

Abstract

Keywords

Introduction

Method

Data sources

Definitions of variables

Subjects

Statistical analysis

Ethical considerations

Results

Table 1. Estimation results of propensity score matching method.

Discussion

Limitations

Conclusion

Data availability

Underlying data

Extended data

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated