Statistical Information on COVID-19 Vaccines in Italy: Methodological Errors and Misleading Communication

Measuring the efficacy and the effectiveness of COVID-19 vaccines

The vaccines under consideration were evaluated with the aim of preventing symptomatic COVID-19 disease (Polack et al., 2020; Voysey et al., 2021), rather than SARS-CoV-2 infection or transmission. Nevertheless, they were often presented as effective tools for blocking contagion, despite the lack of experimental evidence supporting such claims.

In assessing vaccine efficacy and effectiveness, both manufacturers and health institutions relied almost exclusively on relative risk reduction (RRR), neglecting absolute risk reduction (ARR). In the context of COVID-19, RRR quantifies the proportion of cases prevented by the vaccine among individuals who would have developed the disease in the absence of vaccination. By contrast, ARR measures the reduction in the overall risk of developing COVID-19 among all vaccinated individuals, regardless of their baseline susceptibility, thereby quantifying the actual benefit at the population level. Its reciprocal, the Number Needed to Vaccinate (NNV), represents the number of individuals who must be vaccinated to prevent a single case of disease.

During the first two months of the clinical trials, RRR values were high (95% for Pfizer, 94% for Moderna, and 67% for AstraZeneca) whereas ARR was approximately 1% as shown by Olliaro et al. (2021) (see Table 1). Consequently, around 100 individuals would need to be vaccinated to prevent a single case of disease. This point had already been noted for the Pfizer vaccine by Baccini et al. (2021) and Cheli (2021).

Institutional communications emphasized RRR and overshadowed ARR and NNV, which are crucial for assessing the utility of mass vaccination, particularly in contexts of low disease incidence. This contributed to a distorted representation of vaccine benefits¹, suggesting greater effectiveness than that supported by more comprehensive and transparent analyses.

Beyond the modest ARR values, several authors have identified critical issues in the criteria used to assess efficacy based on data from the Pfizer trial (Polack et al., 2020; Thomas et al., 2021), the most widely used vaccine in Italy. Moreover, frequent violations of the double-blind design in the Pfizer trial introduced objective biases into the evaluation of vaccine efficacy (Doshi, 2021; Thacker, 2021; Coombes and Davies, 2022; Quinn et al., 2025).

An additional critical issue concerns the lack of demonstrated effectiveness against mortality. For instance, in the Pfizer study, the total number of deaths in the vaccinated group (15/21,926) was slightly higher than in the placebo group (14/21,931) (see Table 2).

These data, derived from a randomized double-blind study, should represent the most reliable evidence available, yet they contrast with widespread claims that vaccines saved “millions of lives” (Watson et al., 2022; Ioannidis et al., 2025).

Subsequent assessments of relative vaccine effectiveness in Italy were based primarily on data from the Italian National Institute of Health (Istituto Superiore di Sanità - ISS), which exhibit several methodological shortcomings and the associated biases. These include: (i) the failure to specify, among vaccinated individuals, those who had recovered from COVID-19 prior to vaccination; (ii) selection bias due to the “healthy vaccinee effect,” whereby individuals who choose to be vaccinated tend on average to be in better health than those who do not; and (iii) the so-called “14-day rule,” which generates a very large distorting impact, as we will show in the next section.

Fung et al. (2023) showed that, in observational studies based on administrative data, such as those produced by the ISS, the combined effect of these biases can substantially inflate estimates of vaccine effectiveness.

The 14-Day rule and the resulting bias

According to the methodology adopted by the ISS (2021, 2022), all events—SARS-CoV-2 infection, hospitalization, admission to intensive care, and death—occurring within 14 days of the administration of the first vaccine dose were attributed to the unvaccinated population. Likewise, for the second and booster doses, events occurring within 14 days of administration were attributed to the previous vaccination status.

The ISS justifies this classification on the grounds that 14 days are required to develop an immune response. Moreover, since the incubation period of SARS-CoV-2 infection can last up to 14 days, some infections recorded during this interval may have been contracted prior to vaccination.

However, reallocating cases among individuals vaccinated for less than 14 days to the unvaccinated category introduces a dual source of bias: on the one hand, the risk among vaccinated individuals is underestimated, on the other hand, the risk among unvaccinated individuals is overestimated; consequently, vaccine effectiveness results systematically overestimated.

This methodological approach was not confined to Italy or the ISS; it was likewise adopted by comparable institutions in other countries, including the United Kingdom’s Office for National Statistics (ONS), following guidance issued by the WHO (2022).

Fenton and Neil (2023) showed that the overestimation of vaccine effectiveness resulting from the “14-day rule” is so substantial that even a placebo would appear highly effective during the initial months. These authors also provided a spreadsheet tool² that allows the simulation of different scenarios by freely varying the infection rate, population size, and speed of the vaccination campaign, showing that every combination invariably leads to the same outcome.

The type of bias described above also affects the estimates and effectiveness analyses reported in numerous scientific studies published since 2021. In a systematic review, Neil et al. (2024) identified this bias in each of the 38 studies they examined.

Alessandria et al. (2025) documented the presence of the bias resulting from the “14-day rule” in data from an Italian region obtained through a public records access request, showing that it artificially increased mortality among unvaccinated individuals while reducing it among vaccinated individuals, in line with what was described above.

The magnitude of this bias raises serious concerns about the reliability of vaccine effectiveness analyses conducted to date by the ISS (and the UK ONS), as well as those reported in a large proportion of published observational studies.

Surveillance of COVID-19 vaccines

Concerning vaccine safety, the most critical issue is the almost exclusive reliance on passive pharmacovigilance, which entails a substantial underestimation of adverse reactions and adverse events.

According to the 14th AIFA report³ (AIFA is the Italian Medicines Agency), from the beginning of the vaccination campaign up to December 2022 approximately 26,300 serious adverse reactions were reported, corresponding to 18.1 per 100,000 doses administered. Of these, however, only one third were considered to be causally related to vaccination (6 per 100,000 doses). By contrast, randomized double-blind clinical trials indicate that serious systemic reactions amount to approximately 4% in the case of the Pfizer vaccine and 17% in the case of Moderna⁴. Moreover, data from the active surveillance system V-safe in the United States show an even higher number of severe reactions, such as those making persons unable to work, ranging from approximately 12,300 to 20,000 per 100,000 second doses (Rosenblum et al., 2022).

Since the rates derived from pivotal clinical trials and from active surveillance systems are far more reliable than those obtained through passive surveillance (Ahmadizar et al., 2023; Baccini et al., 2025; Carter et al., 2025; Chandra et al., 2026), the figures disseminated by AIFA dramatically underestimate the magnitude of adverse reactions and adverse events.

Reported deaths and causality assessment

Further critical issues concern the method used by AIFA to assess the causal link between vaccination and adverse events, which is based on a WHO algorithm that classifies cases into four categories: consistent with causal association, inconsistent with causal association, indeterminate, and unclassifiable. This method led to the recognition of only 29 deaths as being causally related to vaccination out of 971 deaths reported during the first two years of the vaccination campaign; the vast majority were classified as “inconsistent” or “indeterminate” (AIFA, 2023). Moreover, AIFA assessed only 812 of the 971 reported deaths; the remaining 159 were either never evaluated or, if evaluated, the outcomes of those assessments were not disclosed. Nor did it conduct or report further investigations to resolve the uncertainty surrounding the many cases classified as “indeterminate” or “unclassifiable.” In particular, public communication tends inappropriately to identify the “indeterminate” category with a disproven causal association.

The WHO algorithm entails a standardized analysis based on highly restrictive criteria. Bellavite et al. (2024) highlight how these criteria (temporal relationship, biological plausibility, frequency of the event in the general population, presence of pre-existing conditions or comorbidities, and the existence of evidence in the scientific literature) systematically bias assessments toward excluding a causal relationship or, at the very least, making its recognition highly unlikely. In particular, the presence of pre-existing conditions is systematically used to attribute the adverse event to other causes, even in the absence of specific evidence supporting such a conclusion. Furthermore, the criterion requiring evidence in the literature tends to exclude events that are still poorly studied or not yet formally documented, as initially occurred in the case of post-vaccination menstrual disorders⁵.

When these criteria were applied, no causal association with vaccination was recognized for the majority of reported deaths, either because the events occurred outside the predefined time window or because, under the aforementioned algorithm, establishing a causal relationship is exceedingly difficult. As a result, most assessments concluded as “inconsistent with causal association” or “indeterminate.” (see Deutscher et al., 2024).

In June 2025, AIFA published the Vaccine Report 2023, containing reports of adverse events following immunization (AEFI) for 26 vaccines, including COVID-19 vaccines. One of the identified critical issues of this report is the improper aggregation of active and passive surveillance data, expressed in relation to the total number of administered doses, which leads to a serious underestimation of AEFI rates.

Errors in AIFA’s analysis comparing expected and observed post-vaccination deaths

Another critical issue in AIFA’s assessment of the safety of COVID-19 vaccines concerns serious errors in the analyses comparing “observed” and “expected” post-vaccination deaths, as reported in the 5th and 10th vaccine surveillance reports (AIFA, 2021a and 2022). AIFA compares what it defines as “observed” post-vaccination deaths with those expected in the absence of vaccination, computes the Standardized Mortality Ratio (SMR), and—obtaining values significantly below 1—concludes that there is no evidence of increased mortality associated with vaccination.

However, these conclusions are unreliable due to two fundamental errors. The first is a gross overestimation of expected deaths, which mechanically lowers the SMR value. The most serious error, however, lies in considering as “observed” deaths only those reported through pharmacovigilance systems, rather than all deaths that actually occurred within the vaccinated population. This makes the two terms of the ratio non-comparable, leading to an artificially low SMR (Cheli et al. 2022; Baccini et al., 2022 and 2023).

Use of misleading infographics

A further distortion concerns graphs and infographics that were improperly designed to convey reassuring messages rather than to provide accurate and impartial information. An example is offered by several bubble charts included in the first six AIFA vaccine surveillance reports. One such chart is reproduced on the left-hand side of Figure 1. The television program “Fuori dal coro” (25 April 2023) also showed an internal email from an AIFA executive containing the following instruction: “The area of the circle representing serious adverse events should not be proportional; it could be smaller”. The area of the inner circle should represent 11.9% of the total, but it is actually much smaller (the right-hand side of the Figure reconstructs the chart using correct proportions). This design choice leads the public to perceive a smaller share of serious adverse reactions than that indicated by the numerical data.

Figure 1. Chart published in the 6th AIFA Report on COVID-19 Vaccine Surveillance vs corrected elaboration.

Source: Left-hand side: 6th AIFA Report on COVID-19 Vaccine Surveillance (AIFA, 2021b); right-hand side: our elaboration of left-hand side chart. Reproduced/adapted from publicly available material.

Legend – Translation of the text: Title: suspected serious/non-serious adverse reactions.

In green: non-serious.

In orange: serious.

At the bottom: 0.2% of suspected adverse reactions are unclassified.

Left-hand side (original chart): the area of the orange circle is much smaller than 11.9% of the total.

Right-hand side (authors’ elaboration): the same chart reconstructed using correct proportions.

Further examples include infographics issued by some Regional authorities which, through altered dimensions and symbolism (alongside the biases discussed in Sections 2 and 3), inflate the perceived effectiveness and safety of the vaccination (Baccini et al., 2025).

As an illustrative example, we examine here an infographic published by the Piemonte Region ( Figure 2).

Figure 2. Incorrectly designed infographic by Piemonte Region.

Source: Published on the official Facebook page of Piemonte Region on January 7, 2022. (https://www.facebook.com/regione.piemonte.official/posts/5180080022004802) and subsequently republished by several news outlets, including Gazzetta d’Asti (https://www.gazzettadasti.it/2022/01/08/covid-mortalita-12-volte-piu-alta-per-gli-over-50-non-vaccinati-gazzetta-dasti/) and Cuneo24.it (https://www.cuneo24.it/2022/01/mortalita-da-covid-12-volte-piu-alta-per-gli-over-50-non-vaccinati-in-piemonte-142069/). Reproduced from publicly available material.

Legend - Translation of the text: At the top center: “Piemonte vaccinates you - 12 times higher mortality for unvaccinated individuals over 50 – last 30 days’ data”.

Inside the rectangle: “211 Covid deaths among people aged 50 and over”.

In red: “119 deaths among 205,000 unvaccinated people aged 50 and over”.

In green: “92 deaths among 1.9 million vaccinated people aged 50 and over”.

** not participating in the vaccination campaign or not fully vaccinated.

In the chart shown in Figure 2, first of all, the statement in the yellow box—“211 COVID deaths among people aged 50 and over”—is inaccurate. The correct wording would be “deaths with SARS-CoV-2 positivity” (ascertained at least once over a time interval that, moreover, is never specified). These 211 cases almost certainly include deaths not attributable to COVID-19 (how many, however, cannot be determined from the available information). Consequently, the number of vaccinated and unvaccinated deaths should also be interpreted as “deaths with SARS-CoV-2 positivity.”

However, the most substantial error lies in the design of the figure. The perception of risk – and thus the conditional probabilities P(death | vaccinated) and P(death | unvaccinated) – is driven by the proportions shown in the graph. In the infographic, the yellow circle is disproportionately large relative to the others: while the red circle represents 205,000 people and the green circle 1.9 million, the yellow circle represents only 211 individuals (119 + 92), yet its area is almost the same as that of the red circle. As a result, the visual impression leads the reader to overestimate the relative risk P(death | unvaccinated) over P(death | vaccinated) to an even greater extent than indicated by the numerical values.