Scientific literacy and preferred resources used by Latin American medical students during the COVID-19 pandemic: A multinational survey

Background: This study aims to identify the preferred sources for acquiring knowledge about COVID-19 and to evaluate basic knowledge on critical scientific literature appraisal in students from medical schools located in Spanish speaking countries in Latin America. Methods: We designed an online survey of 15 closed-ended questions related to demographics, preferred resources for COVID-19 training, and items to assess critical appraisal skills. A snowball method was used for sampling. We conducted a descriptive analysis and Chi-squared tests to compare the proportion of correct identification of the concept of a preprint and a predatory journal when considering a) self-perceived level of knowledge, b) public vs private school, c) inclusion of a scientific literature appraisal subject in the curriculum, and d) progress in medical school. Results: Our sample included 770 valid responses, out of which most of the participants included were from Mexico (n=283, 36.8%) and Ecuador (n=229, 29.7%). Participants preferred using evidence-based clinical resources (EBCRs) to learn more about COVID-19 (n=182, 23.6%). The preferred study design was case report/series (n=218, 28.1%). We found that only 265 participants correctly identified the concept of a preprint (34.4%), while 243 students (31.6%) correctly identified the characteristics of a predatory journal. We found no significant differences in the proportion of correct answers regardless of the self-perceived level of knowledge, progress in medical school, or scientific literature critical appraisal classes. Conclusion: This study is novel in its approach of identifying sources of knowledge used by Latin American medical students and provides insights into the need to reinforce training in critical appraisal of scientific literature during medical school.


Introduction
In December 2019, a cluster of cases of pneumonia of unknown origin was reported in Wuhan, China and since then, the virus has spread globally. On March 11, 2020, the World Health Organization (WHO) declared coronavirus disease 19 (COVID-19) as a pandemic. 1 Researchers throughout the globe have worked arduously to rapidly fill in the gaps of knowledge on this novel disease and this has resulted in an unprecedented surge in scientific production. 2 However, the pressure of produce science quickly can cause an overproduction of low-quality research. Notably, before the pandemic, it was estimated that approximately 85% of research was wasted due to flaws in design, methodology or interpretation of the data, and these challenges have been augmented due to the time constraints and poor research infrastructure during the COVID-19 pandemic. 3,4 The role of medical students has been diverse throughout history, from being participatory agents in the evolution of medical curricula to being agents of promotion, prevention and treatment of diseases. 5,6 Medical students have helped during previous disasters and emergencies such as the Spanish flu outbreak of 1918 and the 1952 polio epidemic in Denmark; and they might have an important role during the COVID-19 pandemic through essential tasks such as assistance in clinical care, telemedicine, and fighting against misinformation. 7,8 To provide high-quality healthcare services, future physicians should develop abilities to critically assess research to inform evidence-based decisions during their career, including critical appraisal of scientific literature within the curriculum of medical schools and promoting involvement in research opportunities could help students develop these skills. [9][10][11][12] Notably, the percentage of student authorship in published articles has increased importantly throughout the years. 13 Due to their role, and the unprecedent low quality scientific information, it is indispensable that medical students have enough skills to analyse scientific data. In this study, we intend to identify preferred resources used by medical students in Spanish speaking countries in Latin America to obtain information about COVID-19 and evaluate basic critical appraisal skills through a self-report online survey.

Ethical statement
This study was approved by the Independent Review Board at Universidad San Francisco de Quito (2020-033M). Although data was anonymous and did not include identifying information, electronic informed consent was requested before the survey was collected. Participation was voluntary, and medical students were informed that data from the survey would be used for analysis and scientific publication. Participants did not receive any compensation from participating in the study.

Study design
This was a multinational cross-sectional study based on a self-report online survey. Participants were medical students currently enrolled in a medical school in a Spanish-speaking country in Latin America. The inclusion criteria required for participation were that they were aged 18 or older, with active student status in a medical school based in a Spanishspeaking country in Latin America, and with Spanish language fluency. Exclusion criteria were non-active student status, medical students from a country where Spanish is not the primary language, and previous participation in the survey.
As we intended to include participants from multiple countries and achieve significant variation in characteristics within our purposive sample, a link to the survey and an accompanying standardized text explaining the purpose of the survey was sent via e-mail from the authors to the initial "point of contact" in each country for further distribution following a snowballing approach. The link to the survey was then shared through different social networks by authors to medical students from their own and other medical schools, and authors instructed participants to share the link with their peers. The survey was distributed to students from different academic years to gather a diverse sample in terms of progress in medical school. We also actively tried to contact students from as many countries in Latin America as possible REVISED Amendments from Version 1 In this new version, the methodology has been explained deeper, and more results are analyzed. This includes a new Table 4  that outlines the relationship between percentage of undergraduate studies completed, self-perceived knowledge on  COVID-19 and, source of preference for COVID-19 information. Any further responses from the reviewers can be found at the end of the article through e-mails sent to the medical students' organizations publicly available to ask them to distribute the survey to their members. Data collection took place from October 22 to November 6, 2020. During the first week after the survey was released we obtained a huge amount of the answers, but then there was a very low number of new responses daily, therefore we decided to shorten the period for data gathering.

Survey design
We designed an electronic survey using the Google Forms web application. The survey consisted of fifteen closed-ended questions to gather basic demographic information and evaluate resources used to obtain information on COVID-19 and explore knowledge and attitudes that medical students had in terms of scientific information appraisal. The survey was developed by employing a conceptual framework, in which previous knowledge and experiences that authors had with publications related to medical education, as well as their personal experiences as medical students or faculty, guided the creation of questions. 6,14 Authors were invited to submit potential questions to a shared online document, which then underwent revision through a telematic session by all authors to determine the questions that would be included in the survey. The panel of authors included active students, alumni, and faculty with vast experience in medical education and research.
The survey was developed in Spanish and was reviewed by all authors to ensure that it was appropriate despite regional language variations. In addition, two independent academic advisors reviewed the survey for clarity. At this stage, no changes were required. Then, the survey was piloted in medical students from Ecuador (n=5), Mexico (n=3), and Argentina (n=2), who were mainly friends of the authors, to get their input and identify confusing or unclear questions. These surveys were not included in the analysis. Pretest resulted in minor changes, mostly related to grammar and punctuation After incorporating the feedback, we deployed the survey. As the survey was originally designed and applied in Spanish, we have included a translation to English which was performed by an author (BN) with certified proficiency and then by a native speaker who did not participate in the study (The original version in Spanish and a translated version to English are available as extended data). 14

Statistical analysis
Data captured through the Google Forms web application was exported directly to Microsoft Excel (v. 16.58). An initial descriptive analysis of the responses to the survey to determine demographic characteristics and preferred resources used to gather COVID-19 related information was performed. All variables obtained in the results of the survey were analyzed. There was no missing data as the setting in the online survey did not allow for the submission of incomplete forms. We analyzed the difference in the proportion of correct identification of the concept of a preprint and predatory journals when comparing students based on their self-reported level of knowledge, progress in medical school, public vs. private status of medical school, and inclusion of a scientific literature critical appraisal subject during their curriculum. For data analysis, we decided to group low-medium and advanced-experts' responses in the self-reported level of knowledge and first-second quarter and third-fourth quarter in progress in medical school responses to create dichotomized variables. Due to the categorical nature of our data, we decided to use Chi-squared tests. A one-tailed alpha value of 0.05 was set. All our statistical analyses were conducted on IBM SPSS version 25 (IBM. Corp., Armonk, N.Y., USA).

Trustworthiness
By incorporating different stakeholders in medical education (active junior and senior students, alumni, and faculty), and researchers from different countries, we could design a study that contained different valuable perspectives on the critical appraisal of scientific information.
Although ET is a professor in one of the medical schools where the survey was conducted, participants were not aware of his role in the research. No economic or academic incentive was offered by any of the authors to the participants.
After data gathering, BN and DA performed an initial statistical analysis and shared the results with the other authors. A researcher not involved in this study was invited to validate the statistical analysis. Additionally, all authors had access to the database and agreed with the final data analysis and processing.

Results
There were 770 valid responses to the survey (Underlying data: Survey results-Scientific literacy and preferred resources used by Latin American medical students during COVID-19 pandemic). 14 Almost two-thirds of the participants were females (n=487, 63.2%). Most of the participants were from Mexico (n=283, 36.8%) and Ecuador (n=229, 29.7%). Students equally attended private (n=388, 50.4%) or public universities (n=382, 49.6%). We also found a similar distribution in terms of completed progress in medical school: 24.0% (n=185) in the first quarter, 24.3% (n=187) in the second quarter, 24.9% (n=192) in the third quarter, and the remaining 26.8% (n=206) in the final quarter of medical school. Distribution by gender and country is shown in Table 1.
Most medical students considered that they had a low or medium level of knowledge about COVID-19 (n=539, 70.0%), while 231 participants reported an advanced or expert level (30.0%).
Overall, medical students preferred using evidence-based clinical resources (EBCRs), such as UpToDate or Medscape, to learn more about COVID-19 (n=182, 23.6%), followed by academic journals (n=171, 22.2%), and open access courses provided by entities such as the World Health Organisation (WHO) (n=141, 18.3%). Our survey responses were grouped according to the different sub-regions of Latin America (Table 2).
Regarding the type of article, medical students preferred using case reports or case series (n=218, 28.1%) to learn more about COVID-19, followed by review articles (n=168, 21.8%) ( Figure 1).
The critical factor that medical students considered when reading a scientific article was most frequently the affiliation of the authors (n=366, 47.5%), followed by the journal of publication (n=176, 22.85%), the type of study (n=138, 17.92%), and the publication status (preprint vs. published) (n=44, 5.71%). Data are presented as the number (%). 5 participants answered "other": one from Chile, one from Guatemala, two from Honduras, and one from Mexico.   In terms of receiving news related to the COVID-19 pandemic, medical students preferred official websites of organizations such as the WHO (n=221, 28.7%), followed by scientific journals (n=185, 24.0%), and medical platforms (n=128, 16.6%) (Table 3).
Additionally, relationship between percentage of undergraduate studies completed, self-perceived knowledge on COVID-19 and, source of preference for COVID-19 information were performed (Table 4).
We found that approximately a third of the participants correctly identified the concept of a preprint (n=265, 34.4%), while 243 students (31.6%) correctly identified the characteristic of a predatory journal. On the other hand, 553 students (71.8%) correctly identified the concept of an open-access journal.
Two-thirds of the participants (n=526, 68.3%) had a subject focused on scientific literature critical appraisal as a part of their curriculum during medical school.
We conducted chi-squared tests to identify factors that were associated with a significant difference in the proportion of students that correctly identified the concept of a preprint or predatory journals (Extended data: Frequencies of answers and results of Chi-squares performed). 14 We did not find a significant difference when comparing participants in terms of self-reported level of knowledge (low/medium vs advanced/expert) in the chance of correctly identifying the concept of a preprint (df=1, χ 2 =0.632, P=0.456) or a predatory journal (df=1, χ 2 =0.435, P=0.554).
Additionally, we did not find significant differences in the correct identification of preprint (df=1, χ 2 =0.502, P=0.514) or predatory journal (df=1, χ 2 =0.694, P=0.406) concepts regardless of the inclusion of a scientific literature critical appraisal subject during medical school. Finally, the proportion of participants that answered correctly was not significantly different for preprints (df=3, χ 2 =7.124, P=0.068) and predatory journals (df=3, χ 2 =5.741, P=0.125) despite the progress in medical school (first-second quarter vs third-four quarter).

Discussion
Despite the presence of data from some countries showing an increasing proportion of female medical students, no regional report of the general characteristics of medical students in Latin America has been performed recently. Other cross-sectional studies in Latina American medical schools showed a similar distribution in terms of gender. 15,16 Overall, most medical students perceived that their knowledge about COVID-19 was either low or medium independently of the current academic year they belonged. This differs from what could be expected in previous studies in which students from superior years were more knowledgeable on COVID-19. 17 Medical students who were more advanced in their career (>50%) preferred evidence-based clinical resources (EBCR) for train and skill building. Notable, regardless of selfperceived knowledge of COVID-19, medical students preferred EBCR (Table 4). This result is consistent with previous research that shows the popularity of these resources within the medical community. 15,16 The use of evidence-based electronic resources has been associated with reduced lengths of stay and mortality rates, and it has resulted in improved performances in standardized examinations. 18,19 Additionally, these results represent that with the progress of their career medical students improve their abilities to choose the main source for skill building.
Most of participants preferred getting news related to the pandemic on official websites by entities such as the WHO and the Centre for Disease Control (CDC), which could be a way to combat the current misinformation outbreak (infodemic). Social media platforms provide users with immediate access to massive amounts of content and are the perfect medium to spread questionable information easily. 20 A study showed that fake news is easily spread through Latin American countries through social media platforms, and fact-checking could be a potential solution to fight back against the infodemic. 21 Notably, participants mentioned a preference for case series/reports to learn more about COVID-19, followed by narrative reviews and case-control studies (Table 4). Only 112 participants (15%) preferred systematic literature reviews (SLRs) or metanalyses. The importance of addressing this trend relies on the level of evidence that well-conducted SLRs and metanalysis have compared to case series/reports, which has traditionally been portrayed on the top and the bottom of the evidence pyramid, respectively. 22,23 It would be expected that medical students who had exposure to a subject focused on critical appraisal skills during their formation would choose study designs higher among the evidence pyramid, however, our results did not show a significant difference. These could be partially attributed to the fact that during the start of the pandemic most of the articles published were retrospective studies or case reports focusing on patients' characteristics. 2 Remarkably, we found that regardless of the progress in medical school, self-perceived level of knowledge about COVID-19, and the inclusion of a critical appraisal subject in the curriculum, most of the participants could not correctly identify the concept of a preprint or a predatory journal, despite the enormous surge in preprint production during this pandemic. 24 As well, in our study, only 44 participants (5.71%) chose the publication status (preprint vs published article) as a critical factor when reading an article. Overall, regardless percentage of career completed, or self-perceived level of COVID-19 knowledge medical students take into consideration as main critical factor to read an article endorsement from institutions (Table 4). This is concerning as medical students might fail to recognize that preprints are preliminary reports that have not been peer-reviewed and might contain faulty or low-quality information. 25 Additionally, as medical students are more frequently involved in research, they could be potential targets for predatory journals that promise easy and rapid pathways to publish, through unethical practices, which could further contribute to the infodemic. 13 As expected, predatory journals have exploited the uncertainty and prolonged reviewing and editing processes by multiple scholarly journals due to the pandemic to increase their profits. [26][27][28] A previous study conducted in the Kingdom of Saudi Arabia and New Zealand also revealed a poor understanding of medical students regarding predatory journals. 29 The main limitation to our study is the imbalanced sample, which resulted in multiple countries having few participants. This could be related to the short time that the survey was available or ineffective sampling strategies for some specific countries. The chain-referral sampling method employed, despite its pragmatism, is a significant limitation, as the sampling is non-random due to the referral process, in which the sample is dependent on the researchers' contacts, which is reflected by the significantly higher proportion of participants from Mexico and Ecuador. As the survey was available online, of course there was the possibility of participants outside the target population to complete it, however, this bias was dismissed by employing a chain referral model among medical students. As well, there is uncertainty of whether the sample is representative of the target population. Furthermore, participants could be susceptible to subject bias and might have elected responses that seemed more appropriate for a medical student.
This study is novel in its approach to identify sources used by Latin American medical students to learn about COVID-19 and to evaluate critical appraisal skills used to interpret new scientific information. Additionally, even though we had difficulties in sampling medical students from certain nationalities, we did recruit an equative number of medical students representing different stages of medical degree completion. Our results support the need to reinforce training in critical appraisal of scientific literature during medical school. 30,31 Conclusions Throughout history medical students have an important role, and the COVID-19 crisis is not an exception. Our study found that most of medical students have good behaviors regarding acquire new information related to COVID-19 such as using evidence-based clinical resources to learn about COVID-19 and official websites of recognised organizations to receive news. However, they have problems in terms of correctly identifying predatory journals or preprint articles independently of the region, self-perceived level of knowledge, progress in medical school, or literature critical appraisal classes. Besides, these students prefer a study design that is not at the top of the evidence pyramid. Our findings could provide insights into the need to reinforce training in critical appraisal of scientific literature during medical school in Latin America.

Data availability
Underlying data Open Science Framework (OSF). Scientific literacy and preferred resources used by Latin American medical students during COVID-19 pandemic. https://doi.org/10.17605/OSF.IO/7MS64. 14 This project contains the following underlying data: • Survey results-Scientific literacy and preferred resources used by Latin American medical students during COVID-19 pandemic. (These data contain the answers obtained from the surveys that were applied in medical students about scientific literacy and preferred resources).

Fernando Spilki
One Health Laboratory, Feevale University, Novo Hamburgo, Brazil The issue of information quality proved to be extremely relevant in the context of the COVID-19 pandemic, especially during the first year. An infinity of obscure sources of information, including from social networks to a profusion of material published without any type of quality review, gave the opportunity for untruths to be raised to the level of scientific knowledge, greatly harming the fight against the pandemic in many scenarios.
In this sense, the study becomes relevant and very interesting to assess the levels of understanding of medical students at different levels of training, about what they consider reliable sources of information, and how they judge dubious sources.
From the point of view of minor points that require further investigation, it would be interesting to clarify and discriminate more clearly how students have judged the diverse types of information sources, depending on their level of completion of the medical curriculum, from the beginning to the end of graduation, and how this does evolves or not throughout.

Are sufficient details of methods and analysis provided to allow replication by others? Yes
If applicable, is the statistical analysis and its interpretation appropriate? Yes included active students, alumni, and faculty with vast experience in medical education and research." Is this an expert panel that discusses the survey questions? Or, is this an informal chat between the authors and all the other stakeholders? Accordingly, it is unclear what the purpose of pilot testing on the medical students is. The authors mentioned that they wanted to get "input" from the piloted medical students. However, it is unclear what "input" the authors wanted to get. I suspect that the authors did not provide proper instructions to these students and thus they only obtained some minor comments. Also, it is unclear what the background of these students was (e.g., their gender, their studying progress).
The authors only used chi-squared tests for all the inferential statistics. Only using chisquared tests has the problem of not controlling potential confounders. Given that the sample size was large, the authors should be able to apply multivariable statistics that control potential confounders (e.g., multivariable logistic regression).

2.
I wonder how representative the sample was. Specifically, does the demographic distribution of the present sample similar to these surveyed countries? I found that it is interesting to have a higher proportion of females.

3.
Following the previous comment, I wonder how the authors ensure if the participants were the target participants. For example, maybe a student who was not a medical student received the email and completed the survey. Can the authors catch these invalid responses?

4.
Given that health literacy on COVID-19 is the key information in the present study, I think that the authors should examine other potential factors on the COVID-19 literacy. Specifically, why don't the authors examine if the resources obtaining COVID-19 knowledge is a factor explaining the student's COVID-19 knowledge? Similarly, it would be interesting to examine if different types of articles the students read play a role in their COVID-19 knowledge.

5.
I disagree with the statement "The main limitation to our study is the sample size". Specifically, the entire sample size is good, and the problem is that the authors had an imbalanced sample size. After all, the authors did not separate the participants into different countries for the data analysis. Therefore, the authors did not have the problem of a small sample size in statistics.

Are sufficient details of methods and analysis provided to allow replication by others? Partly
If applicable, is the statistical analysis and its interpretation appropriate? Partly I wonder how representative the sample was. Specifically, does the demographic distribution of the present sample similar to these surveyed countries? I found that it is interesting to have a higher proportion of females.

○
As was described in the Results section, according to our knowledge, the distribution of the data is consistent with other Cross-Sectional studies performed in Latin America Region. Therefore, yes, our sample was representative. In the same sense, we initially had the same concern that reviewer 1 regarding the gender distribution. However, once again, in most of these kinds of studies, females are more active in response.
Following the previous comment, I wonder how the authors ensure if the participants were the target participants. For example, maybe a student who was not a medical student received the email and completed the survey. Can the authors catch these invalid responses?
○ It is important to mention that although the collection of data was performed through an online survey, and the risk of someone outside the target population might answer the survey, we consider that this risk was reduced by employing a chain-referral method with other medical students. Given that health literacy on COVID-19 is the key information in the present study, I think that the authors should examine other potential factors on COVID-19 literacy. Specifically, why don't the authors examine if the resources obtaining COVID-19 knowledge is a factor explaining the student's COVID-19 knowledge? Similarly, it would be interesting to examine if different types of articles the students read play a role in their COVID-19 knowledge.

○
To explore other factors that could contribute to Covid-19 literary among medical students, we perform an additional descriptive analysis where we make a relationship between the percentage of undergraduate studies completed, self-perceived knowledge on COVID-19, and source of preference for COVID-19 information that is summarized in Table 4 and discussion section. This analysis was performed to study if the behavior among medical students changes along with the progress of their career or the level of auto perception in knowledge choosing better sources to acquire medical knowledge.
I disagree with the statement "The main limitation to our study is the sample size". Specifically, the entire sample size is good, and the problem is that the authors had an imbalanced sample size. After all, the authors did not separate the participants into different countries for the data analysis. Therefore, the authors did not have the problem of a small sample size in statistics.

○
We agree with Professor Chung-Ying Li that the correct term is not a sample size, instead, we change the term to imbalanced sample size.
Competing Interests: none