A controlled study on an instrument that couples active learning

Active learning strategies and the use of technology in Background: classes have been widely indicated to enhance learning. Although much has been discussed on these topics, few studies have addressed them with adequate experimental designs. Therefore, this study investigated the effect of a strategy coupling active learning methodology and technology –video lectures production by students – on the students’ learning in comparison with traditional approaches. To investigate the impact of video production on students’ Methods: learning, approximately half of one class of undergraduate students in a Pharmacy program attended traditional classes on one of its modules, while the other half was instructed to elaborate video lectures about the same content. We recorded their scores in two exams on the topic covered by the video lectures, one prior to intervention and the second after the intervention. We also recorded their score in a final exam at the end of the course, which covered all modules in the course, and applied a questionnaire to assess students’ perceptions about the applied methodology. The average score of the students in the video group became Results: 46% higher than the control group’s score. The score on the final exam at the end of the course showed no difference between groups. Most of the students reported that the video lectures they produced in class improved their academic performance. The video lecture production activity, a teaching instrument Conclusions: that relies on active learning and technology, was able to improve learning indicators of a group of randomly selected students in comparison with a control group of students who attended traditional expository classes given by an instructor.


Introduction
Teaching approaches that involve active learning lead to greater motivation to study and increased interaction among students in comparison to expository class given by a professor (Conway et al., 2010;Gleason et al., 2011). Other possible outcomes, such as an increase in learning, have not been unequivocally confirmed so far. Some studies have shown that active learning is no more effective than passive approaches in increasing the level of learning (Conway et al., 2010;Haidet et al., 2004), whereas other studies have shown that some active teaching strategies do increase the level of learning (Letassy et al., 2008). This discrepancy seems to depend on the context in which the method is applied and measured, as well as on the control groups. One factor that may explain such contradictory findings on the impact of active strategies on learning levels is the academic background of students (Haak et al., 2011). There are diverse active learning strategies (Gleason et al., 2011), such as team-based learning (Conway et al., 2010;Parmelee & Michaelsen, 2010), case-based learning (Tayem, 2013), learning by discovery (Spencer & Jordan, 1999), inquiry-based learning (Banchi & Bell, 2008), web-based study activities (Alonso et al., 2005), collaborative and cooperative learning (Smith & MacGregor, 1992), evidence-based teaching/ learning (Kenyon et al., 2016), and problem-based learning (Barrows & Tamblyn, 1980;Hmelo-Silver, 2004;Kilroy, 2004). Many publications addressing active learning strategies focus on presenting their theoretical benefits, describing the methodology used, and making recommendations (Parmelee & Michaelsen, 2010;Passos et al., 2006); however, most of them do not show actual data to support them. Therefore, more studies are needed to fully understand the impact of active learning and teaching on students' performance.
Many publications have explored the use of new teaching technologies and methods (Cubas Rolim et al. 2017;Oliveira et al., 2010;Pandza & Masic 2013;Pertry et al., 2014;Sé et al., 2008). Technological tools can provide notable advantages, giving students flexibility in choosing when and where to study, and increasing their access to information (Fernandez et al., 2014;Kenny, 2002;Weed et al., 2014). Potentially beneficial technologies include: videos used to summarize information at the end of a class in place of a questionnaire or slide review (Sarikcioglu et al., 2011), distance education to promote active learning (Cravener, 1999), medical simulations to build student confidence in clinical practice (Keegan et al., 2012;Reilly & Spratt, 2007), and seminars to improve interactivity among students (Brunton et al, 2000). However, few studies with adequate control groups have investigated the impact of new educational technologies on levels of learning.
Both active learning and new teaching technologies may provide some benefits, but adequate assessments of their impact on learning levels are rare. Even less is known about the effect of combining active learning strategies with technological tools. To form a better view of the effect of active learning coupled to technological tools on learning, we instructed undergraduate students to produce video lectures and compared their academic scores with students that did not produce video lectures. In addition to measuring their academic performance in the course, we also assessed their degree of motivation and satisfaction derived from this learning approach using a questionnaire.

Experimental design
This study involved 50 students attending a Pharmacognosy course during the fifth and sixth semesters of a Pharmacy program at the Integrated Faculties of the Educational Union of Planalto Central (UNICEPLAC, Brasília, Brazil). To be included in the study, the students had to be enrolled in the Pharmacognosy course as part of the Pharmacy graduate program at UNICEPLAC. Therefore, students enrolled in other programs or not enrolled in the Pharmacognosy course were excluded from the study. The study size was determined by the number of undergraduate students that met the inclusion criteria and agreed to participate in the study; all eligible students were recruited as follows. At the first class of the Pharmacognosy course, the students were informed about the intention to conduct the study on the use of active learning methodologies coupled with technological tools. At this moment, written informed consent was obtained. At the second class, students who were not present in the first class were informed, also agreed to participate in the study and signed the informed consent. This course covers several topics related to the use of active substances of plant and animal origin during one semester with a total workload of 54 hours. Three weeks after the beginning of the course, the professor applied a written test to evaluate the students' previous knowledge about the use of herbal medicines, a topic often discussed in the media and covered in previous courses in the Pharmacy program.
To investigate the impact of video production on students' learning, motivation, and satisfaction, students were randomly divided into two groups of equal size. At the fifth class, the students were randomized by placing each student who was arriving at the classroom alternately in one of the two experimental groups. Although no attempt was made to match cases and control, the two groups had similar demographic characteristics (p > 0.05; Table 1). Both groups studied the use of herbal medicines module for ten hours in separate rooms using different learning strategies. The control group continued to study in the way they had done previously, by listening to a fourhour lecture given by the professor using visual resources; in this group, students were encouraged to participate through questions and comments. The video (intervention) group was divided into seven teams with a maximum of five students that were instructed to produce a video lecture on the use of herbal medicines. These students worked together, without the supervision of any instructor, using books, online resources, cameras, cell phones, and computers to create their videos (see extended data for videos (Nascimento et al., 2019b)). As the use of herbal medicines module is equivalent to approximately 20% of the course content, students in the video group experienced 20% of the course using a different methodology from the control group.
After studying the use of herbal medicines, all students were once again assessed on their knowledge of the field through a second written test. Then, the whole class watched the video lessons produced by students and filled out a questionnaire (extended data (Nascimento et al., 2019b)) on their preferred information sources and the degree of motivation and satisfaction they had experienced while studying using the two different learning methods. A third test (the final exam; see extended data (Nascimento et al., 2019b)) was applied at the end of the semester and evaluated all topics covered by the course. Both groups were evaluated by the same tests.

Ethics and consent
This research project was approved by the Research Ethics Committee registered through the Research Ethics Committees (CEPs, acronym in Portuguese) and the National Research Ethics Commission (CONEP, acronym in Portuguese) also known as the CEP/CONEP system (approval number 13930013.6.000 0.5058/244,173). Written informed consent from all subjects involved was obtained for participation in the study and subsequent publication of the videos they produced.

Statistical analysis
Scores of groups before and after the intervention were compared using the Wilcoxon's test. Differences between scores within each group were analyzed using the Mann-Whitney test. The homogeneity of the qualitative variables between the "Control" and "Video" groups was verified using Fisher's exact test. All analyses were performed using R software (version 3.0.1).

Learning indicators
Students were evaluated twice on the use of herbal medicines in order to verify the extent to which producing their own video lectures affected their learning. In the first evaluation, before any learning method had been used, the students in the video group had an average score that was 13% lower than that of the control group (p < 0.05). After studying the use of herbal medicines in two different ways, the average score of the students in the video group had become 46% higher than the control group's score (p < 0.001; Figure 1). The control group's score decreased and the video group's score increased between their first and second evaluations (p < 0.01). The final exam at the end of the course, which included all content covered during the semester, showed no difference between the scores of the video and control groups (p > 0.05; Figure 2). See underlying data for results for each student (Nascimento et al., 2019a). Figure 1. Effects of two different methods of teaching "the use of herbal medicines" module on test performance. Students in the video group answered fewer questions correctly than the control group on the first test, but more questions correctly on the second test. The asterisks denote significant differences between the first and the second exams within the same group (p < 0.01).

Students' perceptions and preferred sources of information
The results obtained from the questionnaire show that students use online resources on the Internet as their main source of information ( Figure 3). They habitually watch video lectures, and most of them believe that these videos improve academic performance (Table 2). When asked who or what factors had encouraged them to watch videos, most cited their professors. In order to investigate the factors influencing the students' perception that video lessons improve learning, students were asked to rate various factors. None of the suggested factors were perceived as responsible for their academic performance improvement (Table 3). When asked which learning methodology they preferred, most preferred the traditional approach, which they were used to ( Table 2). Most of the students (92 %) in the video group reported that producing video lectures during class as a studying strategy stimulates learning when compared to only watching them (Figure 4).

Discussion
In this study, we investigated the impact of a new teaching instrument that combines active learning with the use of videos by enabling students to create their own video lectures. We observed that levels of knowledge on the topic covered in the videos rose in the group of students who produced videos, in comparison with those who attended expository classes given by an instructor. This finding was reinforced by the opinion of the students, who believed that this instrument improved their performance. This research also revealed students' great attraction to the Internet and related technologies. Most students believe that simply watching video lessons (not produced by themselves) stimulate and improve learning. Interestingly, the greatest motivator for the use of video lectures mentioned by the students was their own instructors.
The observed increased learning supports the idea that active learning can be more beneficial than relatively passive learning methods. Other studies measuring learning have also demonstrated that active learning methods can increase knowledge (Wiecha et al., 2006). However, other studies using adequate control groups have not found any learning differences between active and more passive learning methods (Haidet et al., 2004). One factor that may explain this discrepancy is the students' own educational background. Active learning has been shown to provide more benefits for students with weaker academic background than for those with more knowledge (Haak et al., 2011). The students in our research project were enrolled in a private institution, which in Brazil is expected to have students with lower scores in pre-university academic exams than those in enrolled in public universities. Their marked increase in learning through the creation of videos shows that they were well suited to benefit from this active learning instrument. Nevertheless, our results do not allow us to make the generalization that active learning always leads to greater learning gains for every group of people.
There is evidence that, for some groups, active learning methods are no more efficient than more passive methods. One reason is that the traditional approach (expository lectures) varies greatly from professor to professor. Instructors who are very skilled in establishing empathy, creating interest, and offering very popular classes might achieve better learning indicators from the students than those who do not have such abilities (Francis & Tannuri-Pianto, 2012;Sampaio et al., 2011). Indeed, improving the quality of teacher-student interaction causes a sharp increase in student learning (Allen et al., 2011). Even with the successful implementation of a more active system of learning, low professor availability causes great dissatisfaction among students (Gahutu, 2010). It is therefore assumed that students of instructors who are experts in engaging activities, such as public speaking and story-telling (Latey, 2000;Leach, 2005;Stein, 2009) learn more than the students of professors who do not teach in this way. This could be one explanation for the low learning rates associated with purely lecture-based teaching observed in some studies (Cook et al., 2008).
Our student participants reported that the Internet was the learning instrument they used the most. This reflects the increasing use of technology (Chaudhry et al., 2010;Erickson et al., 2010;Giordano & Giordano, 2011). However, our students' high reliance on the Internet may also indicate that they could be less well educated than some others. The Internet might provide incorrect or false information (Barrie & Presti, 1996;Fraval et al., 2012), including many videos that contain misleading or improper content (Murugiah et al., 2011;Sood et al., 2011;Tourinho et al., 2012). Our results showed that the student participants used books less often than they used the Internet, which may indicate some deterioration in the quality of their learning. Specialized health professionals, even with ample access to the Internet, still use books as their main source of information for solving medical problems (AlGhamdi, 2009). On the other hand, most (59%) of highly qualified European doctors, 99% of whom have regular Internet access, report using article research databases like PubMed to access scientific content (Kritz et al., 2013). Considering the reliability problems of Internet data and the dominance of advertising (Hossler & Conroy, 2008;Jenkin et al., 2014;Vance et al., 2009), it is important to assess the causes and outcomes of our students' tendency to avoid books.
Despite many problems with Internet use, there are some obvious advantages, such as the availability of information. However, statements about the potential benefits of using the Internet or new technologies are often unsubstantiated by data and based instead on opinions (Carlon et al., 2012;Pandza & Masic, 2013;Weed et al., 2014). Most studies have focused on factors such as satisfaction rather than on learning (Curran & Fleet, 2005;Kerfoot et al., 2006;Narula et al., 2012). Some studies have shown that the Internet has a positive impact on learning, but many of these had no control group or an inadequate control group (Karpa, 2012;Oliveira et al., 2010;Pertry et al., 2014;Rangel et al., 2010). This makes it impossible to identify the most efficient form of teaching/studying for increased learning (Cook, 2009;Kleinpell et al., 2011;Lantz, 2010;Steele et al., 2009). In contrast to expectations, comparative studies with an adequate control group, such as lectures given by highly skilled instructors, often reveal that  there are no differences between technology-based and traditional learning methods (Chao et al., 2012). We would expect any form of teaching to provide benefits when compared with the absence of a stimulus (Cook, 2009;Lipscomb et al., 2009). Indeed, we found that producing video lectures enhance learning on a specific topic (the use of herbal medicines) of a Pharmacognosy course. Additionally, the present study showed that most students (from both groups) believe that watching video lectures improve their academic performance and those who had produced video lectures (video group) believed that producing video lectures stimulates learning.
In conclusion, this study shows that a new teaching instrument that combines active learning with the use of technology through the creation of student-designed video lectures was able to increase the learning of a group of randomly selected students in comparison with a control group of students who attended traditional expository classes given by an instructor. This result corroborates many articles that have theorized that active learning is better than passive learning, and that using videos and other new technologies can increase learning. To date, few studies with adequate control groups have been able to confirm these theories.  et al., 2019b) This project contains the following extended data:

Grant information
The author(s) declared that no grants were involved in supporting this work. This paper highlights an important issue in online instruction: active learning and technology and how they can be merged to fit into the current learning environment. As the authors state, there is a need for further investigation of this intersection. This paper attempts to fill in the gaps in this space. The video production intervention provides an example of how instructors can incorporate meaningful active learning into their classrooms. The results showing improved learning outcomes for the intervention group is promising and aligns with much of the literature supporting student-centered learning. Is the work clearly and accurately presented and does it cite the current literature? The study is presented clearly and accurately. However, the title may be misleading because technology may not be the only independent factor in this experiment. The learning objective of the intervention and control arms differ on Bloom's taxonomy of learning. The video production intervention is higher on Bloom's taxonomy than the undirected self-study in the control group. Creation, as well as the other higher learning objectives on this taxonomy, has been linked to improved learning outcomes. This possibly confounds the intervention because the higher order learning objective may be responsible for the learning gains, not necessarily the engagement of technology. If the students in the intervention arm were asked to produce something on paper instead of video, Bloom's taxonomy would suggest that similar learning gains would likely result.
The manuscript cites many relevant studies but it would be strengthened by including more recent studies that address some of the gaps that the authors mention. For example: "This result corroborates many articles that have theorized that active learning is better than passive learning, and that using videos and other new technologies can increase learning. To date, few studies with adequate control groups have been able to confirm these theories." Reference to Mayer's multimedia learning principles could fill in part of the aforementioned gap. Richard Mayer is a cognitive psychologist at the University of California, Santa Barbara and whose research focuses on learning outcomes in multimedia learning with controlled psychological testing environments. The manuscript could be improved by including the findings on Mayer's handbook on .

Multimedia Learning
If applicable, is the statistical analysis and its interpretation appropriate?
The statistical analysis and its interpretation are appropriate, but the sample size is small. Increasing the The statistical analysis and its interpretation are appropriate, but the sample size is small. Increasing the sample size is critical for improving the statistical power of this test. Effect size comparisons for each group could be added because the sample data is paired.
Are the conclusions drawn adequately supported by the results?
The conclusions are drawn adequately but additional clarification is required. The control group shows a high first exam score followed by decreased second exam score, while the intervention group shows a lower first exam score followed by an increased second exam score. This finding possibly aligns with the findings of a 2006 study by Roediger and Karpicke, Test-Enhanced Could the authors consider this paper and its implications in the discussion? Learning. Figure 3 indicates an important shift in the learner's experience and the increasing use of the internet for study. This is an area that is important for discussion and further investigation. The questionnaire data revealed that a majority of the students were using the internet as a studying resource and that they believed watching the videos were beneficial for their learning. But it seems that students are also aware that more complex learning objectives can increase learning outcomes. Looking at the dataset, it seems that many students from the control group also indicated yes to the question shown in Figure 4. Yet, Fig. 4 only shows the responses from the students in the video group. Could the authors please clarify why the control group responses were not included here?
Could the authors please provide more information on the instructions given to the video production group?
Could the authors please provide more information about the four-hour lectures given by the professor using visual resources? Did the video production group also listen to the lecture? What were the visual resources? Did both groups receive these visual resources?

If applicable, is the statistical analysis and its interpretation appropriate? Partly
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Partly
No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: Life Science Education, Evidence-Based Teaching, Assessment We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com