F1000Research F1000Research 2046-1402 F1000 Research Limited London, UK 10.12688/f1000research.15799.1 Research Article Articles INCISION e-learning program as a useful teaching tool to enhance surgeons’ knowledge and skills: An Indonesian multi-center cross-sectional pilot study

[version 1; peer review: 1 approved with reservations]

 Harzif Achmad Kemal Conceptualization Funding Acquisition Project Administration Supervision Writing – Review & Editing https://orcid.org/0000-0002-3552-1610 a 1 Rahman M Nurhadi Conceptualization Supervision Writing – Review & Editing 2 Mulawardhana Pungky Conceptualization Project Administration Supervision Writing – Review & Editing 3 Shafira Nadia Investigation Methodology Project Administration Resources Writing – Original Draft Preparation 4 Anggraeni Tricia Dewi Supervision Writing – Review & Editing 5 Sumapraja Kanadi Supervision Writing – Review & Editing 1 Ocviyanti Dwiana Supervision Writing – Review & Editing 6 Division of Reproductive Endocrinology and Infertility, Department of Obstetric and Gynecology, Faculty of Medicine, Universitas Indonesia, Dr.Cipto Mangunkusumo General Hospital, Central of Jakarta, DKI Jakarta Province, Indonesia Department of Obstetrics and Gynecology, Faculty of Medicine, Gajah Mada University, Sarjito Hospital, Jogjakarta, Jogjakarta Special Region, Indonesia Department of Obstetrics and Gynecology, Faculty of Medicine, Universitas Airlangga, Dr. Soetomo Hospital, Surabaya, East Java, Indonesia Department of Obstetric and Gynecology, Faculty of Medicine, Universitas Indonesia, Dr.Cipto Mangunkusumo General Hospital, Central of Jakarta, DKI Jakarta Province, Indonesia Division of Gynecologic Oncology, Department of Obstetric and Gynecology, Faculty of Medicine, Universitas Indonesia, Dr.Cipto Mangunkusumo General Hospital, Central of Jakarta, DKI Jakarta Province, Indonesia Division of Social Obstetrics and Gynecology, Department of Obstetric and Gynecology, Faculty of Medicine, Universitas Indonesia, Dr.Cipto Mangunkusumo General Hospital, Central of Jakarta, DKI Jakarta Province, Indonesia kemal.achmad@gmail.com

No competing interests were disclosed.

 3 12 2018 2018 7 1888 26 11 2018 Copyright: © 2018 Harzif AK et al. 2018 This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Background: Media aids are one of the most important components in the teaching and learning process. This pilot study program was conducted in order to assess the effectiveness of the INCISION e-learning program as teaching media in the surgical teaching and learning process, and its ability to improve surgical skills and knowledge achievement.

Methods: One intervention group and one control group were involved in this study. The intervention group used the hysterectomy INCISION e-learning module, while the control group used conventional teaching approaches. The study was conducted with 14 resident surgeons in three universities in Indonesia: Universitas Indonesia, Universitas Airlangga, and Universitas Gajah Mada. The testing components used were a pre-test, post-test questionnaire (a modified Ritzman questionnaire) and direct observation of procedural skills in the operating room (OR). Data were analyzed descriptively using Mann-Whitney and Wilcoxon tests.

Results: Using a Mann-Whitney test, we found the differences between the average scores of the intervention group and the control group to be statistically significant ( p=0.046). A Wilcoxon test also revealed significant differences ( p=0.028). The modified Ritzman questionnaire also revealed that the residents in the intervention group felt more confident in their surgical knowledge (82%), and made more efficient use of their time in the OR (81%).

Conclusions: These findings reveal a significant improvement in knowledge and skill achievement in residents that underwent training via the INCISION e-learning module, compared to residents taught via conventional teaching strategies.

 INCISION e-learning surgeon skills knowledge The author(s) declared that no grants were involved in supporting this work. Introduction

The use of multimedia as a learning tool is one of the best educational techniques as it is able to engage more than one sense simultaneously, generally the senses of sight and hearing. Multimedia programs provide a variety of different stimuli, including elements of text, speech, sound and music, graphics, animations and still pictures 1– 3 . In undergraduate and graduate medical training, the type of teaching media used depends on the institution as well as the individual teacher and the subject matter being taught 4, 5 .

Traditionally, medical teachers explain theories and demonstrate procedures, followed by practice by the trainees (“see one, do one”) 3, 6 . There are four teaching approaches to surgical education including: standardized/simulated patients; procedure courses, videos, textbooks; web-based training; cadavers and live animals 7 . Nowadays, technological development has influenced the ways in which learning and information presentation takes place, with a variety of technological tools now supplementing and partly replacing paper books 2, 3, 6, 8– 11 . The INCISION e-learning module is a new learning and teaching approach, comprised of an online learning platform designed to transfer procedure-specific knowledge to surgeons, gynaecologists and residents. INCISION also provides some information on pre- and postoperative care; however the primary focus is on procedure and the relevant surgical anatomy.

In this pilot study, we sought to evaluate the effectiveness of the INCISION approach on the transfer of relevant surgical knowledge, as well as to assess the strengths and weaknesses of e-learning via INCISION from the point of view of the surgery resident.

 Methodology Study design

The INCISION pilot study program involved 14 surgeon residents specializing in obstetrics and gynecology (OBGYN). Inclusion criteria included residents in 3–6 th semester who had never been trained for hysterectomies, while exclusion criteria included participants who were not willing to participate in this study or had incomplete filling of the questionnaire.

We divided residents into an intervention group and a control group by random number generation, consisting of 7 surgeon residents in each group. Recruitment of the participants was done by asking residents in 3–6 th semester in person on July until August 2015 during the break after class at three universities: Universitas Indonesia, Universitas Airlangga, and Universitas Gajah Mada.

The intervention group used the hysterectomy INCISION e-learning module, while the control group used conventional teaching approaches. The distribution of participants is shown in Table 1.

Resident surgeon distribution.
Control Group (n=7) Intervention Group (n=7)
N % N %
University
Universitas Indonesia 3 42.9 3 42.9
Universitas Airlangga 2 28.6 2 28.6
Universitas Gajah Mada 2 28.6 2 28.6

This study was approved by the Ethics Committee of Faculty of Medicine, Univesitas Indonesia, on July 6th 2015 (reference number: 564/UN2.F1/ETIK/2015). Permission was obtained to perform this study in all three sites. Written consent for participation was obtained from all study participants.

 Time and location

This study was conducted between July and August 2015, in the medical facilities and hospitals of: Universitas Indonesia, Universitas Airlangga and Universitas Gajah Mada.

 Testing instruments

Four evaluation methods were used. First, we included a pre-test with the purpose of evaluating the residents’ knowledge of procedures prior to training ( Supplementary File 1). Second, a post-test was conducted after having received the course material ( Supplementary File 1). Third, a Ritzman questionnaire was administered to evaluate the residents’ perception of the INCISION approach on the training material, and how it impacted their knowledge ( Supplementary File 2). The Ritzman questionnaire was not taken by the control group, only the intervention group. Every question is rated on a score of 1 to 7, which when converted into a percent is able to be grouped as >80% = good, 50-79% = average, and <50% = bad.

Finally, residents performed the procedure under the supervision of a qualified trainer, which was not assessed.

In-depth interviews were also conducted with the intervention group for assessment value of INCISION e-learning. Residents were interviewed by the research team in the Obstetrics and Gynecology Department's Meeting Room. The research team wrote notes during the interview only. The residents were asked about added value of using INCISION e-learning; weakness, difficulties or obstacles encountered during the study with INCISION e-learning; the difference between INCISION e-learning and conventinal teaching; and whether the residents will recomend INCISION e-learning for their friends.

 Procedure

Residents participating in the study were given a pre-test during the first week of the study period. This test served as an entry exam to gauge their incoming level of expertise with the particular procedure. Starting in the second week, the intervention group of residents commenced learning via the class with INCISION e-learning 2D module, while the control group of residents were taught via presentation methods.

On the third week, trainers and residents from the intervention group took the class with INCISION e-learning 3D module and discussed it together, while the control group watched the trainer and followed the operation of a patient. Finally, all residents conducted the post-test. The residents took the exam at least once during the week after the study period.

Supplementary File 3 contains information about the conventional course.

 Data analysis

Data were analyzed descriptively using SPSS 21 version statistical software, using Mann-Whitney and Wilcoxon tests for statistical analysis. We used a non-parametric test due to the small sample size. The Mann-Whitney test was used to examine the differences in knowledge based on the pre-test examination. The Wilcoxon signed-rank test was also used to evaluate the difference between courses, using data from a paired-sample design. We used the Wilcoxon test to determine whether there was a difference between the intervention group and the control group at the end of the pilot program, based on the administered post-test. A p value <0.05 was considered statistically significant.

 Results

All subjects completed the study; no participants declined to be part of the study.

Figure 1a shows the score distribution of the pre- and post-tests in the control group. Five residents decreased their score and two showed an increase. However, in the intervention group, the majority of residents increased their score, as shown in Figure 1b.

Distribution of pre- and post-test scores from ( a) the control group and ( b) the intervention group.

We evaluated differences between the two groups’ average pre-test scores in order to determine whether there were any differences in knowledge and skill prior to intervention. Using a Mann-Whitney test, we demonstrated that the average value of the pre-test in the control group was not significantly different from that of the intervention group (p=0.561;).

Further evaluation of the average post-test scores between the control and intervention groups was important to determine whether there were differences in knowledge and skill following intervention. Using a Mann-Whitney test, we found that the average score of the intervention group (67), was significantly greater than that of the control group (53). This difference was statistically significant (p=0.046).

We then compared the pre- and post-test results within each group. The average pre-test score in the control group was 52.71, while the average post-test score in the control group was 52.71. There was no difference between the pre- and post-test results in the control group. The average pre-test score in the intervention group was 50.14, while the average post-test score was 66.93. A Wilcoxon test revealed that this difference was significant (p=0.028).

Residents in the intervention group were satisfied with the training outcome, with an average Ritzman questionnaire score of 81% ( Table 2). The highest score, 86%, was given for the statement that residents believe the content of INCISION is useful for their job. On average, residents felt that they appreciated the course (82%), that the learning atmosphere was encouraging (78%), that the learning was fun (73%), and that they obtained beneficial knowledge from the course (80%). The residents also felt that the INCISION e-learning was comprehensible (82%). The content and the language (foreign words and technical terms) was also found to be easy to comprehend (86 and 83%, respectively). They felt that they kept-up thematically with the course (82%), and that the time spent was sufficient for the theme covered (80%).

Ritzman questionnaire analysis.
Ritzman Questionnaire Average Score/item
Training Outcome
Overall, I like the course 82
The learning atmosphere was agreeable 78
The learning was fun 73
I find the approach useful for my job 86
Investing time in the course was useful 82
I can apply the content of the course in my job 88
I derive personal use from the course 80
Average 81
Comprehension
The contents were comprehensible 86
The language (foreign words and technical term) was comprehensible 83
I kept up thematically in the course. 82
The time was sufficient for the theme covered. 80
Average 83
Knowledge gain
I have the impression that my knowledge has expanded on a long-term basis 82
I will be able to remember the new themes well 78
I think that I will bstill be able to report what I learned some time after the course. 71
I will appl what I learned to my day-to-day work 76
I would recommend the INCISION approach to my collegues 80
Average 77
Media
The 2D film were helpful for my understanding 76
The 3D films were helpful for my understanding 90
The online Academy was helpful for my understanding 88
The 3D film was suitable for presenting the contents 94
The 2D film was suitable for presenting the contents 80
The online Academy was suitable for presenting the contents 80
Average 84
Customized additions:
Training outcome
I feel more confident in my surgical knowledge after following the INCISION Approach 82
I feel I made more efficient use of my time in the OR after following the INCISION approach. 81
My understanding of the procedure was helped by the step-by-step approach. 88
I feel patient safety is increased due to the INCISION approach 82
Average 83
Product Feedback
There needs to be more images / stills in the Academy. 80
There need to be more videos in the Academy 78
There need to be longer videos in the Academy 84
Average 80

Additionally, residents in the intervention group felt that INCISION e-learning provides an adequate gain in knowledge (77%). They had the impression that their knowledge had expanded on a long-term basis (82%), that they would be able to remember the new themes well (78%), and that they think they will be able to sufficiently report what they had learned some time after the course (61%). They also expressed that they will apply what they learned to their day-to-day work (76%), and that they would recommend the INCISION approach to their collegues (80%). Overall, the residents rated the INCISION media as good (84%). They preferred 3D film (90%) over 2D film (76%), for ease of understanding the content. With respect to content presentation, they also found 3D film (94%) to be more suitable than 2D film (80%). They felt that the online academy both aided understanding (88%), and was an acceptable medium for presenting the content (80%).

The residents also felt more confident in their surgical knowledge (82%), and made more efficient use of their time in the OR (81%) after following the INCISION approach. They thought that the understanding of the procedure was aided by the step-by-step approach (88%), and feel patient safety would be increased due to the INCISION approach (82%). Regarding product feedback, the residents thought that more images should be provided (80%), as well as more (78%) and longer (84%) video segments.

In the in depth-interview, most residents said that INCISION e-learning via its 3D videos was able to improve knowledge and skills. They said that the advantage of using INCISION e-learning was that it can be accesed anywhere and at anytime, but there were limitations to access the program, for example, when they be located in remote areas that don’t have internet access the program couldn’t be used. Finally, all residents in the intervention group said that they will recommend INCISION e-learning due to their positive learning experience to their colleagues.

 Discussion

The results of this study correspond to our expected outcome regarding the INCISION e-learning paradigm. The initial knowledge and skills of residents in both the control and intervention groups were similar, whereas following INCISION e-learning training of residents in the intervention group, the knowledge outcome was significantly increased. We can therefore conclude that INCISION e-learning is able to increase residents’ knowledge compared to conventional learning.

The limitation of this study design was the small number of participants; however, this meant that we were able to complete this report with in-depth interviews of the residents in the intervention group, as shown in Table 3. They discussed the value of INCISION e-learning, including weaknesses, difficulties, and obstacles encountered during the study, as well as the difference between learning with or without the INCISION module. They also discussed whether they would recommend the program to others.

Interview output.
Student ID University Question
What is added value of INCISION e-learning? What the weakness, difficulties or obstacles encountered during the study with INCISION Academy online learning? Is INCISION e-learning make the material easier to understand? According to you, how much difference when compared with conventional teaching? Will you recommend INCISION online learning Academy to your friends? Please explain why.
R3 Universitas Indonesia Added value of the program is that e- learning materials and 3D lecture can improve our understanding of the visual field operations and facilitate the learning process for us to recognize the anatomy and surgical techniques. Nothing. But since we are currently following the program that also prioritizes education in services to patients, time is a significant constraint during the INCISION program. Yes, of course, with 3D video material we can further understand the surgical technique and anatomy. Yes. The program is very good in improve doctor understanding and learning process, especially in a surgical operator.
R5 Universitas Indonesia The videos are very clear, detail, in a rhythm that is not too fast. Our education system that very busy, cause we did not see the online video. Internet network is not sufficiently reliable for live streaming. I only had one time to look at the demo video together with the trainer. Yes. After viewing the video, it becomes a better understanding of anatomy. However, the skill remains to be honed with practice. I recommend because the given video provide basic skills that required before moving into real action.
R1 Universitas Indonesia The value is we can learn more real because of the technology used by the Incision. And accessible because it can be learned anywhere (there is online module). Constraints and difficulties may be time limited. Yes. The material easier to understand because the real picture given by 3D program. Yes. I would recommend this program to other colleagues because it simplifies the process of learning curve.
R14 Universitas Airlangga The value is we can learn more easily, can be accessed anytime anywhere and the tutorial steps allows us to more completely understand, This program depends to internet connection so if there is a trouble about it, we will difficult to access INCISION e-learning. Yes, of course. INCISION e-learning makes more understood the surgical technique and anatomy. I recommend this program because it’s very useful, especially to the surgeon.
R12 Universitas Airlangga Explanation of the tutorial steps by using audio-visual operations making it easier to learn. The videos using cadaver that is still lacking in a real application in the field. Easy to understand, because it directly learn step by step in audio visual. I recommend this tutorial, because the use of audio- visual methods.
R8 Universitas Gajah Mada The value of Incision is we could have access to learning materials anytime and anywhere. Constraints/difficulties during the learning is in the phase of “action with instructors from the incision”. This phase in our implementation is delayed quite long-time. Actually number of cases of hysterectomy procedures is enough, but the obstacles to bring together participants and instructors schedule quite difficult. If we are at a hospital center, the instructor may not necessarily be in place or have no time to scheduled operations, on the contrary when the instructor can be scheduled operations, our participants were in affiliates hospital. Yes, with Incision learning materials more easily understood. By reading the material and then look at the video (especially in 3D) then it is easier to understand the content and have a clear perception of each step of a procedure. Yes. By recommending Incision elearning, I hope my colleagues can get the material and understanding as I got when follow this lesson.
R10 Universitas Gajah Mada Values over Incision Academy is a learning media that comes with 3D video allows us to more completely understand the operating procedures that anatomically closer to the original condition at the time of surgery. Moreover, it can be accessed online, make student can learn anywhere, anytime. Besides the accesses journal links that accompanies the theories given, it is easier for students to learn more about the procedures and theories. Difficulties experienced is due to the high mobility of us out of town, with internet access that is not capable, consequently we are often difficult to access incision Academy online. Yes it is easy to understand. With the incision Academy online, all learners have the same standard procedure if compared with learning that I get so far. Yes, I would recommend with explain the added value as I mentioned in point number 1.

Most residents reported that INCISION e-learning was able to improve knowledge and skills via its use of 3D videos. Another reported advantage was that it can be accessed anywhere and at anytime. Reported weaknesses of the program was limitations to access should they be located in remote areas. Residents also reported that they will recommend INCISION e-learning to their colleagues due to their positive learning experience.

These results are consistent with previous studies that report the efficacy, and the satisfaction amongst users, of multimedia as learning tool for medical purposes, in particular for surgical learning 8, 9, 11 .

 Conclusion

These findings reveal that there were significant differences in knowledge and skill achievement between students who underwent training via the INCISION e-learning module and students who were trained via conventional teaching strategies. In addition, a questionnaire revealed that resident surgeons in the intervention group appreciated the use of INCISON e-learning.

 Data availability

The data referenced by this article are under copyright with the following copyright statement: Copyright: � 2018 Harzif AK et al.

Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

F1000Research: Dataset 1. Answers for Ritzman questionnaire about the perception of INCISION e-learning in the intervention group., https://doi.org/10.5256/f1000research.15799.d227255 12

F1000Research: Dataset 2. Raw data for pre and post test scores in both control and intervention groups., https://doi.org/10.5256/f1000research.15799.d227256 13

Acknowledgements

The authors would like to thank all participants and supervisors that contributed to the work in this study.

 Supplementary material

Supplementary File 1: Pre- and post-test used.

Click here to access the data

Supplementary File 2: Ritzman questionnaire about the perception of INCISION e-learning.

Click here to access the data

Supplementary File 3: Conventional Hysterectomy Course Schedule.

Click here to access the data

Aloraini S : The impact of using multimedia on students’ academic achievement in the college of education at king saud university. Journal of King Saud University-Languages and Translation. 2012;24(2):75–82. 10.1016/j.jksult.2012.05.002 Issa N Mayer RE Schuller M : Teaching for understanding in medical classrooms using multimedia design principles. Med Educ. 2013;47(4):388–396. 23488758 10.1111/medu.12127 Shariff U Seretis C Lee D : The role of multimedia in surgical skills training and assessment. Surgeon. 2016;14(3):150–163. 26577145 10.1016/j.surge.2015.10.003 Masic I : E-learning as new method of medical education. Acta Inform Med. 2008;16(2):102–17. 24109154 10.5455/aim.2008.16.102-117 3789161 Issa N Mayer RE Schuller M : Research on the applications of multimedia auxiliary teaching system with the computer to the education of clinical medicine. Medical Education. 2015;47(4). Shariff U Kullar N Haray PN : Multimedia educational tools for cognitive surgical skill acquisition in open and laparoscopic colorectal surgery: a randomized controlled trial. Colorectal Dis. 2015;17(5):441–450. 25495835 10.1111/codi.12863 Parmar DJ Shah C Parmar RD : Effectiveness of four step approach for procedures (skill) teaching: Rct. The Southeast Asian Journal of Case Report and Review. 2013;2(5):281–288. Reference Source Choules AP : The use of elearning in medical education: a review of the current situation. Postgrad Med J. 2007;83(978):212–216. 17403945 10.1136/pgmj.2006.054189 2600032 Funke K Bonrath E Mardin WA : Blended learning in surgery using the Inmedea Simulator. Langenbecks Arch Surg. 2013;398(2):335–340. 22878596 10.1007/s00423-012-0987-8 Fingeret AL Martinez RH Hsieh C : Watch what happens: using a web-based multimedia platform to enhance intraoperative learning and development of clinical reasoning. Am J Surg. 2016;211(2):384–389. 26801091 10.1016/j.amjsurg.2015.09.007 Norman E Furnes B : The relationship between metacognitive experiences and learning: Is there a difference between digital and non-digital study media? Comput Hum Behav. 2016;54:301–309. 10.1016/j.chb.2015.07.043 Harzif AK Rahman MN Mulawardhana P : Dataset 1 in: INCISION e-learning program as a useful teaching tool to enhance surgeons’ knowledge and skills: An Indonesian multi-center cross-sectional pilot study. F1000Research. 2018. http://www.doi.org/10.5256/f1000research.15799.d227255 Harzif AK Rahman MN Mulawardhana P : Dataset 2 in: INCISION e-learning program as a useful teaching tool to enhance surgeons’ knowledge and skills: An Indonesian multi-center cross-sectional pilot study. F1000Research. 2018. http://www.doi.org/10.5256/f1000research.15799.d227256 10.5256/f1000research.17246.r41413 Reviewer response for version 1 Jagzape Arunita 1 Referee Department of Physiology, Shri Shankaracharya Institute of Medical Sciences, Raipur, India

Competing interests: No competing interests were disclosed.

 21 12 2018 Copyright: © 2018 Jagzape A 2018 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

The efforts are laudable and adequate weightage can be given since it is a multi-center study but I would like to mention that in the title it is mentioned "to enhance knowledge and skills", out of which only knowledge is tested.

It is mentioned only in the Abstract regarding Direct observation of Procedural Skills, but it was nowhere mentioned in the main text, neither was it evaluated which it should have been, as it could test the skill component as well as performance. The module could also be explained in detail.

In the Methodology, study design and sampling technique is not mentioned. Kindly explain the part in the Methodology - Procedure; that the residents took the exam at least once during the week after the study period.

In the Discussion, the present study is not discussed in light of other studies.

In the conclusions, it is mentioned that there were significant differences in knowledge and skill. If skills were nor assessed, this statement can't be made.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Medical Education Technology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.