Adaptation and Validation of the Interprofessional Socialization and Valuing Scale (ISVS-24) for Utilization in Multiplayer Virtual Reality Environments

Study design

This study was a pilot study using a cross-sectional design aimed at determining the psychometric properties of the adapted ISVS-24 for a virtual environment setting and involved three phases. The first phase involved the translation of the original ISVS-24, which was in English, into Indonesian. The second phase was content validity assessment of the ISVS-24 by a panel of experts. The third phase was a pilot study conducted with participants who completed the questionnaire, which had undergone content validity processes and was adapted for use in a virtual environment setting ( Figure 1).

Figure 1. Translation and validation process.

Figure 2. Viewpoints of different users.

Participants

A total of 72 anesthesiology residents, general physicians, and professional nurses from the Faculty of Medicine, Universitas Indonesia – Dr. Cipto Mangunkusumo Hospital took part in this study and were voluntarily recruited. Participants were recruited through direct invitation to medical and nursing professionals at the institution. As inclusion criteria, the following factors were considered: (1) older than 18 years, (2) free from visual, auditory, and other sensory impairments, (3) and provide informed consent to participate. Participants were not required to have prior interprofessional team experience. However, their professional backgrounds ensured exposure to interprofessional collaboration in clinical settings. Additionally, prior exposure to VR training was not a requirement, as the study aimed to assess interprofessional socialization within a simulated VR setting, regardless of previous VR experience.

Participants were excluded if they had a known history of motion sickness, balance disorders, or any medical condition that could increase the risk of discomfort during VR use.

Development and Pilot Testing of VR-Local Anesthesia of Systemic Toxicity

This study developed Local Anesthesia of Systemic Toxicity scenes using Virtual Reality (VR-LAST) to provide an engaging and immersive learning experience that reflects students’ real-clinical scenarios. VR-LAST is a platform that provides an immersive experience of a virtual world featuring 3D-based user-generated content that supports interactive networking. This platform is used in the simulation of systemic toxicity events due to the administration of local anesthesia in patients undergoing surgery with local anesthesia.

In this platform, we created a scenario where an individual who will undergo operative action must be given anesthesia in the form of local anesthesia, which will be managed collaboratively by anesthesia residents, general practitioners, and professional nurses. As illustrated in Figure 2: viewpoints of different users, VR-LAST incorporates multiple user viewpoints: (A) the anesthesia resident’s view of the Non-Playable Character (NPC), (B) the general practitioner’s view, (C) the nursing student’s view, and (D) the facilitator’s view. The scenario guides participants through the preparation for anesthesia, administration of anesthesia, systemic toxicity events, and subsequent interventions, including assessment and management. The specific tasks performed by participants in the virtual environment are outlined in Supplementary Table. The operating room concept was built to help participants understand the environment and provide a realistic experience. The developed VR is ready to be tested for feasibility. It can enable participants from different times and locations to come together in real-time interactive simulation training to manage LAST emergency events. IPE using simulation-based LAST is a practical approach to prepare students for a collaborative environment, develop technical skills, and enhance soft skills, such as communication and teamwork, to prepare them as work-ready graduates.

After the development of the initial VR-LAST simulation, a preliminary pilot test was conducted involving six participants, including of two anesthesiology specialist, two general physician, and two nurses. The aim of this pilot was to assess the basic functionality, clarity of the scenario, and user experience within the virtual environment. During the session, the team provided structured feedback, particularly highlighting the need to include a practice mode or orientation phase before starting the actual LAST management simulation. This feature was considered important to help future users become familiar with the VR interface and controls. Based on this input, the VR developer revised the content by incorporating a pre-simulation practice session. Following these adjustments, the final version of VR-LAST was deemed ready for implementation in the main study (pilot study).

It should be noted that participants involved in this preliminary VR pilot were different from those recruited for the instrument validation phase.

Instruments

The Interprofessional Socialization and Valuing Scale (ISVS-24) was a 24-item self-assessment questionnaire created by King et al. The self-assessment questionnaire focuses on changes in attitudes, actions, and beliefs of participants after interprofessional training. Respondents are asked to rate their agreement or disagreement with each statement based on their own experiences and perceptions after previous training. The scale aims to capture attitudes, beliefs, and behaviours that contribute to effective teamwork and collaboration across three dimensions: self-perceived ability to work for others, value in working with others, and comfort in working with others.¹⁸

The original ISVS-24 was selected for adaptation instead of its shortened or previously translated versions to preserve the instrument’s full conceptual coverage. This approach allowed for a more precise contextual adaptation tailored to MPVR-based interprofessional learning. Using the full version enabled us to refine the items in a controlled manner while maintaining fidelity to the original construct, which was essential for measuring complex socialization processes in a virtual simulation setting.

Permission to translate and adapt the ISVS-24 into Indonesian and for its use in a VR-based educational context was obtained from the original author prior to conducting the study.

This study utilized virtual reality simulations that were played by three individuals. Each simulation session involved three participants, comprising an anesthesiology resident, a general physician, and a professional nurse. During the simulation, each participant took on specific interprofessional roles relevant to anesthesia management, ensuring a realistic team-based training environment. VR is conceived with specific learning objectives; comprehension of cognitive, motor, communicative, and interpersonal skills in managing local anaesthetic systemic toxicity cases in an immersive surgical room environment. A patient was standardized as NPCs (non-playable characters) with specific scenarios.

Procedure

Phase 1: Translation of the ISVS-24

Two professional bilingual translators, both fluent in English and Indonesian and with backgrounds in healthcare and medical education, independently translated the original ISVS-24 into Indonesian. These two initial versions were reviewed and synthesized into a single preliminary version by the research team, all of whom are native Indonesian speakers with expertise in interprofessional education, simulation-based training, and health professions education.

To ensure conceptual and semantic equivalence, a back-translation was performed by an independent bilingual translator who had no prior exposure to the original ISVS-24. This step was essential to detect discrepancies and maintain the integrity of the original meaning. The back-translated version was compared with the original English version by the research team. Any inconsistencies were discussed in a series of consensus meetings involving all translators and research team members. Through these iterative discussions, item wording was refined and adapted to suit the context of Multi-Participant Virtual Reality (MPVR)-based interprofessional learning, resulting in a final culturally and contextually appropriate version of the ISVS-24 in Indonesian.

Phase 2: Content validation of the Indonesian ISVS-24

To evaluate content validity, the Indonesian ISVS-24 was assessed by a panel of four senior anesthesiology consultants with a minimum of 10 years of experience in health professions education and simulation-based interprofessional training. All panelists had extensive involvement in MPVR-based simulation research and had served as facilitators in interdisciplinary training programs. However, as the instrument targets interprofessional socialization more broadly, the composition of the panel presents a limitation in that it did not include experts from other health professions or individuals with formal training in IPE assessment.

Each item was rated on a Likert scale of 1-5 based on its relevance to MPVR simulation-based setting. The scale was as follows: 1 - highly irrelevant, 2 - irrelevant, 3 - moderate, 4 - relevant, 5 - highly relevant. The experts were also requested to provide qualitative feedback or suggestions on pharsing, clarity, and revisions on certain questions they deemed less appropriate for multiplayer VR. Content validity was then analyzed using Aiken’s V formula, and items that failed to meet the minimum validity threshold (V < 0.80) were either revised or removed based on expert recommendations.

The relatively small number of reviewers (n=4) represents a limitation of this phase. Future studies should incorporate a broader panel to increase generalizability and enhance content representativeness for interprofessional users.

Phase 3: Construct validation of the Indonesian ISVS-24

This phase aimed to assess the clarity, comprehensibility, and preliminary psychometric properties of the translated ISVS-24 instrument. A total of 72 participants were recruited through purposive sampling, consisting of anesthesiology residents, general physicians, and professional nurses. The participants were provided with information regarding the purpose of the study. They were given consent forms after being reminded of the voluntary nature of their participation and the confidential and anonymous handling of their data. Once they had signed the informed consent, the participants were assembled by the principal investigator in the pilot trial, who convened them in a single classroom. All participants in the pilot study were introduced and trained using VR equipment, specifically the Oculus Quest 2 headset. Before each simulation, participants received a 10-minute orientation on VR headset use, scenario overview, and simulation flow. Roles (resident, physician, nurse) were assigned and briefed to ensure shared understanding and coordination. They were divided into small teams consisting of three persons, with one individual assigned as the team leader. All groups underwent MPVR simulation-based training. Upon finishing the simulation, the participants were asked to check the content and the understandability of the scale.

Following the simulation, participants were asked to answer the adapted ISVS-24 items. Items that had previously been identified as invalid during Phase 2 through content validity analysis using Aiken’s V were removed before distribution. The remaining 21-item version of the ISVS-24 was administered post-simulation.

The responses were then analyzed to assess the preliminary construct validity and internal consistency of the instrument. While the sample size was sufficient for an initial pilot study, we acknowledge that it may not meet the optimal requirements for robust exploratory factor analysis. This limitation is addressed in the discussion, and we recommend further psychometric evaluation using a larger, more representative sample.

Data analysis

We used digital scoring and rating sheets to record the collected data manually. SPSS (version 22.0; IBM Corp) was applied to analyse the data quantitatively. First, a descriptive analysis of the results was conducted. The frequency and percentage were computed for the categorical variables, whereas the measures of central tendency and dispersion were determined for the quantitative variables. To determine the questionnaire’s structural validity, a factorial analysis was conducted through principal component analysis. The content validity index (CVI) of the instrument in this study was evaluated using Aiken’s formula. A measurement tool is considered adequately valid if its Aiken coefficient is greater than 0.5.^22–24 The assessment of internal consistency and reliability of the variables in this study was conducted through Cronbach’s alpha reliability analysis. Items with Cronbach’s alpha value above 0.70 were considered reliable. To evaluate the validity of the responses, the Pearson Correlation Coefficient (PCC) was calculated to assess the relationship between individual items and the overall scale.^25–28

Ethical considerations and consent to participate

This study was conducted following the Declaration of Helsinki and was approved by the Ethics Committee of the Faculty of Medicine, Universitas Indonesia - Dr. Cipto Mangunkusumo Hospital with regards of the protection of human rights and welfare in medical research, with approval number 1146/UN2.F1/ETIK/PPM.00.02/2022, on October 31, 2022. All participants provided written informed consent before participation. No participant-identifying information was included in the study data. Data were managed by researchers and assistants who were not involved in any teaching roles, and only the ID codes were noted in the database. No personal data was accessible to the instructional team in the initial phase of the research; adjustments were made to the items in the original ISVS to suit the multiplayer VR environment.

Content validity

Aiken’s method was used to assess the content validity of the questionnaire in this study. Four senior anesthesiology consultants who were experts in MPVR simulation-based education evaluated each item based on its relevance to interprofessional collaboration within virtual environments. Consequently, in this investigation, each questionnaire item underwent scrutiny by the experts who evaluated its appropriateness and relevance. If a consensus (two or more experts) suggested that a specific item was unnecessary, it was subsequently removed from the questionnaire. The analysis of the content validity of the instrument used the Aiken coefficient V can be viewed in Table 1.

Three items (item 5, 7, and 16) received Aiken’s V values ≤ 0.50 and were therefore excluded. These items were considered less compatible with the immersive characteristics of the MPVR context, particularly those referencing client or family-centered care, which were not applicable to the simulated clinical scenario. The removal of these items was based on low content validity indices (Aiken’s V ≤ 0.50) as assessed by four expert reviewers during Phase 2. These three items were excluded prior to Phase 3 and were therefore not tested further with study participants. The decision was made to ensure content relevance in the context of MPVR-based interprofessional learning.

The three items that were removed are:

Following this, twenty-one items with Aiken’s V values ≥ 0.5 were deemed valid and incorporated into the Multiplayer Virtual Reality Adapted Version of the Interprofessional Socialization and Valuing Scale. In addition, all experts agreed to make a slight change to item number 21 to ensure it is framed as a positive statement, changing it from “I believe that interprofessional practice is difficult to implement” to “I believe that interprofessional practice is easy to implement”.

Participant characteristics

Seventy-two participants total participated in the study, of which 40 were female (55.6%) and 32 were male (44.4%); 26 were anesthesiology residents (36.1%), 23 were general physicians (31.9%), and 23 were professional nurses (31.9%). The average age was 29.32 ± 2.54 years old ( Table 2).

Consistency validity

The questionnaire’s validity was verified through the computation of the Pearson correlation coefficient (PCC). The critical value was 0.232, according to the table of critical values. As shown in Table 3, for all our variables, the r table was > 0.232 for p values of < 0.05. Table 3 displays the validity (Pearson’s correlation coefficient) for every item.

Exploratory factor analysis (construct validity)

Exploratory factor analysis (EFA) was conducted to examine the underlying factor structure of the Indonesian-adapted ISVS-24 and assess its construct validity. Suitability for factor analysis was confirmed via Kaiser-Meyer-Olkin (KMO = 0.857) and Bartlett’s Test of Sphericity (χ²(66) = 1193.762, p < 0.05). Although the statistical procedure initially extracted 21 components (equal to the number of items), this is a default outcome of the analysis and does not imply that all components are meaningful.

To determine the optimal number of factors, we used Kaiser’s criterion (eigenvalues > 1) and inspected the scree plot. Only the first three components had eigenvalues > 1, cumulatively accounting for 66.88% of the total variance. A factor loading threshold of ≥ 0.40 was used to determine item retention, and all 21 items met this criterion ( Table 4).

Item reliability test

In this study, Cronbach’s alpha was used to measure the reliability of each item. Table 5 shows the Cronbach’s alpha result. The overall Cronbach’s alpha coefficient for the 21-item scale was 0.959.

Therefore, we acknowledge this limitation in the discussion and recommend future studies to assess internal structure further using McDonald’s Omega and confirmatory analysis techniques.