Impact evaluation of the Steno REACH Certificate Course in Clinical Diabetes Care for health care providers in Malaysia: protocol for a quasi-experimental, mixed-methods research

The burden of diabetes continues to increase in Malaysia, and the public primary health sector has an insufficient number of health care providers well-trained in diabetes care. The Ministry of Health Malaysia collaborated with Steno Diabetes Center to educate primary care doctors and nurses on the fundamentals of clinical diabetes care using a competency-based approach that blends e-learning, classroom-based learning, and clinic-based group work. This programme is called Steno REACH Certificate Course in Clinical Diabetes Care (SRCC). The aim of this study was to assess the effectiveness of the SRCC intervention in improving diabetes-related knowledge, attitudes, skills and clinical practices among non-specialised doctors and general nurses working in public health clinics in Malaysia. This paper presents the study protocol. A quasi-experimental, mixed-methods study based on Solomon’s Four Group Design was applied. Non-specialist doctors and general nurses from ten health clinics were randomly selected to receive the educational intervention. Comparison clinics were purposive selected matching on proxy indicators for quality of diabetes care. The intervention consisted of 50 hours of e-learning, 48 hours of classroom-based learning and approximately 25 hours of work-based learning that covered all main aspects of clinical diabetes care and delivered over a six-month period. Primary outcomes were changes in diabetes-related knowledge, attitudes, skills, and clinical practice. Patients’ perceptions regarding the quality of care provided were classified as a secondary outcome. Other outcome measures included patients' assessment of their chronic disease care and providers' perceptions, attitudes and perceived barriers in care delivery. 1 2 3

The aim of this study was to assess the effectiveness of the SRCC intervention in improving diabetes-related knowledge, attitudes, skills and clinical practices among non-specialised doctors and general nurses working in public health clinics in Malaysia. This paper presents the study protocol.
A quasi-experimental, mixed-methods study based on Solomon's Four Group Design was applied. Non-specialist doctors and general nurses from ten health clinics were randomly selected to receive the educational intervention. Comparison clinics were purposive selected matching on proxy indicators for quality of diabetes care. The intervention consisted of 50 hours of e-learning, 48 hours of classroom-based learning and approximately 25 hours of work-based learning that covered all main aspects of clinical diabetes care and delivered over a six-month period. Primary outcomes were changes in diabetes-related knowledge, attitudes, skills, and clinical practice. Patients' perceptions regarding the quality of care provided were classified as a secondary outcome. Other outcome measures included patients' assessment of their chronic disease care and providers' perceptions, attitudes and perceived barriers in care delivery.

Introduction
The burden of diabetes continues to increase in Malaysia. Data from the 2015 population-based National Health and Morbidity Survey (NHMS) showed a prevalence of 17.5% among adults aged 18 years and above, which translated to about 3.5 million adults 1 . The overall prevalence of diabetes is amongst the highest in the Asia-Pacific region, excluding the pacific island nations 2 .
Primary health care is at the center of the Malaysian healthcare system and is supported by secondary and tertiary care. Malaysia has a well-established, public primary health care structure, with 1,061 health clinics, 1,810 community clinics, and 307 "out-reach" health clinics (called 1Malaysia clinics) 3 . Although Malaysia has a parallel public and private system, the majority of treatment for chronic diseases is provided by the public health system which is heavily subsidised by the government.
The Malaysian healthcare delivery system is facing increasing pressure to provide quality care to patients with diabetes. The latest data showed that about 80% of diagnosed diabetes patients seek treatment at MOH healthcare facilities and this proportion is expected to continue to increase 1 . As the burden of diabetes increases, the public health sector is faced with an insufficient number of well-trained diabetes medical practitioners to handle the increasing number of diabetes patients 4 . This has contributed to a treatment gap that requires MOH healthcare professionals to be better equipped to provide patient-centered diabetes services at the primary care level.
A large part of addressing this treatment gap focuses on human resource capacity building in the management of noncommunicable diseases (NCDs) 5,6 . Continuing medical education (CME) is a cornerstone in developing clinical skills and ensuring high-quality patient care by nurses and medical doctors. Cervero and Gaines (2015) conclude in a synthesis of systematic reviews, that CME improves physician performance and patient outcomes 7 , and while studies on CME in general have reported a positive effect on physician performance and patient outcomes, several have expressed the need for further studies on the implementation of knowledge, skills, and attitudes, as well as the impact of contextual and implementation factors 7 . Despite the importance of CME, few studies have measured the clinical impact of CME on diabetes in a real-world setting 8-10 . These studies have shown varying results 8,10-13 . Studies have also shown that CME is associated with increased satisfaction and a better psychosocial wellbeing of diabetes patients 11 , and that it is very well received among participating healthcare providers (HCPs) 14 .

The SRCC intervention
As part of its efforts to build diabetes management capacity, MOH Malaysia collaborated with Steno Diabetes Center to educate primary care doctors and nurses on the fundamentals of clinical diabetes care using a competency-based approach that blends e-learning, classroom-based learning, and clinic-based group work. This programme is called the Steno REACH Certificate Course in Clinical Diabetes Care (SRCC). The goal of SRCC is to improve the knowledge and skills of participating HCPs in clinical diabetes management, thereby empowering them to provide high-quality diabetes care. The course was designed for primary care doctors and nurses with little training in diabetes care and was facilitated by a team of Malaysian health care professionals trained to deliver the programme by experts from Steno Diabetes Center Copenhagen.
The curriculum included ten modules: Two versions of each module were developed, one for doctors and one for nurses, with content tailored to their individual job roles. The intervention was designed to have learners cycle through a schedule of approximately 50 hours of independent online study and 48 hours of face-to-face classroom time. The e-learning content included all foundational materials delivered in an interactive learning environment. Participants completed the first five modules before coming to the classroom to reinforce key learning outcomes through interactive learning activities. This experience repeated itself for modules 6 to 10. In addition, participants were assigned work-based learning activities to complete in the periods between classroom-based sessions. Work-based learning activities included defining and reflecting on personal learning goals, patient journaling, article discussions with clinic peers, case discussions with clinic peers, and medical record reviews with clinic leaders. Together, work-based learning activities took an additional 25 to 30 hours of learning time.
The intervention was piloted twice between 10 October 2015 and 4 December 2016 to ensure its proper functioning and make any necessary curriculum improvements. Pilot course participants were recruited using a clinic-based recruitment approach. Ten interested clinics were invited to send at least one doctor and one nurse in order to reinforce the team-based care model essential to good diabetes clinical care. A total of 22 non-specialist doctors and 40 general nurses participated in the pilot studies. This paper describes the protocol of the impact evaluation of the SRCC.

General objective
The impact evaluation aims to assess the effect of participation of non-specialised medical doctors and general nurses in the SRCC in selected public health clinics in Malaysia.

Specific objectives
(1) To measure changes in diabetes-related knowledge, attitudes and clinical skills before and after course participation; (2) To assess whether course participation and improved diabetes-related knowledge and clinical skills translate into changes in an individual's clinical practice; and (3) To examine the influence of contextual factors at facility, health system and sociocultural levels on individual diabetes-related clinical practice changes.
The primary hypothesis of this research was that participation in the SRCC will result in increased knowledge about diagnosing diabetes, the role of diet and exercise in the treatment of diabetes, oral and injectable therapies used in the treatment of diabetes, common diabetes complications, and approaches to patient engagement. It was also the hypothesis that improved knowledge would translate into improved clinical skills in these aspects of care and that providers' attitudes about people with diabetes would be positively impacted.
In addition, it was hoped that 1) patients' perceptions of their care experience would improve with the intervention, and 2) primary care providers' perceptions, attitudes, experiences and perceived barriers in implementing the intervention were explored.

Study design
A quasi-experimental, mixed-methods approach was applied in this study, based on Solomon's Four Group Design, which is ideally used in evaluation of educational interventions that contain pre-and post-assessments. This design is recommended when it is possible that the pre-test could influence later tests, for example by learning or priming effects. This design contain four groups.
In addition to the basic pre-test/intervention/post-test groups, three additional groups are included: one that received both tests, but not the intervention; one that received the intervention without the pre-test only; and one with neither pre-test nor intervention. This design serves to reduce the influence of confounding variables and allows the researcher to test whether the pretest itself has an effect on the subjects. Doctors and nurses from comparison clinics not participating in SRCC were also assessed to isolate individual and clinic level changes that may be attributable to the educational intervention.
Each of the four study arms were subjected to a different set of data collection methods at different points of the intervention. This is shown diagrammatically in Figure 1.

Selection of clinics and participants
From a pre-determined sampling frame, health clinics in the states of Kuala Lumpur and Selangor were randomly selected for the four arms of this research. These two states were selected as they have the full range of the diverse types of health clinics located in Malaysia. The four arms were selected to conform to Solomon's Four Group Design. This design allows researchers to control for the possibility of a test-effect. The pilot test of the educational intervention included ten clinics, and based on this experience, ten clinics were deemed an appropriate number of clinics to support a diverse learning experience among participants while still limiting class size to an appropriate level for delivering a heavily facilitated and interactive learning experience. All MOH health clinics in Selangor and Kuala Lumpur meeting the inclusion criteria were subjected to a two-stage stratified random sampling for selection of the ten intervention clinics. Clinics meeting the following criteria were eligible for random selection: Inclusion criteria were: MOH health clinics with more than 1,000 registered active diabetes patients (as defined and registered in the National Diabetes Registry).
Exclusion criteria were: MOH health clinics that have participants enrolled in the two SRCC pilot classes.
A total of 70 possible clinics were reduced to 43 eligible clinics based on the above criteria, and these 43 were included in the stratified, random selection process. For the first stage, the 43 health clinics were divided into the following three categories based on the number of registered active diabetes patients: • Between 1,000 and 1,999 patients • Between 2,000 and 2,999 patients • ≥3,000 patients For the second stage the health clinics were divided into the following two categories based on the variety and complexity of medical services they provide as defined by MOH: • Intermediate clinic • Advanced clinic Based on the above criteria, the number of eligible clinics is summarised in Table 1. Ten clinics were then randomly selected for inclusion as the intervention clinics as shown in Table 2.
These ten clinics were then randomly allocated to one of the two intervention arms. Six clinics were allocated to Arm 1 and four were allocated to Arm 2. The six-to-four split was to allow for more pre-and post-data to be included for the intervention arm of research, as per Solomon's Four Group Design, only Arm 1 of the intervention clinics has both pre-and post-data available.
Comparison clinics were then selected from the remaining post-stratification clinics. Six comparison clinics were selected for Arm 3 using a purposive approach in which comparison clinics were matched based on a proxy indicator for quality of diabetes care. Using data from the National Diabetes Registry (NDR), the mean HbA 1 c value for all patients with diabetes in each intervention clinic was calculated and a comparison clinic with the closest mean HbA 1 c value for all diabetes patients was matched and selected for inclusion in Arm 3. The same process was used to select the four comparison clinics in Arm 4, which were chosen to match the four intervention clinics in Arm 2. From each of the ten health clinics selected to receive the intervention, the participants were chosen by the Family Medicine Specialists (FMS) who are the clinical heads of the health clinic, based on the following criteria: (1) Each clinic nominated at least four participants, two medical officers and two nurses.
(2) The medical officers had to be a non-specialist without advanced training in diabetes care.
(3) The nurses should not have undergone post-basic or advanced training in diabetes care.
(4) The participants at the clinic had to be able to interact as a team at their workplace during the six-month SRCC.
(5) The participants were either already managing or interacting with diabetes patients prior to enrolment, or if the participants were not currently managing or interacting with diabetes patients, that opportunity had to be provided to them during the six-month study period of the SRCC.
Participants from the intervention clinics were enrolled in the programme. Participants from the comparison clinics were then selected so that their clinical experience with diabetes patients matched the participants in the intervention arm as much as possible. Participants from the comparison clinics were offered enrolment in an SRCC class after the completion of the data collection.
The number of participants in each arm of the research is shown in Table 3.

Sample size considerations
Despite the fairly small number of participants in each pre and post-test group, the validity of the pre and post-test results can be achieved through a meta-analysis of the data. A statistical treatment of the quantitative data in Solomon's Four Group Design is possible with a meta-analytical approach 15 . With this approach, the results of different statistical tests (2x2 ANOVA, repeated measure ANOVA and double tailed t-test) are combined to create statistical power without the need for a large sample. Meta-analysis demonstrates how the results from disparate, independent tests of the same hypotheses may be statistically combined even when the significance tests arise through different statistical techniques 16 .

Data collection methods
This study utilised both quantitative and qualitative methods to achieve the specific research objectives. The frequency and timing of these methods were applied depending on the research arm as described in Figure 1. A summary of the different data collection methods and purpose is shown in Table 4. All interviewers, investigators and examiners were trained regarding the study procedures prior to the conduct of the research to minimise variability in the method of data collection.
Quantitative data collection. All course participants were subjected to the same core data collection methods that aim to describe and measure the effectiveness of the educational programme. The quantitative study tools are as follows. The majority  of data collection instruments are available as Extended data 17 ; however, since the multiple choice questions (MCQs) are still being used to assess doctors and nurses, these are only available on request on a case-by-case basis.

Multiple choice questions assessing knowledge
The SRCC test of clinical diabetes-related knowledge was developed by a team of endocrinologist and diabetes nurses based on a review of the full course curriculum for medical officers and nurses. The medical officers' version included 79 MCQs made available in English and the nurses' version included 67 MCQs and was available in English and Malay language. Since these questions are still being used to assess candidates, they cannot be made openly available.

Diabetes Attitude Scale
The third version of Diabetes Attitude Scale 3 (DAS-3) is a previously validated general measure of diabetes-related attitudes developed by the University of Michigan Diabetes Research and Training Center 18 . This method has been shown to be suitable for evaluation of professional education programmes provided that they include the specific topic areas measured by the five DAS-3 sub-scales 18 , including the attitude of HCPs on the need for special training in education, seriousness of type 2 diabetes, the overall value of tight glucose control in diabetes care, psychosocial impact of diabetes on patients, and attitude toward patient autonomy, which is the case with SRCC.
The test was administered to participants in the pilot study and their results were evaluated by an independent medical education consultancy firm who evaluated each version of the test for reliability using Chronbach's α. The resulting α of 0.61 was slightly lower than the gold standard of 0.8, but deemed sufficiently reliable for a non-high stakes examination, particularly given the relatively small sample size of the pilot cohort.

Objective structured clinical examination (OSCE)
An OSCE is a short circuit of stations at which each participant is examined individually by one or more experienced examiners using real or simulated (actor) patients. The OSCE stations were developed by a team of endocrinologists and diabetes nurses, based on a review of the full course curriculum for medical officers and nurses. Here, medical officers were examined at nine stations and nurses at eight stations. At each station, participants were asked to perform a limited number of clinical tasks within a specified time period (15 minutes) during which two trained examiners used a marking scheme to differentiate good performance from poor performance. The examiners were FMS for the medical officers, and senior diabetes educators for the nurses.

Qualitative data collection. Additional qualitative investigations
were also carried out to better understand how knowledge is being acquired and applied in clinical practice and to create a reliable description of the clinical context into which learnings are being integrated. Qualitative investigations were conducted in Arms 1, 2 and 3 clinics, using the following qualitative tools.

In-depth interviews with SRCC participants
Pre-and post-intervention in-depth interviews were conducted with all participants in Arm 1 clinics (23 in total), in their respective clinics, in order to better understand their history with diabetes related training, typical clinical encounters with people with diabetes and daily clinic life. Interviewed participants were also asked about the process of learning from the various learning platforms and each explored the application of new knowledge and skills to clinical practice. These interviews were audio recorded.

Key informant interviews with FMS
FMSs from the intervention (Arm 1 and 2) and comparison (Arm 3) clinics were interviewed pre-and post-interaction to obtain information on the organisation of clinical services for diabetes patients in the 16 health clinics of Arm 1, 2 and 3, in their respective clinics. Post-intervention FMS interviews for Arm 1 and 2 clinics included themes of their involvement and interactions with the SRCC participants during the intervention period. These interviews were audio recorded.

Interviews with patients receiving care from SRCC participants
Short semi-structured interviews lasting between 10 and 15 minutes with 162 randomly selected patients directly after an observed clinical encounter were conducted in Arm 1 clinics, in the respective clinics. Interview data were used to investigate the patients' experiences and perceptions of the particular observed clinical interaction. The inclusions of patient interviews provided an important opportunity to triangulate data from the direct observations, the HCP interviews, and the patients' experience of the clinical encounter. These interviews were audio recorded.

Direct clinical observation of participants and clinics
Lastly, to investigate the translation of new knowledge into actual clinical practice two days of direct observation of clinical practice of the healthcare professionals from the six clinics participating in Arm 1 (23 in total) were conducted by an experienced clinician of the research team. This researcher assessed the level of diabetes related clinical proficiency demonstrated in each observed clinical consultation. In addition to a narrative assessment of the clinical encounter, each observed session was assigned one of five proficiency levels based on criteria established by the National Institutes of Health Proficiency Scale 19 .
An observation-based assessment of the clinical environment was completed and focused on the general clinical environment, interactions between patient and HCPs, and the nature of interactions between health staff working in the clinic.

Data management
In order to manage research data effectively and efficiently as well as to ensure confidentiality, all participating clinics and respondents (participants, FMS and patients) were anonymised and only able to be identified by a sequentially generated ID-number during data collection, follow-up, data processing, analyses and publication. All collected information and data, such as consent forms, background information of respondents, and audio files were stored according to the generated ID-number. Paper records were securely stored in locked file cabinets and also scanned in digital form, while digital records of the files including the audio files were stored and backed-up in a password-protected external thumbdrive and hardisk. All data were only accessed by the principal investigator, research project manager and key research assistants. Data entry and transcriptions were conducted by the research assistants and quality checked by the research project manager and principal investigator.

Data analysis plan
Data were analysed using a convergent parallel design approach, a commonly used mixed-methods analytical approach 20,21 .
The purpose of the convergent design is to "obtain different, but complementary data on the same topic" to best understand the research questions 22 . This design is used to triangulate the methods by comparing and contrasting quantitative statistical results with qualitative findings for corroboration and validation purposes. It is also used to synthesise complementary quantitative and qualitative results to develop a more complete understanding of a phenomenon 20 .

Independent data analysis
Pre and posttest examination, DAS and OSCE scores were analysed using both the 2x2 ANOVA, repeated measure ANOVA test and independent t-test according to the Solomon's Four Group Design 15 .
The interviews were transcribed and analysed using thematic content analysis 20 . The lead investigator coded the transcripts according to recurrent themes. As the main purpose of this analysis was to identify how participation in the SRCC impacted clinical competency and whether or not there are contextual factors that support or hinder the translation of new knowledge into clinical practice, these formed the main a priori domains into which sub-categories can be grouped. However, the analysis also allowed for the inclusion of themes not pre-defined in the template.
Data from the observations were in the form of descriptive field notes evaluating various aspects of each clinical encounter. Since observational data were collected pre-and post-intervention, the short descriptions of each observed clinical encounter were assessed for observed similarities and differences using an inductive approach in which many observations were analysed to find more abstract generalisations and to build a picture of the phenomenon being studied (i.e. whether the acquisition of new knowledge about diabetes clinical care was translated into practice).

Merging and triangulation of data
The observational data were combined with the in-depth interviews and key-informant interviews to understand more thoroughly how participation in the educational intervention facilitated the demonstration of new clinical skills, and to further explore the contextual factors that enabled or inhibited the translation of new knowledge into practice. The data were further explored to identify emerging and recurrent trends within and between qualitative data and to the incidence of themes within and across qualitative data types.
Quantitative and qualitative data were merged using common mixed-methods strategies 20 . Firstly, content areas that were represented in both data types were identified, compared, contrasted and synthesised as findings. Secondly, differences in one set of results were further examined based on defined dimensions found in the other data set in an attempt to identify and explore data that complements or contradicts the findings from the shared content areas. The exact methods and procedures of the merging process would be dependent on the results from each data collection method.
Interpretation of merged data focused on a discussion of the ways in which the quantitative and qualitative data converged, diverged, related to each other and produced a more complete understanding of whether and how the educational intervention results in changes in diabetes care competence.

Expected outcomes
Primary outcomes were changes in diabetes-related knowledge, attitudes, skills, and clinical practice. Patients' perceptions regarding the quality of care provided were classified as a secondary outcome. Other data included patients' assessment of their chronic disease care and providers' perceptions, attitudes and perceived barriers in care delivery.

Research ethics
The Since the study included direct observation of clinical encounters, the possibility of observing sub-standard care existed. If diabetes and/or medical care was witnessed that could lead to acute hospitalisation and/or death, the data collector(s) were instructed to immediately raise this concern to the HCP(s), in order to avoid this risk. For substandard diabetes and/or medical care not likely to lead to acute hospitalisation and/or death, investigators were instructed to avoid intervention, so as not to lead to punitive actions toward the involved HCPs, nor to compromise the HCP-patient relationship.

Discussion
This study has two key strengths. Firstly, the use of the robust Solomon's Four Group Design enabled the researchers not only to assess the intervention effect, but also the presence of pretest sensitisation and the interaction effects between intervention and pretest. The Solomon's Four Group design has not been extensively used in recent studies, partly because it is a complicated design and partly because the statistical analysis is rather complicated. Secondly, the use of mixed methods combining both quantitative and qualitative methodologies provided the researchers with data on intervention effects on knowledge, attitudes, clinical skills and practices, and allowed triangulation of the data.
There are also inherent limitations in the protocol. Observational data collection methods may have limited validity, including the Hawthorne effect 23 ; regarding the interviews, there may be courtesy bias, where participants provide responses that they think the interviewer would like to hear. These potential biases can be addressed through triangulation of data from the different data collection methods 24 .
Part of the statistical analysis required the transformation of qualitative data for the clinical practice into quantitative data. While we acknowledge the concerns of the limitations of quantitised qualitative data for statistical measurement, the transformation process will be clearly described and only simple statistical measures for differences employed.

Data availability
Underlying data No data are associated with this article. This project contains the following extended data:
Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).
The multiple choice questions are currently being used as part of the assessment of doctors and nurses undergoing the Steno REACH Certificate Course in Clinical Diabetes Care organised by MOH Malaysia. We would however be willing to share a copy on a case-by-case basis, if we are assured that the confidentiality of the multiple choice questions is maintained. Any queries can be directed to Dr Feisul Mustapha (email: dr.feisul@moh.gov.my).