The study was conducted from February 16 up to March 30, 2020, in Ayder Comprehensive Specialized Hospital (ACSH), which is located in Mekelle city, the capital city of Tigray regional state of Ethiopia. The city is located 783 km away north of Addis Ababa, the capital city of Ethiopia, and has a total population of 587,000. ²⁷ Out of the total ten public health centers and four public hospitals, ACSH is the only one that provide musculoskeletal physiotherapy service. It is rendering its referral and non-referral services to the 8 million populations in its catchment areas of the Tigray, Afar and Northeastern parts of the Amhara Regional States, including the Eritrean refugees, since 2008. ²⁸

Purposively selected, health care providers (physiotherapists), MsPT service coordinators, and patients were the study participants. In addition, registration books, patient medical records, and the human, physical and material resources of the unit, including the MsPT unit itself, were used to gather information for this evaluation analysis.

Full time physiotherapist, physiotherapy head and service coordinators, who have been serving at least for six months in the clinic, and MsPT patients who were on follow up in the study unit were included in the evaluation. Besides, the clinical records of patients who get service in MsPT unit in the last 6 months prior to the data collection period were included.

This evaluation is to gain and interpret useful information that can assist service improvement. The evaluation was also fully participatory to service stakeholders, which increase evaluation credibility and maximize utilization of evaluation findings. Therefore, this evaluation used formative evaluation approach to obtain concrete information concerning the operation of the musculoskeletal physiotherapy service. ²⁹

The focus of the evaluation is on the process of musculoskeletal physiotherapy service delivery, which aims to clearly understand the internal dynamics of the service operations as it can help in identifying areas where improvements might be made.

The design used for this evaluation was health facility based single embedded case study design which involved both quantitative and qualitative methods to describe the service using variety of reliable information. ³⁰ The information obtained through this method relies on multiple sources of evidence, with data needing to converge in a triangulating fashion and helps to ensure trustworthiness. ³¹

The first dimension was compliance of the MsPT unit with the national physical rehabilitation service implementation standard, and under this dimension, the availability and accommodation components of access are evaluated. The aavailability sub-dimension assesses the presence of the necessary inputs to carry out the MsPT service. The accommodation sub-dimension measures the way that musculoskeletal physiotherapy service organized to accommodate clients including the treatment room, waiting area, and external physical environment of the unit.

The second dimension employed for this study was technical compliance of the MsPT service providers to the national physical rehabilitation standard in providing MsPT services to patients. The compliance dimension assesses whether physiotherapists are diagnosing, conducting physical examination, following up and recording complete information of patients according to the recommended national physical rehabilitation guideline. Therefore, the activities of musculoskeletal physiotherapists were assessed and compared with the standard.

The third dimension used to judge the service delivery of the MsPT unit was acceptability of the service by its users.

Quantitative

We used different source for the different dimensions of the evaluation.

Availability: All the service documents in unit were assessed through a structured checklist format to evaluate resource availability. In addition, physical count was made to estimate proportion of the unit’s patient load with the required human resources.

Accommodation: To assess the way in which the health facilities and the resources are organized according to the standard, direct observation technique was applied. Treatment room setup, waiting area setup, and physical environment setup for safe access were addressed.

Technical compliance: To determine the compliance of health professionals’ performance direct non-participatory observation was conducted on forty client provider interaction. In addition, to investigate if the patients were assessed for MsPT treatment follow-up adherence we retrospectively reviewed the clients medical card using systematic random sampling. The sample size was calculated at the maximum allowable error of 5% and population proportion of 79.7 % from a previous study on physiotherapy documentation of patients’ record. ³² Thus, the calculated sample size was 156 clinical records of patients. However, to accommodate possible missing of the clinical records 10% of the calculated sample was added. Altogether, the sample size established at 172 clinical records. The total patients who were registered and using the service in the unit, six months prior to the data collection time were 486. Thus, the sampling interval (K) was calculated to be three. Among total one hundred seventy-two reviewed client medical records 100 (58.1%) were male client records. The age of clients ranges from 18-74, with a mean age of 46.32(SD ± 14.03) (Table 1, Participant characteristic, Physiotherapist recording compliance, Extended data, https://doi.org/10.7910/DVN/RE3EUL).

Acceptability: Exit interview were done on randomly selected 15 clients who were on reassessment to see whether they are satisfied on their visit in MsPT unit. To determine the level of clients’ adherence, the 172 clients’ adherence assessment sheets were reviewed.

Qualitative

For qualitative data in-depth interview was conducted with purposively selected Key informants who are information rich. Finally, five key-informant; one head of physiotherapy, one MsPT unit clinical coordinator, and three physiotherapists’ staffs from the MsPT unit were interviewed. Additionally, in-depth interview was done on 21 clients who were under MsPT treatment during the data collection period. The data was collected until the information was saturated.

Checklists were used for resource inventory, service setup adequacy, patients’ record review, and care providers’ technical compliance observation. The checklists consist of important indicators to examine the availability of important resources, the accommodation level of the unit, and whether the patients are appropriately diagnosed, physically examined and their medical information recorded according to the national physical rehabilitation service implementation standard (Annex C, D & E, Extended data, https://doi.org/10.7910/DVN/RE3EUL).

Exit interview by structured questionnaire was done with clients to determine their level satisfaction by their health care experience during visit in MsPT unit. To assure tools validity and reliability, questionnaire is adapted from “A brief questionnaire for assessing patient healthcare experience in low-income settings ³³ (Annex F, Extended data, https://doi.org/10.7910/DVN/RE3EUL).

Interview Guide: Interview guide was used to collect qualitative data from service coordinators, health care providers, and patients. In addition, field notebook & audio were used recorder documented information (Annex G & H, Extended data, https://doi.org/10.7910/DVN/RE3EUL).

To minimize observation bias the data collectors were recruited from other health facilities. The principal investigator undertook key informant interviews after the quantitative data collection was completed.

Quality control measures were undertaken to enhance the credibility and trustworthiness of the outcomes of the evaluation. The checklist for data collection was pretested on 10% of the total sample size at Mekelle hospital physiotherapy unit prior to actual data collection for checking the validity and consistency record completeness.

Quantitative

The quantitative data were cleaned for completeness and entered to Epi Data version 4.03 and exported to Statistical Package for Social Science (SPSS) version 25.0 for analysis (IBM Corp., Armonk, NY). For readers seeking an open-source alternative, equivalent analyses can be performed using JASP ( https://jasp-stats.org/) or R ( https://www.r-project.org/). Descriptive statistics was done and data were presented using frequency and percentage. Microsoft Excel was used to compute the evaluation of data obtained by observation. The calculated outputs were used to judge the overall service against the predetermined criteria (i.e. Excellent, Very Good, Good, Fair and Poor).

The qualitative data was collected and audio-recorded and later transcribed by two investigators separately and translated into English. After comparing each translation and checking the consistency, the data was entered, coded and analyzed by three coauthors using open code version 4.03, open source (available at: https://www.umu.se/en/department-of-epidemiology-and-global-health/research/open-code2/). ³⁴ Thematic analysis approach was employed to interpret the data. First, descriptions and specific ideas were coded separately, inter-related or similar codes were then grouped into different categories, and the categories were subsequently clustered into specific themes. Participant quotes were used to emphasize or support emerging themes in the result presentation (Table 2, Extended data, https://doi.org/10.7910/DVN/RE3EUL). Both the quantitative and qualitative findings were triangulated and presented together.

Musculoskeletal Physiotherapy (MsPT): Refers to an area of physiotherapy which deals with patients with musculoskeletal condition. ¹²

Process quality: In this evaluation, process quality defines as the alignment of MsPT unit and compliance of health care providers according to the 2018 national Rehabilitation guideline and service acceptability by the users. ¹² When these three dimensions’ minimum requirements fulfilled, the service will have good process quality.

Evaluation dimension: Important aspects of the service to be explored during the evaluation.

Availability: Means the presence and adequacy of both human and material resources of MsPT unit to deliver proper MsPT services to patients. Judged If >90 excellent, (80-90] very good, (70-80] good, [60-70] fair and <60 poor.

Accommodation: In this evaluation accommodation mainly refers to the adequacy musculoskeletal physiotherapy service setup to accommodate clients including the treatment room, waiting area, and external physical environment of the unit for safe access. Judged If >90 excellent, (80-90] very good, (70-80] good, [60-70] fair and <60 poor.

Technical compliance: Means the process of health care providers (physiotherapists) taking history, conducting physical examination and recording medical and follow-up information of patients at the outpatient musculoskeletal physiotherapy service according to national guideline. Judged If >90 excellent, (80-90] very good, (70-80] good, [60-70] fair and <60 poor.

Acceptability: In this evaluation we consider the service is acceptable by users when clients are satisfied by their health care experience during their visit in MsPT unit and adhere at their treatment follow-up appointments. Judged If >90 excellent, (80-90] very good, (70-80] good, [60-70] fair and <60 poor.

Adherence: The concept of adherence relates to attendance at treatment follow-up appointment.

For further clarity and understanding all indicators are operationalized at table 3 (Table 3, Extended data, https://doi.org/10.7910/DVN/RE3EUL )

For each dimension and indicator different guidelines and literatures were referred to assign the weight. ¹² After discussion & agreement with stakeholders, the final agreed weight was re-adjusted and given. The final evaluation judgment of the service was decided based on the criteria listed in (Table 4, Table 5, Extended data, https://doi.org/10.7910/DVN/RE3EUL ), and the result were presented in tables ( Tables 6, 7, 8, 9 & 10).

The study was conducted in adherence to the ethical principles outlined in the Declaration of Helsinki for research on human participant. ³⁵ Ethical approval to conduct the study was obtained from Mekelle University, Institutional Review Board (IRB) in March 15, 2020 G. C, with approval number IRB001. A support letter was subsequently issued by the university’s Research and Development Office to the Ayder Comprehensive Specialized Hospital’s physiotherapy department. Written informed consent was obtained from the head of the unit for the MsPT unit observation and chart review.

All participants were provided with comprehensive information regarding the study’s objectives and its importance for program improvement. Written informed consent was obtained from each participant, ensuring their understanding and voluntary agreement to partake in the research. Confidentiality was strictly maintained, and no unique identifiers were collected to ensure anonymity. Additionally, no compensation or reimbursement was provided to participants for their involvement. Full supporting documentation, including consent forms and the study protocol, can be found in the extended data ( Annexes A and B, https://doi.org/10.7910/DVN/RE3EUL).

Availability of resources: The compliance of the MsPT unit to the national physical rehabilitation service implementation standard was assessed based on the availability of various resources. The findings revealed that the overall evaluation of the availability dimension was rated poor (53.7%).

Availability of human resource: There was a shortage of five physiotherapists in the unit, as the current patient flow requires ten physiotherapists per shift. As a result, the unit was rated as ‘poor’ (50%) in ensuring the availability of licensed physiotherapists to deliver MsPT services ( Table 6).

According to key informant’s report, the shortage was due to physiotherapists being assigned to other duties, such as lecturing, and clients not adhering to follow-up schedules. This was best described by one physiotherapist (citation #1, Table 2).

Availability of premises: In MsPT unit, out of the fourteen required physical infrastructures, only nine were available, leading to a ‘fair’ rating (64.3%) for the availability of needed premises as per the service standard ( Table 6).

Availability of guidelines: During this evaluation, one manual of the 2018 FDRE Ministry of Health Physical Rehabilitation Service Guideline was available in the MsPT unit, which resulted in an ‘excellent’ rating (100%) for the availability of the service guideline ( Table 6).

Availability of clinical assessment tools: The study also revealed that only eight clinical assessment tools were available in the MsPT unit, with some key tools, such as the weight scale (which was non-functional), hand-held dynamometer, pinch gauge, inclinometer, megatoscope, floor scale, and spirometer, either missing or non-functional. Informants confirmed the unavailability of these tools before the evaluation. Consequently, the unit was rated ‘poor’ (53.3%) for the availability of the necessary clinical assessment tools ( Table 6).

Availability of therapeutic exercise equipment: The study identified a lack of essential therapeutic exercise equipment, including therapy putty, squeeze balls, power web, adjustable wrist and ankle weights, dumbbells, and other critical items. As a result, the MsPT unit was judged ‘poor’ (44.4%) for its compliance with the standard in fulfilling the necessary therapeutic exercise equipment ( Table 6).

The qualitative finding shows that the shortage of clinical assessment and therapeutic exercise equipment was primarily due to delays in purchasing new items and the unavailability of spare parts for damaged equipment. A physiotherapist with 14 years of experience emphasized this challenge (citation #2, Table 2). Similarly, another key informant states that budget shortage was the main challenge to avail the basic resources to the physiotherapy service (citation #3, Table 2).

We have asked physiotherapists what they do when they experience shortage of materials for patient examination or treatment. Accordingly, all the interviewed responded that, they customize already available materials in the unit to suit the individual requirements (citation #4, Table 2).

Accommodation of the service: The overall evaluation of the accommodation subdimension of the Compliance of the MsPT unit to the national physical rehabilitation service implementation standard was rated poor with 53.3% overall observed score. The evaluation of each component of the accommodation dimension is as bellow.

Adequacy of treatment room setup of MsPT unit: Direct observation with a standard checklist confirmed that only 55.6% of the treatment room setup in MsPT was in agreement with the standard. Out of the expected MsPT treatment delivery room setup to accommodate MsPT service clients only five of them were fulfilled. The most important components of the standard which are missed in the unit were the lack of private room for patients to change closing before and after treatment, separate wash room with bath room facility, functional running water in disabled accessible location, and ability of the unit to maintain visual privacy of clients. As a result, the adequacy of the treatment room setup in MsPT unit is rated ‘poor’ ( Table 7). This was also supported by the qualitative finding as stated by one patient with slipped disk (citation #5, Table 12).

Adequacy of waiting area setup of MsPT unit: The study findings revealed that out of the eight considered necessary components of waiting area setup only half is available in MsPT unit. Lack of adequate number of sits per patient per day, guide, audiovisual corner with TV for educating patients and their families, and supporting devices such as wheelchairs and stretchers are the most important components which were missed. Consequently, the adequacy of the waiting area setup in MsPT unit is rated ‘poor’ (50%) ( Table 7). The findings regarding the inadequacy of the waiting area setup of the MsPT unit was also supported by the clients’ report in the qualitative interview (citation #6, Table 2).

Adequacy of the physical environment setup of MsPT unit (for safe access): The adequacy of the physical environment setup of MsPT unit for safe access of patients to the facility is rated ‘poor’ (50%) due to the absence of disabled accessible parking and smooth pavement pathway in the unit ( Table 7). This compromises the safe access of disabled clients to the unit (citation #7, Table 2).

The treatment room, waiting area, and physical environment setups in the MsPT unit was also below the recommended (citation #8, Table 2). The main reason for the existence of MsPT unit setups below the standard was the lack of budget, as described key informant (citation #9, Table 2).

Direct non-participatory observation was conducted on forty client provider interaction (two providers with 20 clients each) to assess the compliance of MsPT service providers. Accordingly, the physiotherapists’ practice in appropriate clients’ history taking, physical examination and recording compliance is rated ‘fair’ (65%), excellent’ (100%) and poor’ (30.2%) respectively ( Table 8).

According to the qualitative study the main reason for lack of consistency in complying with guideline while taking history and documenting clients’ information was patient load (citation #10, 11, Table 2). Moreover, patients’ treatment follow-up assessment sheet was examined to investigate if the patients were assessed for MsPT treatment follow-up adherence. Accordingly, out of the 172 records 78 (45.3%) were assessed and 94(54.7%) not assessed. Thus, the providers’ practice in assessing the patients for their MsPT treatment follow-up adherence, during their follow-up schedule rests on 45.3% which is ‘poor’ ( Table 8).

The satisfaction of clients with their health care experience during their visit in MsPT unit was rated ‘ good’ 73.3%(11 out of 15) ( Table 9). The courtesy of physiotherapists in answering their questions, listening to their concerns, and clearly explaining treatment procedures along with their inclination to give advice to patients, were mentioned by clients as a source of satisfaction in their health care experience during their visit in MsPT unit (citation #12, 13, Table 2).

In contrast, the incongruity of number of patients with the number of health care provider in the MsPT unit, which resulted in inconsistency of time spent with care providers were mentioned by clients as a source of dissatisfaction in their health care experience during their visit in MsPT unit (citation #14, Table 2).

Providers’ attitude towards the MsPT service: We have asked physiotherapists how interested they are in providing MsPT service. Accordingly, they described that the patients recovery is their main source of satisfaction while absence of clear payment guideline and shortage of human resource were mentioned as main source of physiotherapists’ unhappiness while providing MsPT services (citation #15, 16, Table 2).

Provider-client relation: We have asked physiotherapists and key informants about the provider-client relation in MsPT unit. Key informants and health care providers mentioned the existence of a well-built provider-client relation. Even when problems arise, attempts will be made to solve problems on spot or bring issues to the department level to discus and take institutional measures (citation #17, 18, Table 2).

Clients’ adherence to treatment follow-up schedules: Review of patients’ treatment follow-up adherence assessment form indicated that out of the 78 client records assessed for adherence 57(73%) were adherent to the recorded schedules which is rated as ‘good’ ( Table 9). The minimum delay was one treatment follow-up session and the maximum was six treatment follow-up sessions.

Regarding the reason for lack of treatment follow up, unstructured interview was held with six MsPT patients to explore reasons that threaten their treatment follow-up adherence. The reasons reported by patients were: perceived wellness, lack of care giver, having work to do, and feeling of shame of losing one’s money (citation #19, 20, 21, 22, Table 2). Overall service judgment

The overall process quality of MsPT service was judged USING 14 selected indicators in three dimensions: observed compliance within the MsPT unit, technical compliance of physiotherapists, and acceptability. Results indicated that compliance levels were rated as poor (56.1%) for unit compliance, fair (69.1%) for technical compliance, and good (73.1%) for acceptability (Table 10, Extended data, https://doi.org/10.7910/DVN/RE3EUL). Accordingly, based on the weighted scores agreed upon by stakeholders and the observed evaluation results, the overall quality of the service process was classified as fair.

The evaluation included both qualitative, quantitative methods, and the study-maintained interest of stakeholders by incorporating their opinion to give relevant information for decision-making and to increase information use. However, due to the Covid-19 pandemic, the communication with the stakeholders was not regular and was limited. Moreover, during health care provider’s observation, even though an attempt was made to minimize the hawthorn effect, the magnitude of hawthorn effect was not assessed. Hindering and facilitating measured effect of the MsPT service towards other physiotherapy services in the physiotherapy department weren’t included in this evaluation study.