The QPM (see Figure 1) is based on the MSSS strategic plan (2029-2023 period) and framework for assessing the performance of the public healthcare and social services system ( MSSS, 2012), Accreditation Canada’s quality management framework ( Agrément-Canada, 2014), and PLANTREE’s person-centred approach ( Cosgrove, 1994). The QPM encompasses four dimensions of performance: customer, accessibility/quality, mobilisation, and optimisation. Each dimension presents an aspect of the CCSMTL performance and encompasses a number of sub-dimensions enriched by the perspectives of the customers and the community (see Table 1).

Accessibility/quality and customer are the two main dimensions of the QPM. They are presented in the form of a compass needle that points in the right direction to achieve better performance. Positioned at the centre of the model, customer dimension focusses on customer satisfaction regarding services and care provided. It presents three sub-dimensions: partnership, experience, and communication. Customers refer to people who receive a service or care and their relatives, as well as the community and the population in general. Accessibility/quality is the organisation’s ability to meet safely the needs and expectations of customers and ensure accessible and continuous services. It presents three sub-dimensions; accessibility, safety, and relevance/effectiveness of services. Mobilisation is the use of skills and talents of all people working in the organisation as well as fostering personal achievement and commitment to fulfil collectively the organisation’s mission. It encompasses four sub-dimensions, efficiency, agility, innovation, learning, and sustainability. Finally, optimisation is the search for continuous improvement in order to use efficiently available resources and provide services that are adapted to customer needs. Its sub-dimensions include healthy work environment, knowledge development, collaboration, and focus on people.

The CCSMTL also implemented its visual management system formed by a network of control rooms at different decision levels of the organisation. Put simply, a control room is a dedicated space where managers and employees meet regularly to acquire information about current situation and initiate discussions to improve future performance ( Lagacé et Landry, 2016). At the highest organisational level, we find the strategic room, which is used by the CEO and senior managers. Data is used to evaluate the organisational performance, monitor current strategy, and support strategic decision-making (e.g. prioritising major projects). Tactical rooms are deployed in the different programs and sub-programs. Managers discuss their departments’ performance and decisions to be made at their level. Some of these decisions flow-up to the strategic room for validation while others flow-down to ‘intermediate’ tactical rooms or operational control rooms to be executed, in a cascade-escalation mechanism ( MSSS, 2015). Some tactical rooms referred to as “intermediate tactical rooms” might play an intermediate role between tactical and operational levels. Operational control rooms (or visual stations) ensure day-to-day operations management ( MSSS, 2015). The QPM is aimed at being implemented in all CCSMTL control room networks to support performance management.

However, as mentioned earlier, QPM deployment is very challenging for the CCSMTL. First, it must be implemented at different decision-making levels, organisational structures, and territories. However, performance indicators may differ greatly from one service to another or from one territory to another. Second, some performance aspects such as customer and mobilisation dimensions are not sufficiently or well measured, which leads to an unbalanced performance management system. Finally, CCSMTL managers need to measure the overall performance of a given service or program, through integrated information. Currently, such information is not available nor easy to obtain.

A performance measurement system could be seen as the process of assessing the organisation’s progress in achieving its goals ( Kairu et al., 2013). It contains financial and non-financial measures ( Okwo & Marire, 2012). It should enable the correct deployment of the strategic and tactical objectives and provide a structured framework enabling the relevant feedback of information to the appropriate levels to facilitate decision-making and control ( Bititci et al., 1997). Botton et al. (2012) classified performance measurement systems according to three categories of approaches: strategy-oriented (BSC ( Kaplan & Norton, 1996), performance prism ( Neely et al., 2002)), hierarchical (performance pyramid), and process-based (macro-process model ( Brown, 2008)). The QPM can be considered as strategy-oriented. Due to its multi-dimensional and strategy-focussed aspects, it is consistent with the BSC model. In their recent systematic review, Amer et al. (2021) provided managers and policymakers with evidence to support using BSC in the healthcare sector. They conclude that BSC implementation demonstrated positive outcomes for patient satisfaction and financial performance. In their systematic literature review, Betto et al. (2022) attempted to understand the evolution of BSC in healthcare and their results revealed that the majority of studies focussed mainly on the BSC design process, rather than BSC implementation, use, or review. To Bohm et al. (2021), the composition of BSC development teams in healthcare should consider including patients as well as their families where appropriate and select highly achievable and valid performance measures aligned with the organisational strategy.

Developed in 1992 by Kaplan and Norton, the BSC is the result of a re-examination of performance measurement systems that usually give too much importance to the financial aspect. It is designed to measure four perspectives: finance (measures financial performance and the efficient use of financial resources), customer (concerns the organisation’s impact on the market in terms of customer satisfaction), internal processes (focusses on the efficiency and quality of internal processes) and learning & growth (concerns the culture and perspectives of the organisation, human resources, and information system). The term “balanced” underlines the concept of balance between all four axes of the model ( Kaplan & Norton, 1992). In BSC model, we distinguish leading indicators, which measure processes, and lagging indicators that measure outcomes. Causality is one of the most important aspects in the implementation of BSC: there is an implicit causal relationship between BSC perspectives and lag-lead indicators ( Porporato, Tsasis, et Vinuesa, 2017). To this end, a visual map (or strategy map) is built to show the objectives and causal links needed to execute a strategy. Scholey (2005) provides a well-structured six-step based process for creating a strategy map. Kaplan and Norton stressed the need to measure causal relationships, by using a complementary analytical approach ( Barnabè et al., 2012) even though, it is not easy to analytically validate all causal relationships ( Lorino, 2001; Schrage & Kiron, 2018). This is also the case for healthcare ( Porporato et al., 2017).

Canada leads in hospital performance measurement and historically, it is among the first countries to develop scorecards in healthcare ( Porporato et al., 2017). In Toronto, Ontario, Women’s College Hospital started using a scorecard based on key objectives in the 1990’s ( Porporato et al., 2017). Since then, Ontario hospitals have gained great experience in designing and using scorecards. Chan (2009) presented the evolution of healthcare systems in implementing BSC and strategy maps in Ontario. From 1992 to date, BSC has evolved from a strategic alignment tool (normative form) to a strategy construction tool. The normative version proposes a “top-down” deployment approach to operationalise the organisational strategy. Studies have shown that BSC can contribute to the emergence of a new strategy, and it would be better to opt for a “bottom-up” approach, where strategic objectives would be the result of collaborative management ( Choffel & Meyssonnier, 2005; Ittner & Larcker, 2003; Norreklit, 2000; Simons, 1994). In addition, the learning & growth axis should aim to shape the organisation’s culture. Kaplan (2020) presented the case of merging St. Mary’s Hospital and Duluth Clinic (Minnesota, US) who co-created a new strategy, a strategy map and a common BSC. Montalan and Vincent (2011) proposed an approach for the co-construction of the strategy of a French hospital. They developed and validated a strategy map by a multidisciplinary teamwork following a bottom-up approach. Figure 2 shows the evolution of BSC over time and application in healthcare.

In order to measure the performance while considering the different dimensions of the BSC (healthcare and other sectors), some studies in the literature combined the BSC with multi-criteria decision-making techniques (MCDM). Most of these studies used the well-known Analytic Hierarchy (or Network) Process (AHP/ANP) ( Anjomshoae et al., 2019; Chan, 2006; Modak et al., 2019; Regragui et al., 2018; Yaghoobi & Haddadi, 2016). Chan (2006) combined AHP and BSC to identify global metrics and compare performance across Canadian healthcare organisations. Regragui et al. (2018) also combined BSC and AHP to develop a framework for assessing performance in hospitals. By using AHP, the authors weighted different performance perspectives and indicators, leading to a ranking of these indicators. In other studies, AHP has been successfully used to consolidate, prioritise, and weight performance indicators ( Bentes et al., 2012; Hossain et al., 2019; Yaghoobi & Haddadi, 2016). For more on AHP applications in healthcare, see Hummel and IJzerman (2011). Recently, Daneshmand et al. (2022) developed a BSC-AHP based model for assessing the health system of the Social Security Organisation in Iran. They conclude that, the most important performance dimensions are internal processes and social obligations. Table 2 presents most recent BSC applications in healthcare including studies that combined BSC and AHP. Our proposed approach is inspired by the work of Regragui et al. (2018).

The data collection phase began in May 2019 (the study started in January 2019). Collected data was used to gain a more in-depth understanding of CCSMTL internal processes and current practices and progress regarding the implementation and use of its visual management system and QPM. To this end, three methods were used: review of documents, field observations, and interviews.

Documents reviewed include MSSS’s Law on healthcare and social services, Health Standards Organization (HSO) standards ( HSO, 2021), Accreditation Canada ( Agrément-Canada, 2014), Ministerial reference framework for evaluating performance in public healthcare and social services systems ( MSSS, 2012), MSSS strategic plan 2019-2023, CCSMTL annual reports ( CCSMTL, 2016), and PLANETREE documents (e.g. Cosgrove, 1994). Regarding field observations, we participated in the QPM Coordinating Committee and the QPM Advisory Committee meetings. The Coordinating Committee, which is composed of a limited number of managers, aims at insuring the implementation of the QPM within the CCSMTL, in accordance with the MSSS strategic plan. Periodically, the Coordinating Committee invites the heads of all CCSMTL departments to discuss decisions to be made related to the QPM and collect their feedback. This enlarged group forms the QPM Advisory Committee. In addition, we participated in a performance improvement workshop and visited several control rooms of the CCSMTL at different levels of the organisation: the strategic room, five tactical and intermediate rooms, and 10 operational rooms. Finally, we visited two hospitals and two rehabilitation centres to learn more about day-to-day healthcare operations. A total of 12 interviews were conducted with 13 managers (from different departments/services and decision-making levels) between May 2019 and September 2021. The objective was to validate and consolidate our understanding of the performance management and measurement system and tools being used and the progress in implementing the QPM and visual management system. The structured interview was used ( DiCicco‐Bloom and Crabtree, 2006) (i.e., the interviewer asks several structured questions, and the interviewee responds freely to the questions). The participants were selected in collaboration with the deputy director of organisational performance at the CCSMTL, who is responsible for QPM implementation. If required, a second meeting was scheduled with the same participant(s).

The SWOT method was used to structure and analyse the collected data. Our findings and conclusions are summarised in Table 3. These results have been validated by two senior managers of the CCSMTL involved in the QPM implementation.

One of the strengths identified is the well implementation of the QPM at the strategic level. There is strong interest by senior managers in aligning the organisation’s practices with QPM objectives. In addition, there are several practices put in place to ensure service quality and customer satisfaction and there is a real willingness to embed a customer-focussed culture through the implementation of purposeful projects. The deployment and use (even partially at the moment of conducting this study) of the visual management system at different decision-making levels demonstrates a real desire to build an effective management and measurement performance system. We also identified that there are enough indicators in the accessibility/quality dimension. Another important aspect is that managers at different hierarchical levels have the possibility to propose performance indicators in compliance with general ministerial objectives (even though some indicators are imposed by the MSSS). Moreover, each department holds regular meetings, which helps to maintain communication within the departments and support the information cascade-escalation process. While the QPM is well implemented at the strategic level, it is not sufficiently operationalised at the tactical and operational levels. We also identified that the QPM’s dimensions are unbalanced in terms of number of performance indicators used within each dimension. We find many indicators measuring the quality/accessibility dimension, but there is a lack of indicators in optimisation, mobilisation, and customer dimensions. There is also a lack of specific targets for those indicators. With the current performance management system, it is not possible for senior managers to track performance indicators at tactical/operational levels or to verify the overall performance without consulting a large amount of information. In addition, it is not possible to examine easily the performance of a specific service or department. There is a discontinuity in the assessment of performance between hierarchical levels. That is, each department/service measures its performance independently of the higher hierarchical level and other departments/services. We identified that dependencies and conflicts among some indicators exist, and this is challenging for managers to deal with. For example, there is a need to simultaneously reduce waiting time and improve direct service time (time with customers). An improvement in one indicator can decrease the other and vice versa. Regarding opportunities, the CCSMTL must continuously challenge itself to maintain/obtain accreditations and certifications such as PLANTREE, Accreditation Canada and to also comply with the MSSS’s strategic plans. Finally, like most organisations in a continuously changing socio-economic context, the CCSMTL might be threatened by pandemics such as COVID-19 or workforce shortage. These aspects can notably lead to disrupt service accessibility.

Based on the results of our SWOT analysis and inspired by previous studies in the literature, we propose to use BSC principles to create a more balanced QPM. To build our balanced-QPM, the four axes of the BSC and the four dimensions of the QPM are compared to each other. General causal relationships in this new balanced-QPM are also explicitly represented. Finally, the generic strategy map of the CCSMTL is derived from the resulting balanced-QPM.

Figure 4 shows the dimensions of the proposed balanced-QPM. BSC’s customer perspective corresponds to the customer dimension of the QPM. In the CCSMTL context, this dimension reflects the organisation’s mission and takes the first position in the balanced-QPM ( Figure 4). BSC’s internal process perspective is equivalent to the accessibility/quality dimension in the QPM. We recall that accessibility/quality refers to the ability to safely meet customer needs and expectations by providing accessible and continuous services. This dimension takes the second position in the balanced-QPM, forming with the customer perspective the “True North” of the CCSMTL. BSC’s learning & growth aspect is found in both mobilisation and optimisation dimensions of the QPM. Learning & growth presents three main components: human capital, information capital, and organisational capital. Human capital refers to the management of skills and behaviours of employees. This aspect is part of the mobilisation dimension of the QPM (use of skills and talents of everyone in the organisation, partners, customers, and their families, encouraging personal development and commitment to accomplish the mission of the CCSMTL). The mobilisation dimension takes the third position in the balanced-QPM. Organisational capital is defined as the organisation’s ability to align employee objectives with the strategy while information capital is the way organisations use their information systems, networks, and databases to support accomplishing their operations. Both aspects are part of the optimisation dimension of the QPM. We recall that optimisation dimension aims at continuous improvement and innovation to ensure, over time, that the services offered are adapted to the needs of customers. BSC’s financial perspective is also part of optimisation dimension. Kaplan and Norton ( Kaplan & Norton, 1999) stated that the financial perspective serves as a constraint, not an objective, for non-profit organisations such as healthcare systems. Therefore, optimisation dimension, takes the last position in the balanced-QPM.

Figure 4 shows that the customer dimension of the balanced-QPM corresponds to the organisation’s mission, and the other three dimensions to the CCSMTL’s capacity to fulfil this mission. Additionally, Figure 4 shows global causal links between the four dimensions. This aspect was absent in the current QPM. The optimisation dimension has direct impact on the accessibility/quality dimension. There is also a mutual relationship between optimisation and mobilisation. Mobilisation has an impact on the accessibility/quality of services provided by the CCSMTL, which in turn has direct impact on customer satisfaction. Finally, customers put pressure on the MSSS to provide the necessary funding supporting the CCSMTL capacity in terms of optimisation, mobilisation, accessibility/quality aspects.

To build the CCSMTL’s strategy map, we focus on the six-step guidelines suggested by Scholey (2005) adapted to a non-profit organisation. The first step is to determine the primary objective, which is the desired end result once the strategy is implemented. For the CCSMTL, the primary objective is to satisfy its customers by providing general and specialised healthcare and social services while insuring accessibility, security, efficiency and quality. An example of a target to achieve for this primary objective within a specific time frame would be to increase the access delay compliance rate to 95% within one year. The second step is to select the appropriate value proposition. According to Treacy and Wiersema (2007), there are three value propositions for an organisation: operational excellence, product excellence and customer relationship. Since the CCSMTL follows a customer-focussed approach, its value proposal focusses on customer experience. The third, fourth, fifth, and last step aims at identifying the objectives of the customer, accessibility/quality, mobilisation, and optimisation dimension, respectively. The determination of these objectives can be guided by both the QPM sub-dimension definitions enriched by the customer perspective ( Table 1) and objectives set by the department managers, in relation to their specific mission. Rigorous criteria for selecting relevant objectives such as specificity, validity, sustainability, reliability, and utility were used to support this process. Figure 5 presents the generic strategy map of the CCSMTL that can be used as a basis for building “customised” strategy maps for different departments.

The causal links between the four dimensions shown in Figure 5 correspond to those identified in Figure 4. For the customised strategy maps, these causal links should be defined more precisely. This is illustrated with the DI-TSA-DP program case study. Validating analytically these causal links is beyond the scope of our study (time will be required to generate results and validate the causal links).

This phase aims at providing relevant indicators measuring performance following the objectives specified in the strategy map in each performance dimension.

We propose to create a structured system that provides indicators as well as performance indexes for each performance dimension at each hierarchical level of the CCSMTL. This general structure is based on the CCSMTL organisational chart and its visual management system. To provide and monitor the performance indicators, scorecards presenting the four dimensions (and their indicators) can be used at each department and hierarchical level, forming a network of scorecards (information is presented and discussed in the control rooms during regular meeting) ( Figure 6). This proposal is inspired from the study of Voyer ( Voyer & Voyer, 1999) that developed a network of interrelated dashboards used at different levels within an organisation. Béland and Abran (2004) also used this concept.

From the established strategy map, performance indicators are selected for each dimension following strategic objectives that are relevant to the department/service. Existing indicators should be reviewed first. Second, additional indicators are proposed according to criteria such as specificity, validity, sustainability, reliability, utility, etc. At the end of this step, each department will have a set of indicators for each dimension of the Balanced-QPM called “before normalisation indicators” ( X bn ).

The “before normalisation indicators” may have different units of measurement and different magnitudes. In some cases, the measurement unit is expressed in monetary units (e.g. expenses), and in others, in time units (e.g. hours) such as waiting time. In other cases, there is no measurement unit (e.g. number of services provided). In addition, the same indicator may have distinct targets in different departments. For example, waiting time target in an emergency department is significantly less compared to psychosocial services. Therefore, it might be difficult to compare them to each other based on their absolute values. Due to this, normalising the indicators is very helpful. Normalisation techniques for processing data have been used by many researchers in different fields. We refer the reader to Singh and Singh (2020) for more on normalisation. We selected the Min-Max normalisation technique ( Singh & Singh, 2020), mainly because minimum and maximum limits can be easily distinguished in available data provided by the CCSMTL. Following this technique, data is usually rescaled within the range 0 to 1 or (-1) to 1. The general equation is given as follows: X norm = Nmin − Nmax X bn − min max − min + Nmin (1)

X norm : Normalised value of the indicator

X bn : Indicator value before normalisation

min : Minimum value in the data

max : Maximum value

Nmin Nmax : Normalised Min and Max

For the CCSMTL context X norm is expressed as a percentage ranging from 0% to 100%, therefore Nmin et Nmax values are 0% and 100%, respectively. Given that in some cases the maximum value of an indicator is the undesirable value while the minimum value is the desired value (e.g. the maximum value for waiting time is non-desired while the minimum value is desired), we set the best value in the data set as max parameter (which is not necessarily the highest value) and the worst value in the data as min (which is not necessarily the lowest value) before normalisation. Outliers must be eliminated before choosing the best and worst values. To link a given indicator to a desired objective and to ensure its continuous improvement, max value corresponds to the indicator target. For each performance dimension, there is a set of associated indicators. Each indicator takes a numerical index i in the set and each indicator has an index dim referring to the first letter of the dimension to which it belongs (e.g. C for Customer). Indicators before normalisation take the index bn, referring to “before normalisation”. Normalised indicators in the original formula ( Equation (1)) X norm become X i dim in the new formula adapted to the CCSMTL ( Equation (2)). We remove the index norm for the sake of simplicity. X i dim = 100 × Xbn i dim − min i target i − min i (2)

Each department/service at different hierarchical levels calculates its normalised indicators ( X i C , X i A , X i M , X i O ). These are the inputs of the fourth step.

Step 4 aims at weighting the normalised indicators of the same dimension at the same hierarchical level. AHP technique is used. AHP is usually used to evaluate and rank a set of evaluated based on several criteria. In the CCSMTL context, there is no need to rank or choose an option, but rather weighting the indicators within each performance dimension and aggregating them to obtain performance indexes for each dimension for a given department (see Figure 7). We selected AHP due its simplicity ( Kumar et al., 2017) since it is very convenient for decision-makers who are not familiar with analytical tools. Moreover, it relies on pairwise comparisons, and according to Ishizaka and Labib ( Ishizaka & Labib, 2011), psychologists argue that it is easier to express an opinion based on two elements rather than on all elements simultaneously. Figure 8 shows how AHP is adapted to our study.

Criteria and indicators that are usually used in AHP, correspond to the four dimensions of the QPM and to the normalised indicators X i dim , respectively. The normalised indicators are compared to each other by using pair-wise comparison matrices (one matrix for each dimension) according to their relative importance regarding the dimension to which they belong. These preferences are expressed by managers. A given element a _ij of a matrix A corresponds to the relative importance of indicator i (in the row) compared to indicator j (in the column). The preference values range from one to five, which is adapted from the preference scale proposed by Saaty (2005). For example, a _ij = 1 means that the indicators are equally important and a _ij = 5 that indicator i is extremely important compared to indicator j. The weights of the performance indicators are calculated by using a specific algorithm described in detail by Brunelli (2014). All AHP calculation steps are presented in Appendix A. Comparisons make sense only if the pair-wise comparison matrices are coherent or quasi-coherent. A consistency ratio (CR) is used to check this aspect. If CR is less than 10%, the matrix is consistent, otherwise, the pair-wise comparisons are repeated (for more details see equations 4, 5 and 6 provided in Figure 10). Once consistency is validated, the weights of the indicators, W i dim ( W i C , W i A , W i M , W i O ) are obtained.

Indicators within each dimension are aggregated (see Figure 8), and four performance indexes P C P A P M P O are calculated (one index for each dimension) following Equation (3): P dim = ∑ i n W i dim × X i dim (3)

P dim : Performance index of dimension dim

W i dim : the weight of indicator i within the performance dimension dim

X i dim : i th normalised indicator within performance dimension dim

DI-TSA-DP case study

To illustrate how phases III and IV could be applied in practice, the “Physical disability, intellectual disability, and autism spectrum disorder” (DI-TSA-DP) program was proposed by the CCSMTL. First, DI-TSA-DP program is presented. Second, its strategy map is proposed (Phase III of our approach). Finally, the preliminary results of Phase IV (PMMS deployment) are presented, and a short discussion concludes this section.

The CCSMTL encompasses nine programs of which three; physical disability (DP), intellectual disability (DI), and autism spectrum troubles (TSA), are grouped together to form the joint program DI-TSA-DP. It provides services to 2,200 customers in local community service and rehabilitation centres. It receives approximately 10,000 service requests yearly and it encompasses 2,150 employees and 58 physicians. It includes three main sub-programs; DI-TSA, which provides services to customers with an intellectual disability or autism spectrum disorder, DP, which provides services to customers with a physical disability, and RMVS, which provides rehabilitation services in substitute living environments such as elderly homes. DI-TSA-DP mission is to provide specific, specialised and super-specialised habilitation and rehabilitation services to its customers to promote their integration and social participation.

At the moment of conducting this study, DI-TSA-DP has put in place one tactical room (top management level of the program), and three intermediate control rooms at the sub-program level (DI-TSA, DP, and RMVS). Operational rooms are established at the team unit level within the three sub-programs. Control room deployment at the team unit level was still in progress. Therefore, each sub-program had only reached a certain deployment rate, which is seen as a measure of performance-culture maturity level in the CCSMTL. Following the recommendation of our collaborators, we focussed on DP sub-program, since it had the highest deployment rate. The approach can be applied to the two other sub-programs as well. DP includes three services: AT (Technical Aids), which provides special assistance devices (prosthesis, wheelchair, etc.), LN (Locomotor-Neurology), which provides services to customers with physical disabilities due to neurological or locomotor disorders, and SL (Sensory-Language), which provides services to customers with linguistic or sensory disabilities. We focus on LN service, again based on its operational room deployment rate (100%). There are nine care team units within LN service, and each unit has its own operational room. We worked in close collaboration with two teams, Locomotor team (A-BOG), which provides services for customers with an amputation or serious orthopaedic injuries, and Neurological team (AVC), which provides services for customers who have had a stroke ( Figure 9).

Prior to the application of our approach to DI-TSA-DP, we organised five meetings with DI-TSA-DP managers (strategic and tactical levels) in order to understand the specific mission and structure of DI-TSA-DP, collect data regarding the QPM deployment within the program, and describe our approach to the managers. These meetings also enriched the SWOT analysis (see Table 3). One conclusion was the absence of a well-structured performance indicator system within DI-TSA-DP. Moreover, there were no explicit links between the QPM dimensions, sub-dimensions, and DI-TSA-DP strategic objectives. Existing objectives were set independently from the sub-dimensions and performance indicators were not explicitly related to these objectives. The strategy map built for DI-TSA-DP presented in the next paragraphs contributes to addressing these lacks.

The proposed strategy map for DI-TSA-DP ( Figure 8) is consistent with the generic strategy map ( Figure 5). It was co-created with the DI-TSA-DP program executive manager.

It illustrates objectives determined following the QPM sub-dimensions (including the primary objective) and causal links established between these objectives and/or sub-dimensions. The objectives selected are based on the internal DI-TSA-DP strategic vision 2018-2021 provided by DI-TSA-DP executive manager. Each objective was checked against the sub-dimensions of the QPM to identify to which one it is related. In total 16 objectives were classified within the sub-dimensions and dimensions of the QPM. The DI-TSA-SP mission was chosen as the primary objective and added in the customer dimension. Since no objective exists for customer experience, a new objective was proposed based on customers’ definition of this sub-dimension ( Table 1). Additional objectives should be set for this sub-dimension and other ones that currently do not present any specific objectives (i.e., communication, learning and focus on people).

Examples of causal links in the strategy map, are the relationship between objectives within customer sub-dimensions (‘partnership’ and experience’) and the primary objective (DI-TSA-DP mission). It is clear that positive customer experience, customer involvement as a partner in the system, and providing a climate of trust for the customers contribute to achieving DI-TSA-DP mission. It is interesting to notice that causal links also exist between sub-dimensions/objectives within the same dimension. As an example, partnership objectives contribute to achieving customer experience objectives. That is, when customers are involved as partners in the system, or when a climate of trust and collaborations is established with them, they are likely to have a positive experience. Some objectives/sub-dimensions mutually impact each-other, e.g. efficiency (optimisation dimension) and healthy work environment (mobilisation dimension). Reducing indirect labour and extra working hours (efficiency sub-dimension) leads to a healthy work environment, which in turn has a positive impact on efficiency.

We conducted five working meetings of two hours each with different DI-TSA-DP managers. We started with the operational level (A-BOG and AVC) and moved to the tactical level (LN and DP service and sub-program, respectively), and finally, to the highest level (DI-TSA-DP program). The following paragraphs present the process of applying the developed PMMS to DI-TSA-DP case.

Figure 9 shows a simplified organisational chart of DI-TSA-DP program. This flowchart shows only the departments needed to illustrate our approach. We distinguish four levels: 1) CCSMTL strategic level, 2) tactical level, which includes DI-TSA-DP program and DI-TSA, DP, and RMVS sub-programs, and 3) operational level (within DP sub-program) that includes SL, LN, and AT teams.

Figure 9 also shows the network of control rooms deployed in these different levels. The performance indicators structure follows the organisational chart and control room network structure. They are presented in the form of scorecards following the four QPM dimensions (indicators, values and targets, performance indexes).

To apply the rest of Phase IV steps, LN service was considered, and the process was performed in collaboration with LN coordinator. First, indicators being used by LN service were classified according to the four dimensions of the QPM and the strategy map’s sub-dimensions/objectives. Next, we checked the relevance of each performance indicator (against the criteria specificity, validity, simplicity, relevance, sustainability, reliability, and utility). When an indicator does not meet all criteria, it is either changed for another indicator that satisfies the criteria, or more information is gathered to ensure that the indicator meets all criteria. As an example, indicator ‘Customer satisfaction rate’ ( Xan 1 C ) ( Table 4), which was proposed to measure customer experience sub-dimension objective, did not meet simplicity criterion. LN managers did not have information on customers’ satisfaction and did not put in place specific methods to collect this information. To address this issue, we relied on the results of a survey used to collect the feedback of customers regarding their satisfaction level regarding experience and service quality. Table 4 shows the 10 final indicators selected for LN service.

The columns of Table 4 show the indicators measuring the objectives (and therefore sub-dimensions) selected in previous step, their notations, their values before normalisation, their targets, their worst values, and their values after normalisation (obtained by using equation (2)).

Due to confidentiality reasons, all real values of the indicators have been modified. Note that indicators that are already expressed as percentages before normalisation and which are aimed to be maximised to reach the desired target have not been normalised. The worst value is set to zero for those indicators (e.g. Customer satisfaction rate, Bed occupancy rate, and Respect of access deadlines) . Moreover, indicators before normalisation meant to be minimised to reach their target have been converted into indicators to be maximisation after normalisation (e.g. Complaint rate and Ratio of salary insurance hours).

Pairwise comparisons of normalised indicators within each QPM dimension were performed with the LN coordinator. The values obtained were entered in a VBA program that checks the consistency of the matrices and generates the weights following AHP algorithm (see Appendix A). The final results that represent the weights of the indicators are presented in Table 5.

Step 5 enables us to calculate aggregated performance indexes for each dimension that can be used by the service managers or by higher hierarchy level departments (DP sub-program and DI-TSA-DP programs in our case) to evaluate how the service performs globally in each dimension of the QPM. For instance, if the global performance of a given dimension is not satisfying, the performance indicators within that dimension should be analysed in more detail to identify the problem and take actions to improve the situation. The results of calculating the four performance indexes PI LN C PI LN A PI LN M PI LN O of LN are given in Table 5. Each performance index corresponds to one dimension of the balanced-QPM. Again, the VBA program is used to automate these calculations following AHP algorithm. Figure 10 provides more details about the AHP calculation steps Figure 10. .