Towards an Integrated Performance Management and Measurement System for healthcare organisations: a case study in Montreal

Current situation analysis (Phases I and II)

The data collection (phase I) began in May 2019 (the study started in January 2019). Collected data was used to gain a more in-depth understanding of CCSMTL internal processes and current practices and progress regarding the implementation and use of its visual management system and QPM (deployment level, indicators, implementing obstacles and issues, etc.). To this end, three methods were used: review of documents, field observations, and structured interviews (see Table 3).

The Coordinating Committee of the CCSMTL is composed of a limited number of managers from the Quality, Evaluation, Performance and Ethics department. It aims at implementing the QPM within the CCSMTL, in accordance with the MSSS strategic plan. Periodically, the Coordinating Committee invites the heads of all CCSMTL departments to discuss decisions to be made related to the QPM and collect their feedback. This enlarged group forms the QPM Advisory Committee. The objective of the interviews (with a duration of 1 to 2 hours) was to validate and consolidate our understanding of the performance management and measurement methods and tools being used and the progress in implementing the QPM and visual management system. The structured interview was used (DiCicco‐Bloom and Crabtree, 2006) the interviewer asks several structured questions, and the interviewee responds freely to the questions. The participants were selected in collaboration with the deputy director of organisational performance at the CCSMTL, who is responsible for QPM implementation. When required, a second meeting was scheduled with the same participant(s). For more details, the reader may refer to Ben Fradj et al. (2020).

Next (Phase II), the SWOT (Strengths, Weaknesses, Opportunities, Threats) method was used to structure and analyse the collected data. SWOT analysis is a popular tool from which strategies can be developed and which improves organisational performance (Pickton and Wright, 1998). It helps to clarify the strengths to be maintained, the weaknesses to be addressed, the opportunities to benefit from, and the constraints to be handled. In our case, we used the SWOT to identify the strengths and weaknesses related to the current implementation of the QPM model. Once the data was collected, the research team brainstormed to cocreate the preliminary version of the SWOT matrix. Next, four discussion meetings have been organised with three different managers from the CCSMTL to validate and complement the final version of the SWOT (see Table 4 in the Results Section).

PMMS Development (Phases III and IV)

Balanced-QPM and strategy map development (Phase III)

Based on the results of our SWOT analysis and inspired from the literature, we use BSC principles to create a more balanced QPM. In other words, we use the BSC in order to address the weaknesses of the QPM model identified in our SWOT analysis. Indeed, by comparing BSC and QPM models, we identified similarities and complementary aspects. First, there is a strong similarity between the dimensions of performance of the QPM and the axes of the BSC, in the way that they express equivalent pillars of performance (customer, internal processes, resource management, etc.). Second, both models enable defining performance indicators and creating a measurement system. Third, the QPM and the BSC provide a basis for stakeholders to share and discuss strategic objectives. Finally, the BSC is often recommended in the healthcare sector as mentioned in the literature review section.

First, the axes of the BSC, i.e., the version adapted for non-profit organisations by Kaplan, and the dimensions of the QPM are compared to each other. For example, BSC’s customer perspective corresponds to the customer dimension of the QPM. In the CCSMTL context, achieving the goals of this dimension reflects the organisation’s main mission and therefore would take the first position in the Balanced-QPM. Similarly, BSC’s internal process perspective is equivalent to the accessibility/quality dimension in the QPM. We recall that accessibility/quality refers to the ability to safely meet customer needs and expectations by providing accessible and continuous services. This dimension would take the second position in the Balanced-QPM. The positioning of customer and accessibility/quality dimensions is consistent with the CCSMTL vision which considers the two dimensions as the “True North” of the organisation. Second, based on the causal relationship between the BSC axes, we identify the causes and effect links specific to the dimensions of the QPM model. This aspect was absent in the intitial QPM. For instance, the optimisation dimension has direct impact on the accessibility/quality dimension, which in turn has a direct effect on the customer dimension. The result of these two processes (positioning the QPM dimensions and identifying causal relationships between them) leads to what call the Balanced-QMP (see Figure 5 in the Results Section).

To create the CCSMTL’s strategy map, the causal links between the Balanced-QPM dimensions are further analysed. We were inspired by the six-step guidelines suggested by Scholey (2005). To the author, this process leads to the creation of a comprehensive strategy map, ensuring a clear and communicable strategy throughout the organization. It consists in 1) identifying the primary objective, 2) selecting the value proposition, 3) determining general financial strategies to follow, 4) establishing customer-focused strategies, 5) aligning internal processes to support strategy execution, and finally 6) implementing necessary skills, capabilities, and employee programs to achieve the strategy.

For the CCSMTL, the primary objective is to satisfy its customers by providing general and specialised healthcare and social services while insuring accessibility, security, efficiency and quality. An example of a target to achieve for this primary objective would be to increase the access delay compliance rate to 95% within one year. For the value proposition selection, we relied on the study by Treacy and Wiersema (2007), who identified three value propositions; operational excellence, product excellence and customer relationship. Since the CCSMTL follows a customer-oriented approach, its value proposal focusses on customer relationship. The remaining step are based on the QPM sub-dimension definitions enriched by the customer perspective (see Table 1) that could be further expanded by the objectives set by the department managers, in relation to their specific mission. In addition, rigorous criteria for selecting relevant objectives such as specificity, validity, sustainability, reliability, and utility could be used to support this process. The resulting strategy map is shown in Figure 6 in the Results Section.

Indicator structure design and performance assessment (Phase IV)

This phase aims at providing relevant indicators measuring performance following the objectives specified in the strategy map in each performance dimension.

Step 1: Indicator system structure

We propose to create a structured system that provides indicators as well as performance indexes for each performance dimension at each hierarchical level of the CCSMTL. This general structure is based on the CCSMTL organisational chart and its visual management system. To provide and monitor the performance indicators, scorecards presenting the four dimensions (and their indicators) can be used at each department and hierarchical level, forming a network of scorecards (information is presented and discussed in the control rooms during regular meeting. This proposal is inspired from the study of Voyer (Voyer & Voyer, 1999) who developed a network of interrelated dashboards used at different levels within an organisation. Béland and Abran (2004) also used this concept (see Figure 7 in the Results Section).

Step 2: Indicator identification and selection

From the established strategy map, performance indicators are selected for each dimension following strategic objectives that are relevant to the department/service. Existing indicators should be reviewed first. As suggested by Bull and Callander (2022) and Nuti et al. (2017) patient-reported measures and indicators based on administrative data can be used. Existing indicators and new ones (selected based on the literature or on recognized standards) should be elected according to criteria ensuring specificity, validity, sustainability, reliability, utility, etc. Such criteria are well documented in the literature for selecting relevant performance objectives and indicators. At the end of this step, each department will have a set of indicators for each dimension of the Balanced-QPM called “before normalisation indicators”, noted as Xbn in the following paragraphs.

Step 3: Indicator normalisation

The “before normalisation indicators” Xbn may have different units of measurement and different magnitudes in healthcare contexts. In some cases, the measurement unit is expressed in monetary units (e.g. expenses), and in others, in time units (e.g. hours) such as waiting time. In other cases, there is no measurement unit (e.g. number of services provided). In addition, the same indicator may have distinct targets in different departments. For example, waiting time target in an emergency department is significantly less compared to psychosocial services. Therefore, it might be difficult to compare them to each other based on their absolute values. Due to this, normalising the indicators is very helpful. Normalisation techniques for processing data have been used by many researchers in different fields. We refer the reader to Singh and Singh (2020) for more on normalisation. We selected the Min-Max normalisation technique (Singh & Singh, 2020), mainly because minimum and maximum limits can be easily distinguished in available data provided by the CCSMTL. Following this technique, data is usually rescaled within the range 0 to 1 or (-1) to 1. The general equation is given as follows:

With:

$X_{\mathit{norm}}$ : Normalised value of the indicator

$X_{\mathit{bn}}$ : Indicator value before normalisation

$\mathit{min}$ : Minimum value in the data

$\mathit{max}$ : Maximum value

$\mathit{\text{Nmin}};\mathit{\text{Nmax}}$ : Normalised Min and Max

For the CCSMTL context $X_{\mathit{norm}}$ is expressed as a percentage ranging from 0% to 100%, therefore $\mathit{\text{Nmin}}$ et $\mathit{\text{Nmax}}$ values are 0% and 100%, respectively. Given that in some cases the maximum value of an indicator is the undesirable value while the minimum value is the desired value (e.g. the maximum value for waiting time is non-desired while the minimum value is desired), we set the best value in the data set as $\mathit{max}$ parameter (which is not necessarily the highest value) and the worst value in the data as $\mathit{min}$ (which is not necessarily the lowest value) before normalisation. Outliers must be eliminated before choosing the best and worst values. To link a given indicator to a desired objective and to ensure its continuous improvement, $\mathit{max}$ value corresponds to the indicator target. For each performance dimension, there is a set of associated indicators. Each indicator takes a numerical index $i$ in the set and each indicator has an index dim referring to the first letter of the dimension to which it belongs (e.g. C for Customer). Indicators before normalisation take the index bn, referring to “before normalisation”. Normalised indicators in the original formula ( Equation (1)) $X_{\mathit{norm}}$ become $X_i^{\mathit{dim}}$ in the new formula adapted to the CCSMTL ( Equation (2)). We remove the index norm for the sake of simplicity.

Each department/service at different hierarchical levels calculates its normalised indicators ( $X_i^C$ , $X_i^A$ , $X_i^M$ , $X_i^O$ ). These are the inputs of the fourth step.

Step 4: Indicator weighting

Step 4 aims at weighting the normalised indicators of the same dimension at the same hierarchical level. AHP technique is used. AHP is usually used to evaluate and rank a set of evaluated based on several criteria. In this study, there is no need to rank or choose an option, but rather weighting the indicators within each performance dimension and aggregating them to obtain performance indexes for each dimension for a given department (see Figure 4). We selected AHP due its simplicity (Kumar et al., 2017) since it is very convenient for decision-makers who are not familiar with analytical tools. Moreover, it relies on pairwise comparisons, and according to Ishizaka and Labib (Ishizaka & Labib, 2011), psychologists argue that it is easier to express an opinion based on two elements rather than on all elements simultaneously.

Figure 4. AHP adaptation for performance measurement for the CCSMTL case.

Criteria and indicators that are usually used in AHP, correspond to the four dimensions of the Balanced-QPM and to the normalised indicators $X_i^{\mathit{dim}}$ , respectively. The normalised indicators are compared to each other by using pair-wise comparison matrices (one matrix for each dimension) according to their relative importance regarding the dimension to which they belong. These preferences are expressed by managers. A given element a _ij of a matrix A (see Equation (4) below) corresponds to the relative importance of indicator i (in the row) compared to indicator j (in the column). The preference values range from one to five, which is adapted from the preference scale proposed by Saaty (2005). For example, a _ij = 1 means that the indicators are equally important and a _ij = 5 that indicator i is extremely important compared to indicator j. The weights of the performance indicators are calculated by using a specific algorithm described in detail by Brunelli (2014). All AHP calculation steps are presented in the next paragraphs. Comparisons make sense only if the pair-wise comparison matrices are coherent or quasi-coherent. A consistency ratio (CR) is used to check this aspect. If CR is less than 10%, the matrix is consistent, otherwise, the pair-wise comparisons are repeated (see Equations (5) and ( 6) below). Once consistency is validated, the weights of the indicators, noted as $W_i^{\mathit{dim}}$ ( $W_i^C,W_i^A,W_i^M,W_i^O)$ are obtained.

Step 5: Indicator aggregation

Indicators within each dimension are aggregated (see Figure 8), and four performance indexes $(P^C,P^A,P^M,P^O)$ are calculated (one index for each dimension) following Equation (3):

${\mathit{P}}^{\mathit{dim}}$ : Performance index of dimension dim

$W_i^{\mathit{dim}}$ : the weight of indicator i within the performance dimension dim

$X_i^{\mathit{dim}}$ : ith normalised indicator within performance dimension dim

Equations (4) to (6) provide more details about the AHP calculation steps.

With:

A: pair-wise comparison matrix

n: number of indicators to be compared, which corresponds to the matrix size

$\mathrm{A}={\mathrm{a}}_{\mathrm{ij}}$ , ${\mathrm{a}}_{\mathrm{ij}}>0$ ; $\mathrm{i},\mathrm{j}\in ⟦1,\mathrm{n}⟧$

$\frac{1}{{\mathrm{a}}_{\mathrm{ij}}}={\mathrm{a}}_{\mathrm{ji}}$ $\forall \mathrm{i},\mathrm{j}\in ⟦1,\mathrm{n}⟧$

${\mathrm{a}}_{\mathrm{ij}}$ =1 $\forall \mathrm{i}=\mathrm{j}$

2. Calculation of consistency ratio (CR)

With:

CR: Consistency ratio

CI: Consistency index

RI: Random index (depends on n)

${\mathrm{\lambda }}_{\max }$ : Maximum eigenvalue of the matrix

n: Matrix size

Results of the SWOT analysis

From Table 4, we can see that one of the strengths identified is the well implementation of the QPM at the strategic level. There is strong interest by senior managers in aligning the organisation’s practices with QPM objectives. In addition, there are several practices put in place to ensure service quality and customer satisfaction and there is a real willingness to embed a customer-focused culture through the implementation of purposeful projects. The deployment and use (even partially at the moment of conducting this study) of the visual management system at different decision-making levels demonstrates a real desire to build an effective management and measurement performance system. We also identified that there are enough indicators in the accessibility/quality dimension. Another important aspect is that managers at different hierarchical levels have the possibility to propose performance indicators in compliance with general ministerial objectives (even though some indicators are imposed by the MSSS). Moreover, each department holds regular meetings, which helps to maintain communication within the departments and support the information cascade-escalation process.

While the QPM is well implemented at the strategic level, it is not sufficiently operationalised at the tactical and operational levels. We also identified that the QPM’s dimensions are unbalanced in terms of number of performance indicators used within each dimension. We find many indicators measuring the quality/accessibility dimension, but there is a lack of indicators in optimisation, mobilisation, and customer dimensions. There is also a lack of specific targets for those indicators. With the current performance management system, it is not possible for senior managers to track performance indicators at tactical/operational levels or to verify the overall performance without consulting a large amount of information. In addition, it is not possible to examine easily the performance of a specific service or department. There is a discontinuity in the assessment of performance between hierarchical levels. That is, each department/service measures its performance independently of the higher hierarchical level and other departments/services. We identified that dependencies and conflicts among some indicators exist, and this is challenging for managers to deal with. For example, there is a need to simultaneously reduce waiting time and improve direct service time (time with customers). An improvement in one indicator can decrease the other and vice versa. Regarding opportunities, the CCSMTL must continuously challenge itself to maintain/obtain accreditations and certifications such as PLANTREE, Accreditation Canada and to also comply with the MSSS’s strategic plans. Finally, like most organisations in a continuously changing socio-economic context, the CCSMTL might be threatened by pandemics such as COVID-19 or workforce shortage. These aspects can notably lead to disrupt service accessibility.

Balanced-QPM, strategy map and indicator structure system (CCSMTL level)

Figure 5 shows the dimensions of the proposed Balanced-QPM. BSC’s customer perspective corresponds to the customer dimension of the QPM. BSC’s internal process perspective is equivalent to the accessibility/quality dimension in the QPM. Learning & growth axis is found in both mobilisation and optimisation dimensions of the QPM. Learning & growth presents three main components: human capital, information capital, and organisational capital. Human capital refers to the management of skills and behaviours of employees. This aspect is part of the mobilisation dimension of the QPM (use of skills and talents of everyone in the organisation, partners, customers, and their families, encouraging personal development and commitment to accomplish the mission of the CCSMTL). The mobilisation dimension takes the third position in the Balanced-QPM. Organisational capital is defined as the organisation’s ability to align employee objectives with the strategy while information capital is the way organisations use their information systems, networks, and databases to support accomplishing their operations. Both aspects are part of the optimisation dimension of the QPM. We recall that optimisation dimension aims at continuous improvement and innovation to ensure, over time, that the services offered are adapted to the needs of customers. BSC’s financial perspective is also part of optimisation dimension. Kaplan & Norton (1999) stated that the financial perspective serves as a constraint, not an objective, for non-profit organisations such as healthcare systems. Therefore, optimisation dimension, takes the last position in the Balanced-QPM.

Figure 5. Proposed Balanced-QPM.

Additionally, Figure 5 shows global causal links between the four dimensions. The optimisation dimension has direct impact on the accessibility/quality dimension. There is also a mutual relationship between optimisation and mobilisation. Mobilisation has an impact on the accessibility/quality of services provided by the CCSMTL, which in turn has direct impact on customer satisfaction. Finally, customers put pressure on the MSSS to provide the necessary funding supporting the CCSMTL capacity in terms of optimisation, mobilisation, accessibility/quality aspects.

Figure 6 shows the strategy map of the CCSMTL. The causal links between the four dimensions shown in the figure correspond to those identified in Figure 5. For a customised strategy maps, e.g. DI-TSA-DP program, these causal links are defined more precisely (see Figure 8 for an example). The CCSMTL’s strategy map is the basis for building “customised” strategy maps for the different departments of the organisation.

Figure 6. Generic strategy map proposed for the CCSMTL.

Finally, Figure 7 shows the indicator structure system of the CCSMTL. We recall that this structure is based on the CCSMTL organisational chart and visual management system (CCSMTL’s network of control rooms). We propose to use a network of scorecards presenting the performance indexes and indicators of each dimension of the Balanced-QPM of the different departments of the CCSMTL to evaluate and discuss the performance at different hierarchical levels of the organisation. In other words, information on performance would be presented in scorecards (For an example, see Figure 10) and discussed in CCSMTL’s control rooms, and then would flow-up from operational to tactical and strategic levels (through tactical and strategic control rooms) or flow-down from strategic to tactical and operational levels, following the cascade-escalation mechanism used in the current visual management system of the CCSMTL.

Figure 7. Proposed hierarchy and network-based scorecard structure.

Balanced-QPM scoredcard at the visual station level.

DI-TSA-DP case

DI-TSA-DP program offers services to 2,200 customers and receives approximately 10,000 service requests per year, mainly in the South-Centre of Montreal region and throughout the province of Quebec. 2,150 employees and 58 physicians work in this department. It includes three main sub-programs; DI-TSA, which provides services to customers with an intellectual disability or autism spectrum disorder, DP, which provides services to customers with a physical disability, and RMVS, which provides rehabilitation services in substitute living environments such as elderly homes. DI-TSA-DP mission is to provide specific, specialised and super-specialised habilitation and rehabilitation services to its customers to promote their integration and social participation.

At the moment of conducting this study, DI-TSA-DP has put in place one tactical room (top management level of the program), and three intermediate control rooms at the sub-program level (DI-TSA, DP, and RMVS). Operational rooms are established at the team unit level within the three sub-programs. Control room deployment at the team unit level was still in progress. Therefore, each sub-program had only reached a certain deployment rate, which is seen as a measure of performance-culture maturity level in the CCSMTL. Following the recommendation of our collaborators, we focussed on DP sub-program, since it had the highest deployment rate. The approach can be applied to the two other sub-programs as well. DP includes three services: AT (Technical Aids), which provides special assistance devices (prosthesis, wheelchair, etc.), LN (Locomotor-Neurology), which provides services to customers with physical disabilities due to neurological or locomotor disorders, and SL (Sensory-Language), which provides services to customers with linguistic or sensory disabilities. We focus on LN service, again based on its operational room deployment rate (100%). There are nine care team units within LN service, and each unit has its own operational room. We worked in close collaboration with two teams, Locomotor team (A-BOG), which provides services for customers with an amputation or serious orthopaedic injuries, and Neurological team (AVC), which provides services for customers who have had a stroke.

DI-TSA-DP strategy map

The proposed strategy map for DI-TSA-DP ( Figure 8) is consistent with the generic strategy map ( Figure 6). It was co-created with the DI-TSA-DP program executive manager.

Figure 8. Proposed strategy map for DI-TSA-DP program.

It illustrates objectives determined following the QPM sub-dimensions (including the primary objective) and causal links established between these objectives and/or sub-dimensions. The objectives selected are based on the internal DI-TSA-DP strategic vision 2018-2021 provided by DI-TSA-DP executive manager. Each objective was checked against the sub-dimensions of the QPM to identify to which one it is related. In total 16 objectives were classified within the sub-dimensions and dimensions of the QPM. The DI-TSA-SP mission was chosen as the primary objective and added in the customer dimension. Since no objective exists for customer experience, a new objective was proposed based on customers’ definition of this sub-dimension ( Table 1). Additional objectives should be set for this sub-dimension and other ones that currently do not present any specific objectives (i.e., communication, learning and focus on people).

Examples of causal links in the strategy map, are the relationship between objectives within customer sub-dimensions (‘partnership’ and experience’) and the primary objective (DI-TSA-DP mission). It is clear that positive customer experience, customer involvement as a partner in the system, and providing a climate of trust for the customers contribute to achieving DI-TSA-DP mission. It is interesting to notice that causal links also exist between sub-dimensions/objectives within the same dimension. As an example, partnership objectives contribute to achieving customer experience objectives. That is, when customers are involved as partners in the system, or when a climate of trust and collaborations is established with them, they are likely to have a positive experience. Some objectives/sub-dimensions mutually impact each-other, e.g. efficiency (optimisation dimension) and healthy work environment (mobilisation dimension). Reducing indirect labour and extra working hours (efficiency sub-dimension) leads to a healthy work environment, which in turn has a positive impact on efficiency.

The strategy map was key for aligning DI-TSA-DP vision and strategy with the four dimensions/sub-dimensions of the QPM. This step helped to bring clarity and logic to the strategic objective determination process in relation to the QPM dimensions/sub-dimensions and performance measurement. Based on the strategy map, it was also possible to align LN service performance indicators with the four dimensions of the QPM through the strategic objectives and sub-dimensions selected as shown in the next paragraphs.

Indicator structure design and performance measurement

We conducted five working meetings of two hours each with different DI-TSA-DP managers. We started with the operational level (A-BOG and AVC) and moved to the tactical level (LN and DP service and sub-program, respectively), and finally, to the highest level (DI-TSA-DP program). The following paragraphs present the process of applying the developed PMMS to DI-TSA-DP case.

Step 1: Indicator system structure

Figure 9 shows the indicator system structure proposed for DI-TSA-DP. It is based on DI-TSA-DP organisational chart (and visual management system) and is part of the CCSMTL indicator system structure ( Figure 7). Figure 9 shows only the departments needed to illustrate our approach. We distinguish four levels: 1) CCSMTL strategic level, 2) tactical level, which includes DI-TSA-DP program and DI-TSA, DP, and RMVS sub-programs, and 3) operational level (within DP sub-program) that includes SL, LN, and AT teams.

Figure 9 also shows the network of control rooms deployed in these different levels. The performance indicator structure follows the organisational chart and control room network structure. They are presented in the form of scorecards following the four Balanced-QPM dimensions (indicators, values and targets, performance indexes). For an example of a scorecard, see Figure 10.

Figure 9. Simplified indicator system structure proposed for DI-TSA-DP.

Step 2: Indicator identification and selection

To apply the rest of Phase IV steps, LN service was considered, and the process was performed in collaboration with LN coordinator. First, indicators being used by LN service were classified according to the four dimensions of the QPM and the strategy map’s sub-dimensions/objectives. Next, we checked the relevance of each performance indicator (against the criteria specificity, validity, simplicity, relevance, sustainability, reliability, and utility). When an indicator does not meet all criteria, it is either changed for another indicator that satisfies the criteria, or more information is gathered to ensure that the indicator meets all criteria. As an example, indicator ‘Customer satisfaction rate’ ( ${\mathit{Xan}}_1^C$ ) ( Table 5), which was proposed to measure customer experience sub-dimension objective, did not meet simplicity criterion. LN managers did not have information on customers’ satisfaction and did not put in place specific methods to collect this information. To address this issue, we relied on the results of a survey used to collect the feedback of customers regarding their satisfaction level regarding experience and service quality. Table 5 shows the 10 final indicators selected for LN service.

Step 3: Indicator normalisation

The columns of Table 5 show the indicators measuring the objectives (and therefore sub-dimensions) selected in previous step, their notations, their values before normalisation, their targets, their worst values, and their values after normalisation (obtained by using Equation (2)).

Due to confidentiality reasons, all real values of the indicators have been modified. Note that indicators that are already expressed as percentages before normalisation and which are aimed to be maximised to reach the desired target have not been normalised. The worst value is set to zero for those indicators (e.g. Customer satisfaction rate, Bed occupancy rate, and Respect of access deadlines). Moreover, indicators before normalisation meant to be minimised to reach their target have been converted into indicators to be maximisation after normalisation (e.g. Complaint rate and Ratio of salary insurance hours).

Steps 4 and 5: Indicator weighting and aggregation

Pairwise comparisons of normalised indicators within each QPM dimension were performed with the LN coordinator. The values obtained were entered in a VBA program that checks the consistency of the matrices and generates the weights following AHP algorithm (see Equations (4) to (6)). The final results that represent the weights of the indicators are presented in Table 6.

Step 5 enables us to calculate aggregated performance indexes for each dimension that can be used by the service managers or by higher hierarchy level departments (DP sub-program and DI-TSA-DP programs in our case) to evaluate how the service performs globally in each dimension of the QPM. For instance, if the global performance of a given dimension is not satisfying, the performance indicators within that dimension should be analysed in more detail to identify the problem and take actions to improve the situation. The results of calculating the four performance indexes $({\mathit{PI}}_{\mathit{LN}}^{\mathit{C}},{\mathit{PI}}_{\mathit{LN}}^{\mathit{A}},{\mathit{PI}}_{\mathit{LN}}^{\mathit{M}},{\mathit{PI}}_{\mathit{LN}}^{\mathit{O}})$ of LN are given in Table 6. Each performance index corresponds to one dimension of the Balanced-QPM. Again, the VBA program is used to automate these calculations following AHP algorithm. These aggregated performance indexes are finally visualised on scorecard so that managers can examine each performance dimension and put actions to improve the critical ones. Figure 10 illustrates the LN scorecard with the performance indexes selected for LN for each QPM dimension and performance indicators for each dimension.

Figure 10. LN scorecard.

It is worth noting that, when using AHP, we identified a compensation effect between indicators that results from aggregating two (or more) indicators having opposite values, which is not desired. For instance, within Mobilisation dimension, the normalised indicator ‘Staffing level’ ( $X_1^M$ ) has a value of 92% while the normalised indicator ‘Employee satisfaction rates ( $X_{13}^M$ ) has a value of 70%. Their aggregation together with the normalised indicator ‘Ratio of salary insurance hours’ ( $X_2^M$ ) (value of 84.5%), leads to a performance index with a value of 82,3% ( ${\mathrm{PI}}_{\mathit{LN}}^M$ ) ( Table 6). It is easy to see that the high value of ‘Staffing level indicator’ compensates the lower value of ‘Employee satisfaction rate’ indicator. This result also depends on the weights assigned to the three indicators (0.25, 0.25, and 0.50, respectively in our case).