Development and Psychometric Validation of a Multidimensional Clinical Coding Quality Assessment Instrument in Casemix Systems

Content validity

The content validity analysis indicated that experts largely concurred on the significance and pertinence of the instrument items. The Item-level Content Validity Index (I-CVI) values presented in Table 1 ranged from 0.909 to 1.000, indicating that all items were sufficiently relevant for acceptance. The Content Validity Ratio (CVR) scores varied from 0.818 to 1.000, indicating that the expert panel deemed the majority of the items significant. Items related to specialized compliance and reselection appropriateness (I5, I9–I11) exhibited relatively lower CVR values compared to other items, however remained within acceptable limits. These variances may reflect differing expert viewpoints on complex coding decisions, particularly those requiring clinical judgment beyond simple rule-based evaluation.

The Scale-level Content Validity Index (S-CVI) was 0.972, exceeding the recommended threshold of 0.90, indicating that the content is very authentic. Overall, these findings indicate that the instrument effectively embodies the multifaceted concept of clinical coding quality inside casemix systems.

Inter-rater reliability

Inter-rater reliability analysis demonstrated varying levels of agreement across items, as presented in Table 2. Cohen’s Kappa values ranged from 0.403 to 0.802, indicating agreement levels from moderate to almost perfect.

The greatest degree of concordance was noted for primary diagnostic documentation (I1; κ = 0.802), indicating robust consistency among raters in evaluating distinctly defined and objective documentation components. Likewise, elements pertaining to coding precision and adherence (I3, I5, and I8) exhibited significant concordance, indicating that rule-based coding components often yield more uniform assessments.

Conversely, less agreement was noted in questions requiring more intricate clinical judgment, including procedure reselection (I11; κ = 0.403) and diagnostic reselection (I9–I10). The findings suggest that diversity among raters is more significant in domains necessitating interpretative decision-making than in those involving simple rule application.

The average Kappa value was 0.583, signifying moderate to high agreement among raters ( Table 3). This indicates that although the instrument offers a systematic evaluation framework, certain aspects of clinical coding quality intrinsically require subjective clinical judgment.

Internal consistency

The internal consistency analysis revealed an exceptional degree of reliability for the CCQAI–CM instrument. Table 3 demonstrates that Cronbach’s alpha coefficient was 0.982 (95% CI: 0.966–0.997), signifying an exceptionally high level of internal consistency among the items. All items demonstrated robust item–rest correlations, indicating that each item significantly contributes to the overall construct. Nonetheless, some items had significantly high inter-item correlations, suggesting possible redundancy among conceptually analogous markers. Collectively, the inter-rater reliability data indicate that the instrument exhibits robust internal consistency, although discrepancies in inter-rater agreement may signify changes in clinical judgment rather than inherent deficiencies of the instrument.

Structure of the CCQAI–CM instrument

Table 4 delineates the structural and operational framework of the CCQAI–CM (Clinical Coding Quality Assessment Instrument for Casemix). The instrument encompasses various categories that represent essential elements of clinical coding quality, such as documentation completeness, coding compliance, coding correctness, and appropriateness of reselection. Each item is assessed according to established criteria and rating categories, facilitating systematic evaluation across various clinical circumstances. The inclusion of an applicability component enables flexibility in cases where certain items are not relevant, thereby improving the usability of the instrument in real-world clinical contexts.

Scoring and interpretation of the CCQAI–CM

The structure and operational framework of the CCQAI–CM, including its domains, items, applicability status, and rating categories, are presented in Table 4. Items I3, I7, and I11 are conditional items and should only be assessed when procedures or medical interventions are relevant to the case. Positive ratings are scored as 1, whereas negative ratings are scored as 0. Conditional items marked as not applicable should be excluded from the denominator when calculating the final score. The final score is calculated as follows:

The final score is then interpreted into three categories, as shown in Table 5.

We used set % thresholds to figure out what the final result meant. A score of 85% or higher was considered “Good,” which meant that the case being looked at met at least 85% of the relevant CCQAI–CM criteria. A score of 70–84% was considered Fair, which means that 70–84% of the relevant criteria were met and that certain items or areas need to be improved. A score of less than 70% was considered Poor, which means that fewer than 70% of the relevant criteria were met. This could mean that more general remedial activities are needed, such as reviewing documentation, training coders, and improving clinical documentation.

Summary of psychometric properties

The combined results of content validity, inter-rater reliability, and internal consistency indicate that the CCQAI–CM demonstrates strong psychometric performance. The high S-CVI confirms the relevance and representativeness of the instrument items, while the moderate to substantial Kappa values indicate acceptable agreement between ratters in practical application. In addition, the high Cronbach’s alpha reflects strong internal coherence among items. Together, these findings suggest that the instrument is both conceptually valid and practically reliable for assessing clinical coding quality in casemix-based healthcare systems.

1. Content validity and conceptual strength

The findings demonstrate that the CCQAI–CM instrument possesses exceptional content validity, as indicated by a high S-CVI score (0.972), reflecting substantial consensus among experts concerning the relevance and representativeness of the questions. The concurrent application of I-CVI and CVR enhances the validation process by assessing the relevance and necessity of each item, ensuring that each item significantly contributes to the concept of clinical coding quality.^28,29

Specifically, items related to speciality conformance and reselection suitability had somewhat lower CVR values. These differences are likely to be due to differences in opinion among expert coders on complicated clinical coding judgements that need clinical judgement rather than straightforward rule-based assessment. This corresponds to²⁹ original premise that disagreement among experts is more common in fields of interpretative complexity than in those with standardised standards.

Importantly, this finding aligns with real-world coding audit practices, where disagreement among auditors often arises not from ambiguity in coding rules, but from the interpretative complexity of clinical documentation and decision-making processes. In practice, auditors need to pull together information from many sections of the medical record (e.g., discharge summaries, progress notes, test data, procedure reports) to identify the most appropriate diagnosis and assignment of codes. Variability might arise from inconsistent or inadequate documentation, or documentation that lacks sufficient clinical clarity, forcing auditors to exercise professional judgement in understanding the clinical context. Variations in training, experience, and familiarity with coding guidelines may also impact the way coders interpret and prioritise clinical data, as described within clinical documentation improvement (CDI) frameworks, resulting in discrepancies in coding outcomes despite codified rules.³⁰ This emphasises that assessing the accuracy of coding is not simply a technical quality assurance exercise, but a cognitive exercise requiring clinical judgement and contextual decision making.

2. Inter-rater reliability

Inter-rater reliability analysis demonstrated moderate to substantial concordance among items, with increased consistency noted in documentation and rule-based coding elements. The findings demonstrate that organised and standardised coding features are perceived more consistently among raters, highlighting their objective nature.^31,32

In contrast, diminished consensus on reselection-related items shows the intrinsic complexity of clinical coding judgements that necessitate contextual interpretation, including severity evaluation and resource allocation. Comparable results have been documented in research investigating discrepancies in coding techniques, with greater disagreement evident in instances necessitating interpretative clinical reasoning.^5,6 This pattern reflects findings in coding audit literature, where variability is particularly evident in areas requiring contextual interpretation, such as secondary diagnosis capture and rule-based reselection decisions.^18,33 The observed Kappa values align with known benchmarks for clinical audit investigations,²⁶ signifying that the instrument exhibits sufficient reliability for practical application.

3. Internal consistency

Excellent internal consistency is shown by the Cronbach’s alpha value of 0.982, which is indicative of good coherence among the items of the CCQAI–CM.³⁴ Very high alpha values can sometimes be a sign of item redundancy, but in the case of this instrument, the high inter-item correlations are better viewed as a reflection of the conceptual and practical interdependence among clinical coding quality characteristics. Documentation quality is a core aspect that directly impacts coding compliance, coding accuracy and diagnosis/procedure reselection. This interpretation is reinforced by clinical documentation improvement literature, which emphasises that accurate and thorough clinical recording is critical to correct coding, compliance, payment integrity, audit preparedness, and reliable health data reporting.^30,35

Incomplete or imprecise documentation might influence the implementation of ICD regulations, the choice of primary and secondary diagnoses and the validity of the coded data in casemix-based reimbursement systems. Previous research has established a close relationship between the quality of coding and the validity of administrative data and the completeness and accuracy of clinical documentation, and the aim of clinical documentation improvement initiatives is to enhance the specificity of documentation, the accuracy of coding, and data quality.^5,30,36 Thus, the high internal consistency found in this study suggests the integrated nature of the clinical coding process and not simply repetition of items.

4. Integration of psychometric findings

The combination of excellent internal consistency and moderate inter-rater reliability highlights the multidimensional and interpretative nature of clinical coding quality assessment. While Cronbach’s alpha reflects strong internal coherence among the CCQAI–CM items, Kappa values capture variability in rater interpretation. This distinction is well established in psychometric theory, where different reliability measures assess different aspects of measurement quality. ³⁷Therefore, the observed difference between internal consistency and inter-rater agreement should not be viewed as contradictory, but rather as complementary evidence regarding the performance of the instrument.

The very high internal consistency observed in this study is better interpreted as reflecting the conceptual and operational interdependence among clinical coding quality dimensions rather than simple item redundancy. In clinical coding practice, documentation completeness, clinical support, coding compliance, coding accuracy, and diagnosis/procedure reselection are not isolated processes. However, they are part of a workflow, where the quality of one element immediately impacts the next. Incomplete or inconsistent clinical documentation may hinder the coder’s capacity to appropriately apply ICD guidelines, identify the correct primary and secondary diagnoses, and confirm that coded data accurately reflect the patient’s clinical status. This interpretation is consistent with clinical documentation improvement literature stressing the necessity of precise, thorough and specific documentation for coding accuracy, compliance, reimbursement integrity, audit readiness and accurate health data reporting.^5,30

Kappa values were moderate, indicating that reviewers need to consistently assess and interpret multiple items clinically. Discrepancies in agreement are expected, particularly for items involving clinical consistency, diagnosis reselection and relevance of secondary diagnosis and procedure reselection since these require the coders to assess clinical evidence, follow coding guidelines and prioritise diagnostic or procedural importance in the casemix context. The variation in kappa values observed is likely to reflect variation in clinician judgement and coder interpretation rather than limitations of the instrument itself. This interpretation is consistent with health information and data quality frameworks that suggest data accuracy is determined by more than structural consistency, standardised formats, coding systems or set regulations. Their reliability is a function of the consistency with which the data definitions are interpreted and applied in practice by users. Indeed, even with standardised data items, variability may occur due to differences in clinical documentation quality, coder training, experience and knowledge of coding standards. This is particularly true for casemix-based reimbursement systems, as coding decisions affect the quality of the administrative data, outcome grouping and financial implications.^35,36,38

The psychometric results indicate that the CCQAI–CM exhibits robust internal consistency and effectively addresses aspects of clinical coding that necessitate interpretative judgement. The elevated Cronbach’s alpha indicates strong conceptual integration of the instrument, while the Kappa results underscore the necessity for coder calibration, systematic training, and enhancements in clinical documentation to promote consistency in instrument use among raters and contexts.

5. Domain-Based interpretation of findings

A domain level analysis gives more insight into the performance of the instrument. Unsurprisingly, documentation and coding compliance domains had higher levels of agreement, as they rely on structured and standardised criteria. These areas correspond with the processes involved in coding audits which verify completeness of documentation and adherence to coding standards.^11,30 This is consistent with contemporary coding audit frameworks, which emphasize that coding quality extends beyond simple code matching to include documentation validity and rule application.^8,33

In contrast, reselection domains showed lower agreement, pointing to their reliance on clinical judgement and context-specific interpretation. In such domains, the evaluator needs to combine multiple factors, such as severity of diagnosis, use of resources, and clinical relevance, which naturally leads to variability.^39,40 Critically this confirms that clinical coding quality is not simply technical but a hybrid construct that combines procedural accuracy and clinical reasoning. Finally, the addition of an applicability component adds flexibility when some items are not relevant, thus enhancing the usability of the instrument across different clinical settings.

6. Novelty and contribution

The CCQAI–CM is a major contribution, providing a multidimensional, rule-based and context-sensitive framework for assessing clinical coding quality in casemix systems. This instrument, unlike traditional methods that focus on coding correctness or inter-coder agreement, contains documentation completeness, coding compliance, coding accuracy and reselection logic based on casemix principles. Previous research predominantly considers accuracy measurements or agreement statistics as critical indicators of the quality of codes, often overlooking upstream issues such as the sufficiency of documentation and adherence to coding standards which have a significant impact on the coding results.^5,31,32

From a data quality perspective, the instrument is consistent with and builds upon multidimensional data quality frameworks that emphasise “fitness for use” as a central concept.^36,38 Here, data must not only be syntactically correct but also semantically consistent, clinically valid, and consistent with decision rules that govern classification and payment. Moreover, the inclusion of documentation conformity as a core assessment domain further strengthens the link between clinical documentation and coding quality, bolstering the principle that the validity of administrative data is based on the consistency between coded data and verifiable clinical evidence.^35,41,42

From a governance perspective, the CCQAI–CM assists in conceptualising coding quality as a health information governance mechanism, rather than a mere technical outcome. Information governance systems focus on accountability, standardisation, transparency and auditability at all stages of the health information lifecycle.^7,35,43 The development of the instrument according to rules and standards is in accordance with these principles, allowing for systematic monitoring of the instrument, traceability of coding decisions and systematic implementation of coding standards and national casemix rules. The instrument also contributes to reimbursement integrity theory by directly linking coding quality assessment with casemix grouping logic and principles of resource utilisation. Casemix systems are built on the assumption that diagnostic and procedural codes are proxies for patient complexity and intensity of care.⁴⁰ The CCQAI–CM operationalises reimbursement integrity at the micro-level of coding decisions. It conceptualises the link between coding practices and payment system sustainability through assessment of the appropriateness of diagnosis and procedure reselection.

In practical terms, the instrument provides healthcare organisations with a structured way to systematically evaluate coding performance and identify underlying weaknesses in documentation and coding workflows. It reveals patterns of non-compliance, documentation gaps and misapplication of rules and facilitates targeted interventions such as coder retraining, clinician documentation improvement and audit-driven corrective actions. These applications align with modern coding audit approaches that aim to improve revenue integrity and compliance readiness.⁴⁴

These contributions together position the CCQAI-CM not just as an audit tool but as a complete conceptual model which links rule-based coding systems to clinical reasoning. This integrated framework provides a more realistic and operationalisable approach to measuring the quality of clinical coding in casemix-based healthcare systems and also establishes the foundation for future research into coding governance, audit effectiveness and quality of administrative data.