Te Waka Kuaka, Rasch analysis of a cultural assessment tool in traumatic brain injury in Māori

Study procedures and data collection

A 46 item draft scale was developed from verbatim quotes taken from transcripts of an earlier phase of the study (see Supplementary File 1), and refined using a culturally responsive method (Elder & Kersten, 2015). Rangahau Kaupapa Maori (Māori research approaches determined and conducted by Māori, with the goal of supporting Māori health advancement) were utilized.

The statements used in the first iteration of the tool came from Māori participants in marae wānanga (traditional learning fora). The items were then refined via four focus groups, with the final group of participants having experienced TBI. This was to ensure the items were acceptable to those with direct experience, and that the items had face validity in addressing the sub-scale areas and were easily understood. The measure was then completed by 319 participants from a range of settings in the North Island of Aotearoa, New Zealand, between June and November 2015. They included attendees at Kura Reo; a week-long total immersion Te Reo Māori wānanga (Māori language learning environment). The attendees had a range of proficiencies in speaking Te Reo Māori, from beginner to expert level.

People were invited to participate in two ways. First, via Māori health service providers, appointments were set up with the first author. Second, wānanga groups were offered participation and the first author provided a presentation about the project, answered questions and provided oversight of completion of the tool. Inclusion criteria were Māori with TBI, or non-Māori who were part of Māori whānau (extended families), for example by marriage, whānau members, friends of Māori with TBI, those with work connections with Māori with TBI and Māori community members concerned about TBI. TBI was defined by self-reporting, as either confirmed, possible or unknown. Information was collected about TBI severity and placed into mild, moderate, severe and unknown categories, however given the questionable accuracy of self-reporting, this data was not included in our analysis. The emphasis here was on offering participation to whānau as well as to individuals affected by TBI. This reflects the centrality of whānau as a health and wellbeing construct, which is well recognised in Māori scholarship (Durie, 2001) and tikanga (cultural lore) (Moko-Mead, 2003). Indeed, the theoretical basis of this tool proposes that TBI affects the whole whānau and that the whole whānau needs to considered as “the patient” (Elder, 2013a). All 319 participants provided written informed consent. The research was approved by the Health and Disability Ethics Committee of NZ (14/CEN/17) and by Te Whare Wānanga o Awanuiārangi, the first author’s institution (EC14 034HE). Participants were supervised by the first author or a research assistant, when completing the draft 46-item outcome measure. These data were then entered into the Rasch analysis software programme, RUMM2030 (Andrich et al., 2010).

The instrument resulting from the earlier research (Elder, 2013a) contained four subscales and 46 items. The four subscales were labeled Wā (time), Wāhi (place), Tangata (people) and Wairua practices (Wairua is defined here as an aspect of health and well-being characterized as a unique connection between Māori people and all aspects of the universe). The participants were invited to score each of the items as strongly agree, agree, disagree or strongly disagree. While debate continues around whether or not to include a neutral response option in surveys or assessment tools, the rationale used here aligns with others who have shown absence of a neutral option encourages mental effort to engage with the item and negates the effect of social desirability bias (Krosnick et al., 2002). Other demographic information was collected about each participant as presented in Table 1.

Data analysis

All analyses of each of the subscales were carried out using RUMM2030 (Andrich et al., 2010) in order to determine the fit of the data to the Rasch model. Rasch analysis is a probabilistic mathematical model that draws on item response theory with the advantage of estimating the item difficulty and the person ability separately, which is not possible using measures based on classical test theory (Hays et al., 2000). The 1-parameter logistic function enables item difficulty to vary but assumes all items discriminate equally. Before Rasch analysis is used to transform ordinal observation data into linear measures, the Rasch fit statistics are examined to enable assessment of any threats to linear measurement (Haigh et al., 2001; Whiteneck et al., 2011).

Rasch analysis is used to assess the measurement properties of existing measures and to guide the development of new ones (Czuba et al., 2016). The Rasch model states that the outcome of an encounter between a person and an item is governed by the product of the construct of interest of the person, together with the easiness of the item (Bond & Fox, 2001). The person’s estimate of cultural needs is derived by dividing the percentage of items that scored highly, by the percentage of items scored in the low range, and then by taking the natural log.

Scalable items are important because they capture difficulty, and make the measurement useful in a practical sense. For instance, an item with high difficulty means it more urgently needs to be acted upon, and fluctuations can be monitored. Likewise, items which capture low, and intermediate levels of need are important in a measure, so that both lower and intermediate levels of need can be identified, and changes over time can be monitored and responded to. In the Rasch model, the item difficulty is estimated by calculating the odds of success in identifying those who scored highly and those who scored in the low range.

Each item within the scale has its own level of difficulty on the trait (item parameter), and every person has his or her own level of “ability/trait”. Item parameters are estimated independently from the person parameters and once they are identified they can be placed along the same interval scaled ruler. The item and person performance probabilities determine the interval sizes on the “ruler” of the measure.

A number of tests were performed to assess the fit of the subscales to the Rasch model. Fit to the assumptions of the model can have a number of contributing factors which are explained in detail elsewhere (Andrich, 1988; Bond & Fox, 2001; Kersten & Kayes, 2011; Tennant & Conaghan, 2007). It is important to note that ‘misfit’ should not be taken to mean that the item has no merit or is of no interest, but rather that it does not fit the unidimensional structure of a measure (or in this case domain). If this is the case, collapsing scores or moving an item to a different domain is considered, for items that do not fit but add discriminatory information. Table 2 presents a brief overview of the central Rasch analytical concepts and the actions that can be taken in the case of conditions not being met for the transfer from ordinal to linear scores.

For Rasch analyses, reasonably well-targeted samples of 150 are reported to have 99% confidence that the estimated item difficulty is within ± ½ logit, and n=243 for poorly targeted samples (Linacre, 1994). Our sample of 319 was therefore optimal for the purpose of this analysis.

Wā (time)

The proposed subscale had 9 initial items all concerned with the broad concept of time. These items were not specifically linked to issues such as time to access treatment or time since injury. Rather, time in this subscale is concerned with what needs to happen first in time, the role of time in facilitating healing, taking time for a range of purposes and flexibility of time schedules.

The initial analysis of the Wā subscale showed that there were no items with statistically significant disordered thresholds, and the scale was unidimensional. However, the scale did not fit the Rasch model because of a significant (p=0.0005) item-trait interaction chi-square, and a particularly high mean persons location (2.8; SD=1.5). 23% (n=60) of the sample had extreme scores, and so were deleted from the analysis, the remainder of n=259 provided a robust sample to analyse. Deletion of the subgroup improved the mean persons location (2.3; SD=1.2), but did not result in an improvement in item-trait interaction.

Further examination of the items revealed three (items 3, 5 and 9) that were misfitting the model. Item 3, “whakawhanaungatanga (the process of making connections with others) at the beginning sets the scene for the journey” functioned differently according to iwi (tribe), with the “other” group being an outlier. Also, the item did not fit the Rasch model, with item fit residual of -2.825, and chi-square probability of 0.006. Importantly, the item seemed to identify issues already captured by items 1 (Starting the process of wairua healing is the first thing that needs to happen for our whānau), 2 (The journey of wairua healing is enhanced with time), and 8 (whakawhanaungatanga time builds, to keep hope and dreams alive). Hence, it was deleted from this subscale.

Item 5, “It is important that kaimahi (health workers) are flexible in their schedules of work”, had a high fit residual (2.722; p=0.0006) indicating the item does not fit the scale. It was also deleted from the subscale.

Lastly, Item 9, “Whānau unity and strength builds healing” showed local dependency problems with item 3 “whakawhanaungatanga at the beginning sets the scene for healing”. It also displayed non-uniform differential item functioning (DIF) for relationship (see Table 1). A number of possible solutions described in Table 3 were tested, however, only deletion of the item led to solving the local dependence with item 3.

These modifications improved the fit of the Wā subscale and provided the final solution (see Table 3). The resulting 6-item scale was unidimensional and the item-trait interaction was non-significant (p= 0.1237). The reliability of the subscale is relatively low (PSI=0.56). The targeting of the subscale Wā was skewed, suggesting people on average scored towards the upper end of the scale (Figure 1)

Figure 1. Person-Item Threshold Distribution for the Wā (time) subscale.

Wāhi (place)

The proposed Wāhi subscale included 10 items concerned with aspects to do with places, such as those of cultural significance as well as clinics and hospitals.

Upon initial examination of the Wāhi subscale, it was found that the item-trait interaction chi-square was highly significant (p<0.00001) and the scale was not unidimensional. None of the items showed disordered response category thresholds. Further analysis of DIF and fit statistics revealed four items that required specific attention: items 10, 11, 16 and 17.

Items 10, “The use of pepeha within treatment would support the healing”, and item 17, “Whānau from home are an essential link with home”, had uniform DIF by TBI severity. These items were combined into a testlet with item 13, “Whakaairo (carvings) teach important lessons that help with healing”, which had showed non-significant DIF in an opposite direction. This resulted in these opposing directional DIF cancelling each other out.

Item 11, “Being inside buildings like hospitals does not help me”, had a very high fit residual of 7.785 (p<0.00001), demonstrating it did not fit the subscale. This item was therefore removed.

Item 16, “Gathering, preparing and eating food from home is an important part of healing”, showed uniform DIF by location and TBI. This item was combined into a testlet with item 19 as this item visually showed to have DIF in the opposite direction (non-significant), “being on the marae is a good place to start to feel strong again”, and therefore these DIF in opposite directions cancelled each other out.

These modifications improved the fit of the subscales to the Rasch model and provided the final solution (Table 3). The final subscale had 9 items and was unidimensional, the item-trait interaction was non-significant, and no DIF was observed. The reliability of subscale Wāhi was good (PSI=0.78) and the targeting acceptable (Figure 2).

Figure 2. Person-Item Threshold Distribution for the Wāhi (place) subscale.

Tangata (people)

This subscale is concerned with people involved with the person with TBI and their whānau and had a total of 15 statements.

The initial analysis of the Tangata subscale showed that the scale was unidimensional and none of the items had statistically significant disordering of response categories thresholds. However, the scale did not fit the Rasch model with statistically significant item-trait interaction chi-square (p=0.0002).

Further examination revealed three pairs of items with high residual correlations. Item 22, “Within whānau there are a lot of resources”, was locally dependent (residual correlation = 0.25) on item 23, “within the whānau is the rongoā” (rongoā is the Māori word for medicine). From a theoretical point of view, these two items consider two very similar concepts. However, item 23 is focused more specifically on the healing process, whereas item 22 (Within whānau there are a lot of resources) is much less specific as to what sort of resources might be available, when and for what purpose. Furthermore, item 23 showed a better spread on the latent trait of interest (3.8 versus 2.8 logits). Therefore, item 22 was deleted from this subscale.

Item 26, “Māori have a different point of view from Pākehā (non-Māori of European ancestry)”, was locally dependent upon item 27 (0.405) “Māori cultural needs are different from Pākehā”. Theoretically, cultural needs secondary to the culturally determined injury to wairua are critical to the functioning of this tool, in order to best understand how whānau conceptualise these needs. While asking about similar issues, item 27 more specifically asks about cultural needs, whereas item 26 was more vague, referring only to a different point of view. Hence, the decision was made to delete item 26.

Item 28, “When health workers relate to the culture of the whānau outcomes are improved”, was locally dependent (residual correlation = 0.444) on item 29, “When health workers support whānau to address wairua outcomes are improved”. Item 29, was deemed to be theoretically more important, because it more directly measures the issue of wairua, which is central to the theory of the cultural aspect of injury. Therefore, item 28 was deleted from the subscale.

Deletion of these three items improved fit of data to the model and provided the final solution (Table 3). The item-trait interaction chi-square was non-significant, the scale was unidimensional and no DIF was observed. The reliability of the subscale Tangata was good (PSI=0.740) and the targeting was acceptable (Figure 3).

Figure 3. Person-Item Threshold Distribution for the Tangata (people) subscale.

Wairua practices

Wairua practices is a phrase used to describe activities that strengthen wairua. Wairua is an area of hauora (health and wellbeing) that conveys the unique connection between Māori and all aspects of the universe. While wairua is mentioned in other subscales, wairua is the primary focus of this subscale. This subscale consisted of 12 items.

The initial analysis of the Wairua subscale found that the scale was unidimensional, but it did not fit the Rasch model (p<0.0001). Moreover, there was one misfitting item, one item showed non-uniform DIF, two items were locally dependent and a number of items had statistically significantly disordered response category thresholds.

Item 35, “Practices that strengthen wairua are as important as clinical interventions”, was found to be misfitting with chi-square p=0.00014. The item was deleted and fit to the model improved.

Examination of item 46, “Use of Te Reo Māori means wairua is being strengthened”, identified non-uniform DIF by location and statistically significant disordering of response category thresholds. The decision was made to delete this item and this improved fit to the model.

Items 43, “Romiromi (type of massage) can be a powerful healing tool”, and 42, “Mirimiri (type of massage) can be a powerful healing tool”, were found to be locally dependent (residual correlation = 0.638). Because these types of massage are very similar and mirimiri (massage) is more commonly known, item 42 was retained and item 43 was deleted.

Five items, 36, 38, 39, 44 and 45 showed statistically significant disordered thresholds. The lower two response categories (“strongly disagree” and “disagree”) of these items were collapsed into one category. This modification further improved fit of data to the model and provided the final solution for the Wairua subscale. The scale fit the model with non-significant item-trait interaction and was unidimensional. The reliability of the scale was good (PSI=0.733) and the targeting was acceptable (Figure 4). Scoring was modified accordingly.

Figure 4. Person-Item Threshold Distribution for the Wairua practices (activities that strengthen spiritual connection) subscale.

Item difficulty

Table 4 presents the relative difficulty of each item of the Te Waka Kuaka subscales. The easier the item, the higher the expected scores are for people with high levels of investigated construct.

Table 4. Rasch estimates of true item difficulty.

Item Difficulty	Wā		Wāhi		Tangata		Wairua
Less	Item	Location (logits)	Item	Location (logits)	Item	Location (logits)	Item	Location (logits)
	Q8	-0.753	Q18	-0.583	Q34	-0.56	Q38	-0.882
	Q2	-0.165	Q15	-0.394	Q27	-0.28	Q36	-0.468
	Q7	0.03	*Q16&19	-0.312	Q30	-0.271	Q40	-0.292
	Q6	0.137	*Q10&13&17	0.265	Q31	-0.15	Q42	-0.246
	Q4	0.277	Q12	0.4	Q33	-0.005	Q39	0.469
	Q1	0.474	Q14	0.623	Q29	0.015	Q44	0.549
					Q23	0.06	Q41	0.559
					Q21	0.067	Q45	0.581
					Q32	0.233
					Q24	0.35
More					Q25	0.541

Key:

*: Indicates a testlet made of two or more original NFI items. Testlet is scored by summing up the scores from included items.