Introducing the Multidimensional Toolkit for the Assessment of Play in Schools (M-TAPS): a reliability study

2.1 Materials: Multidimensional Toolkit for the Assessment of Play in Schools (M-TAPS)

The M-TAPS was created by the authors and is available at: https://osf.io/qjf8b/. It aims to provide insight into children’s play by combining across child and observer perspectives. Given this, the M-TAPS includes:

These three components have been designed to be complementary; they can be combined but each could be used in isolation, depending on specific research questions. Each component will now be described in turn.

2.1.1 Scan observations

For the scan observations, playgrounds are divided into defined areas which allow the researchers to be stationary and view the entire area without any significant obstructions. The number of areas required will vary by school but each area should be scanned at least twice. We recommend that observers visit the school to select the areas and pilot data collection with those areas ahead of starting the scan observations. A complete round of scan observations is then conducted for each area before moving to the next area. During each round of scan observations, observers count the number of children doing and not doing each of the following types of play (see extended data for full definitions for each code – available at: https://osf.io/qjf8b/):

These categories of play were initially developed based on those included in the OPP (Massey, Ku, et al., 2018a) and aligned with the activities on the Activities During Playtime measure (see 2.1.3). However, following discussion with adults familiar with UK playgrounds and piloting, some edits were made: we added ‘markings’ to the equipment category, added ‘landscape’ to the nature category and added two additional categories of ‘sport/play equipment’, ‘loose parts/recycled materials’. In addition, we removed ‘traditional playground games’ because during piloting it became clear that defining this was challenging given different childhood experiences of coders.

Alongside these play categories we also explored the extent to which adventure level and affect could be coded during playground scans. This was motivated by the growing interest in adventurous play and a desire to capture children’s emotional experience if possible. Thus, the following were also counted during scan observations (see extended data for full definitions for each code):

Observers scan from left to right imagining a straight line moving across the space. Each child is counted as the imaginary line reaches them and, for each of the play categories above, is classified as doing (yes) or not doing (no), each activity. For example, an initial scan is completed from left to right with all children categorised as playing/not playing. Then a new scan begins left to right counting the number of children playing sport and so on until all categories have been coded. For all categories children are counted as doing the activity/adventure level (yes) or not doing it (no) with the exception of affect which is coded as positive, negative or unclear. Once scans have been completed for all categories, that round is complete and observers move to the next area and repeat.

2.1.2 Focal observations

Specific children are selected for focal observations. These children can be selected at random or according to demographic or other characteristics depending on the purpose of the study. Observers first locate a focal child on the playground and then begin the observation. The observer observes the child for 10 seconds and then records:

Observers are given 15 seconds to record this information and then the next 10 second observation period begins. For this study, observers completed 15 × 10 second observations per focal child before finding the next focal child and beginning their observation.

2.1.3 Child-report questionnaire pack

The child report questionnaire pack includes two questionnaires.

Emotions during playtime. An adapted version of the Positive and Negative Affect Scale for Children (PANAS-C; Laurent et al., 1999) was created to capture the emotions that children experience during school playtimes. The PANAS-C asks children to respond using a 5-point Likert scale (from ‘Not much or not at all’ to ‘A lot’) the extent to which they have experienced each of 30 emotions over the previous 2 weeks. To create the M-TAPS Emotions During Playtime measure we adapted the PANAS-C. The emotions and response scale are identical to the original measure but children are asked to respond to each item thinking about how they have felt during playtime over the past week. The PANAS-C was selected as the basis for the measure because of the broad range of positive and negative emotions included and because it is a well-established measure with strong psychometric properties (Laurent et al., 1999). Note that although the PANAS-C originally included 30 items, following Laurent and colleagues, only 27 items contribute to the positive and negative affect scales with alert, fearless and daring excluded from the scale scoring. We included all 30 items to support future research which could explore emotional profiles of playtimes and also to consider whether individual items might give useful insights. The PANAS-C also exists as a 10-item measure (Ebesutani et al., 2012) so a 10-item version of the Emotions During Playtime measure could be used if only an approximate measure of overall positive and negative affect during playtime is needed. Given that children’s report of their emotions may be affected by their state emotion, the Emotions During Playtime questionnaire is designed to be completed on two occasions at least 2 days apart, with responses averaged across the two completions.

Activities during playtime. A second questionnaire was developed to capture what activities children say they do during playtimes. Children were asked to think about school break and lunchtimes over the past two weeks and for each of eight activities state how much they do the activity. Activities were: playing sports; playing on fixed play equipment or markings; playing with loose objects; playing with nature; playing chase games or running around; play fighting or wrestling; fighting, arguing or trying to break things; not playing (examples were provided). Children responded on a three-point scale: ‘Not at all’, ‘A little’, ‘A lot’. These categories of play were developed based on those included in the Observation of Playground Play (OPP; Massey, Ku, et al., 2018a). At the end of the questionnaire, children were asked to write any other types of play and their favourite activity.

2.2. Methods: Reliability study

2.2.1 Participants

To evaluate the reliability of each part of the M-TAPS, we recruited participants from a primary school located in Bristol, UK. The school was located in a relatively low-income urban area where residents are predominantly white; the school population and study participants reflect this demographic. Data were collected in March 2022.

For child self-report, all children from one class in each of years 3, 4 and 5 respectively (aged 7-10 years) were invited to participate. We chose to include only children aged 7 years and above given the reading and cognitive capacity required to respond to the questionnaires. In total 62 children started the questionnaire pack.

Scan observations were conducted across the entire playground meaning that all children from years 1 to 6 (aged 5 to 11 years) were included in these observations. In the UK, children attend primary school from age 4 to age 11 (or a combination of infant and junior school). Age 4-5 is termed Reception. These children follow a different curriculum to the older children and their playgrounds are often separate from the rest of the school. Given this, we chose to focus on children from Year 1 upwards but we have no reason to believe that the M-TAPS could not be used with younger children.

Focal observations were conducted with 44 children (22 male, 22 female) selected at random. We did not collect any identifying information about these children but, because children at this school have playtimes staggered by age, we know that a range of ages were included. Of the focal children, 13 were in Year 1 (aged 5 or 6 years), 17 in Years 2, 3 or 4 (aged 6 to 9 years), and 14 in Years 5 or 6 (aged 9 to 11 years).

For an inter-rater ICC of .8 (against a null of.5), a sample of 37 gives 99% power to detect a significant association between two raters and 95% power to detect a significant association across two points, therefore the sample size was adequate to address the primary aims related to assessing reliability (Walter et al., 1998).

Following consultation with the appropriate ethical review board, we chose to use an opt-out consent procedure. The school provided proxy informed consent for all participants and parents were given detailed information about the study with options to withdraw their child. No parent chose to remove their child from the study. The benefit of this approach is that a representative sample can be obtained and typical playtimes, with all children on the playground, could be observed. A limitation of this approach is that, to minimise any risks associated with opt-out consent, all data were collected anonymously. This means that no personal or demographic information was collected about individual participants.

2.2.2 Procedure

The methods and study procedures were approved by the University of Reading Research Ethics Committee (Ref: 2021-163-RM) on 15th November 2021. The study adhered to the Declaration of Helsinki principles. Parents were provided with written information about the research and asked to inform the school or research team if they did not want their child to complete questionnaires and/or be on the playground during the observations. A range of response options were given including emailing, phoning or texting the school or research team. Children were given information about the study via a short, pre-recorded presentation and asked to provide assent before completing questionnaires. They were also given opportunity to ask their teacher and/or the researchers questions. Children were given paper copies of the questionnaire packs to complete during class time under the supervision of their class teacher and learning support assistant. The pre-recorded presentation told children that the questionnaires should be completed alone and that they should not discuss the answers with other children. Teachers were also given written information from the research team asking them to create test conditions for the completion of the questionnaires. Time 1 data collection consisted of two sessions (completed an average of 2.30 days apart (range 0-12 days)). During the first, all questionnaires were completed once. In the second session just the Emotions During Playtime measure was completed. At Time 2 (which began an average of 5.51 days after Time 1 (range 0-10 days), the same procedure was followed (the two sessions at Time 2 were completed an average of 5.16 days apart (range 2-12 days)).

For the scan observations, the research team viewed the playground and divided it into five defined areas during a pre-visit. Data collection was then piloted to ensure that these areas were appropriate and to confirm standing positions for the coders. This process also gave children time to get used to the observers being on the playground and reduced the likelihood that the presence of the observers would affect children’s play. In total 15 area scans were completed by both observers with each area scanned at least twice (area 1 = 6 scans; area 2 = 3 scans; area 3 = 2 scans; area 4 = 2 scans; area 5 = 2 scans). Note that the Year 1 children were only allowed to use area 1 for their playtime so this area was scanned more frequently. Observers synced the start time of each scan to ensure they were observing the same play, for the purpose of assessing reliability.

For the focal observations, the coders selected a child to observe at random and coordinated the start of the observation to ensure they were both observing the child at the same time. Where specific children need to be observed, we have piloted using coloured wrist-worn sweatbands to identify those children and this approach has worked well. Children wear the sweatbands on the outside of any sleeves or coats so that they are visible and each child is assigned a colour during the coding period.

After coordinating the start of scan and focal observations, coders did not consult with one another about what codes to allocate to ensure that the reliability assessments were accurate and to avoid observers biasing one another. Scan and focal observations were supported by an app developed for the purpose of this research. The app was programmed in Microsoft PowerApps and can store data locally on a tablet/phone that is later transferred to secure databases within the Microsoft SharePoint environment. The materials required to create a usable instance of the app, including instruction videos and data templates, are available here: https://osf.io/qjf8b/.

2.2.3 Missing data

The questionnaire booklet was started by 62 children and the questionnaires were completed by the following numbers of children: Emotions During Playtime Positive Affect scale (T1 = 53; T2 = 53; T3 = 49; T4 =43); Emotions During Playtime Negative Affect scale (T1 = 54; T2 = 53; T3 = 48; T4 = 43); Activities During Playtime questionnaire (responses varied from 53 responses to 56 responses for each activity at T1 and from 47 responses to 50 responses at T2). For the focal observations, of 660 possible individual observation data points (15 per child x 44 children) a minority were missing, primarily due to technical errors or interruptions during coding: 11 for activity, 17 for adventure, and 20 for affect. Analyses are conducted with available data.

3.1 Scan observations

3.1.1 Activity

A total of 15 scans were conducted by each rater. During the scans each visible child in the area being coded was coded as engaging (or not) in each activity in turn. Table 1 shows each activity and the minimum, maximum and mean percentage of participants coded as engaging in each activity. The minimum and maximum values relate to single scans (i.e. 0% indicates that during at least one scan no child was observed engaging in that activity), with the mean values showing the mean percentage across all of the scans. The majority of children observed were coded as engaged in play (mean = 86.2%), with the most popular activities being playing with sports or play equipment followed by active play or chase. ICCs ranged from 0.73 to 1.

3.1.2 Adventure

During the scan observations, each child’s adventure level was also coded. As shown in Table 1 only around one-quarter of children were observed to be playing with at least moderate adventure during scan observations. No children were coded by either observer to be playing with high adventure during scans. ICC for proportion of children playing with at least moderate adventure was 0.80 (see Table 1).

3.1.3 Affect

During the scan observation, each child’s affect was also coded. For the majority of children affect was coded as unclear with only a very small minority of children expressing any clear negative affect. The ICC values ranged from 0.73 to 0.76 (see Table 1).

3.2 Focal observations

Focal observations were made by each rater independently. Each rater coded the activity, adventure level and affect they observed during each observation period. For information, the proportions of total observations coded for each activity, adventure level and affect coded across all participants are shown in Table 2, split by rater. The minimum and maximum values relate to single participants (i.e. 0% indicates that at least one child did not engage in that activity at all during their focal observation, and 100% indicates that at least one child engaged in that activity throughout their focal observation), with the mean values showing the mean percentage across all of participants. Table 3 also shows the proportion of adventurous play codes that were assigned to each type of risk. Note that risk categories were only available when raters coded adventure as moderate or high, so these proportions are from a total of 45 observations for Rater 1 and 48 observations for Rater 2.

Raters gave the same activity code on 96% of observations, Kappa = 0.95. For adventure level, raters agreed on 98.4% of observations, Kappa = 0.89. For affect, raters agreed on 96.4% of observations, Kappa = 0.88. For risk categories, only Height, Impact, Speed, Routh and Tumble and other were observed; raters agreed on 97.6% of the 41 observations where both coders categorised play as moderate or high adventure, Kappa = 0.96. The ICCs for all focal observation categories were excellent, ranging from 0.9-1 (see Table 2).

3.3 Questionnaires

Section 2.2.3 shows the number of children who completed each questionnaire.

3.3.1. Emotions during playtime

The Emotions During Playtime scales were completed twice at Time 1 and twice at Time 2 as detailed in 2.1.3 and 2.2.2. Table 4 displays the mean and standard deviation for the positive affect score and negative affect score for the Emotions During Playtime measure, at each completion point. Scores based on the 27-item version and the 10-item version (to align with the different versions of the PANAS-C) are included for comparison purposes. The values show that positive affect was slightly higher than negative affect across all completion points and that scores were slightly lower for both positive and negative affect when only 10 items were used.

Our a priori intention was to average across the two completions at each timepoint to give an indication of emotions experienced during playtime. The test-retest reliability when this approach was used was good (CCC = 0.87 for positive affect and CCC = 0.81 for negative affect with 27 items; CCC = 0.82 for positive affect and CCC = 0.81 for negative affect with 10 items). To explore whether this approach was required or whether the consistency between a single completion of the Emotions During Playtime measure at Time 1 and Time 2 would be adequate the CCCs for the first Emotions During Playtime measure completed at T1 and the first completed at T2 were evaluated. These were weaker (CCC = 0.7 for positive affect and CCC = 0.55 for negative affect for 27 items; CCC = 0.72 and CCC = 0.56 for 10 items). This suggests that completing the Emotions During Playtime measure on two occasions gives a more reliable estimate of both positive and negative affect.

Cronbach’s alpha was calculated for the positive affect and negative affect scales independently at each completion point, with complete data only and with all data. For positive affect, alpha values ranged from 0.80 – 0.90 and, for negative affect, alpha values ranged from 0.87-0.92 (for the 10-item scale, values ranged from 0.73 – 0.83 for positive affect and 0.73 – 0.85 for negative affect).

As described previously, we included all 30 items to support future research which could explore emotional profiles of playtimes and also to consider whether individual items might give useful insights. We therefore also evaluated test-retest reliability for each item (see Table 6). The majority of the items showed reasonable consistency over time and sufficient variation that they could be used in isolation in future research if needed. Notable exceptions were ‘happy’, ‘disgusted’, ‘miserable’, ‘strong’ and ‘gloomy’ which had poor consistency.

3.3.2 Activities during playtime

Table 5 shows the proportion of children who responded that they played each activity ‘Not at all’, ‘A little’ or ‘A lot’ during playtime. At both timepoints the most commonly reported activities were sport and chase/tag. Test-retest reliability regarding specific activities was relatively poor; Kappas ranged from 0.13 – 0.41 for the consistency of responses. We also examined whether reliability would be stronger if we collapsed ‘A little’ and ‘A lot’ responses to give a binary variable but reliability was only slightly improved with Kappas from 0.19 to 0.54 (see Table 5).