Development, validation and use of artificial-intelligence-related technologies to assess basic motor skills in children: a scoping review

Introduction

Developing basic motor skills (BMS) is a fundamental process for children.¹ These skills involve a series of body movements such as walking, jumping and running, throwing and catching objects, and the sense of balance.² Several studies have shown the social and well-being benefits of healthy BMS development. For example, children with BMS stimulation tend to participate more in physical activities (e.g., scholarly games and sports), suggesting other benefits such as the early prevention of obesity.³ Several intervention programs, as well as early interventions for promoting healthy BMS development, have been designed, applied and recommended.⁴ To evaluate the efficacy of these interventions and monitor the optimal BMS development in children, valid and reliable measurement tools are needed.

Several instruments are frequently used to assess BMS. For example, the Test for Gross Motor Development Second Edition (TGMD-2),⁵ the Peabody Motor Development Scale-2 (PDMS-2),⁶ the Bruininks-Oseretsky Test of Motor Proficiency-2 (BOT-2),⁷ and the Movement Assessment Battery for Children-2 (MABC-2).⁸ These assessment instruments have been translated and adapted for people in different countries, such as the USA, China and Iran.⁹ For all these instruments, BMS assessment is expected to be performed by trained professionals who observe, describe and measure children’s responses to physical tasks.^10,11 However, differences between observers (e.g., small differences when marking each task, even after being trained) can introduce noise in the BMS measurements, making the evaluation less accurate and leading to wrong conclusions. For example, two children with similar BMS evaluated with the same instrument but by two different observers can be marked with different BMS levels in the same task, which is an undesirable error. When it becomes systematic (i.e., a constant deviation from the correct BMS measurement), this error is known as observer bias and has been largely investigated.^12,13 The BMS evaluation based on artificial intelligence (AI) is a good alternative to avoid observer bias.¹⁴

AI-related technology for the recognition and classification of human motion patterns involves several components and steps.¹⁵ We describe these general steps in Figure 1, providing some details in the next lines. Sensor or video devices are needed for collecting data on human movement. These data are pre-processed by applying filtering techniques such as Fast Fourier Transformation or wavelets or by reducing the high dimensional space with tools such as principal components analysis or linear discrimination analysis.¹⁶ Next, feature selection methods come into play, determining a subset of features from the initial set that is highly suitable for subsequent classification while adhering to various optimisation criteria. Among the efficient methods for feature selection are Sequential Forward Selection, which starts with an empty set and iteratively adds the feature that best meets the optimisation criterion, and Backward Selection, which involves removing features from the set in a repetitive manner.¹⁶ Finally, AI or machine learning classifiers are required to identify the corresponding class of motion, in our case, a class that reflects the BMS development of a child (e.g., delayed, normal or advanced for its age group). Machine learning tools include binary classification trees, decision engines, Bayes classifiers, k-Nearest Neighbour, rule-based approaches, linear discriminant classifiers and Support Vector Machines.¹⁷ More sophisticated deep learning tools, such as neural networks, are also used.¹⁸ From here onwards, we indistinctly use the expression ‘AI-related technology’ for referring to the full process described in Figure 1 or just the classification tools.

Figure 1. Process of recognition and classification of human motion patterns performed by artificial intelligence (AI)-related technologies.

There is a growing use of AI-related technology for physical performance assessment.¹⁹ For example, machine learning techniques have been applied to assess the intensity of physical activity performed by an adult.²⁰ In a recent review, at least 53 studies on motion detection with deep or machine learning were identified, of which 75% have been performed since 2015.²¹ AI has also been used to recognise alterations in walking or walking style and identify specific health problems related to walking^22,23 or to detect certain difficulties in motor skills associated with initial symptoms of other diseases.²⁴ Other AI algorithms are implemented to identify and evaluate psychomotor learning performance in students.²⁴ However, to date, there is no formal review of the multiple technologies used to assess BMS in children.

We aimed to perform a scoping review on studies related to the development and use of AI-related technologies to assess BMS in children. Our objectives were to: 1) determine the general characteristics of the studies; 2) describe the engineering of the AI technologies designed to assess BMS in preschoolers; 3) determine the substantive validation performed on the AI technologies identified, and 4) describe the current use of these AI technologies in applied research.

Methods

The protocol for this review is available here.⁴⁴ The PRISMA Extension recommendations for scoping reviews were applied for the full review, whereas the COSMIN guidelines were applied for objective 2.^25,26 The checklists of these guidelines can be found here.⁴⁷

Results

We identified 672 studies in the first search step, from which 12 studies were finally selected. Among these studies, five were focused on AI-related technology use, while seven were focused on AI-related technology engineering and/or validation (Figure 2).

Figure 2. PRISMA diagram for the scoping review.

During the last decade, most studies were performed in Asian and European countries (n=9/12, 74.9%) (Table 1). Almost all studies were carried out in children of both sexes (n=9/12, 75%), and only one was focused on children with some type of motor problem.

To capture the child’s movement, researchers mostly used simple devices such as digital video cameras (n=5/7, 71.4%) (Table 2). More sophisticated devices were less common, such as sensors attached to the body (n=2/7, 28.6%) or multimedia devices connected to personal computers (n=2/7, 28.6%). The software used for each device was different for each study. The most common type of AI-related technology was machine learning tools for movement pattern recognition (n=4/7, 57.1%), while deep learning algorithms were rarely used (n=1/7, 14.3%). Only a few of these tools are free-access (n=2/7, 28.6%). Most codes were implemented in Python (SCR_008394) and supported by libraries such as OpenGL (which produces 2D and 3D graphics)^29–31 and Numpy (SCR_008633) (which creates vectors and matrices, and mathematical functions) (45) that helps to process images that are captured in real-time and obtain an accurate representation of the movement.

For the COSMIN evaluation, we considered seven studies that developed a substantive validation of AI technologies (Table 3). More than half of the studies reported the evaluation of content validity (n=4/7, 57.1%), reliability (n=1/7, 14.2%), and construct validity (n=1/7, 14.2%) with an adequate level. However, other measurement properties, such as structural validity, measurement error and responsiveness, were inadequately or not evaluated in all studies, according to COSMIN standards (n=5/8, 62.5%). It was not unusual that a declared formal evaluation of a psychometric property (e.g., reliability) was followed by no reporting of final results.

In studies using AI-related technology, the children’s movements were captured by trained personnel (n=2/5, 40%) using digital cameras or camcorders (n=4/5, 80%) (Table 4). In addition, some supporting technologies that provide high-quality video motion capture, such as “Quintic Biomechanics software”, was also reported. Users reported some advantages of these technologies; for example, the short-term evaluation needed and precise and consistent measures that allow a detailed analysis of motor skills. However, no formal generalization of the conclusions to larger populations was reported as a technical limitation.

We identified 10 studies that updated and/or applied the exact AI-related technology reported in Tables 2 and 3 (Table III, supplemental material). Among those studies, 7/10, (70%) were used for the assessment of motor skills; and 3/10, (30%) were updated and used (i.e., a new version of the technology).

Discussion

We performed a scoping review of AI-related technologies developed and used to assess motor skills in children. Engineering work and technological features have been prioritized in these studies; for example, the use of advanced systems for motion capture or the training of sophisticated machine learning algorithms for movement patterns recognition. More importantly, the validation of these algorithms was strictly based on engineering criteria; it means, no substantive knowledge of the medical or psychological aspects of motor skills was integrated into the validation process. Technical features typically evaluated in psychometric instruments designed for assessing motor skills in children were also ignored (i.g., COSMIN criteria). The use of these AI-related technologies in scientific research is still limited.

Most studies on AI technologies engineering ignored the standard psychometric validation process (i.e., COSMIN standards). Although many of them complied with the good practices in the development of image processing-oriented software, none of them integrated a substantive validation. AI-related technology is good for identifying movement patterns that are rare in children or patterns that children of a certain age should show, and they are not. This capacity has enormous value for clinical and educative purposes. However, for these AI measures to be integrated into a formal clinical evaluation, some technical features must be confirmed. For example, the measurement error estimate is essential for evaluating individuals from the target population, allowing the definition of critical ranges (i.e., minimum and maximum values) to contrast individual measures and conclude an advantaged, normal or sub-normal motor skill development. Another important psychometric characteristic is responsiveness, which reveals whether any change seen between within-individual AI measurements performed before and after an intervention corresponds to true changes in motor skills (smallest detectable changes), which is linked to investigating when these changes are clinically relevant (minimal important changes).

A previous review of AI technologies for evaluating motor skills in paediatric populations warns that the validation of these tools is limited.³² As we do here, they concluded that this limitation has practical implications in the assessment precision and applicability in clinical contexts. Without a standard psychometric validation process, AI developers do not collect the correct and sufficient evidence to ensure the minimal validity and reliability required for this kind of measurement. For example, one of our reviewed studies reported that the AI algorithm was reliable and valid because it was based on a test previously declared reliable by its original author.³³ Differences between the population for which the original test was created and the sample used to develop the AI version can seriously compromise the reliability of the measures and their clinical interpretation criterion due to cultural/ethnic, linguistic, social, economic and age differences.¹² In practice, clinical interpretation is an essential component of measurement validity and usually requires evidence beyond the standard qualification norm. For example, the recent study reported a new video-based technology that was based on a classical motor skill test (i.e., that needs paper, pencil and evaluator’s criteria), showing concurrent validity against another measure of motor skills.³⁴ Contrasting AI measurements against external independent criteria is essential, not only to confirm that the algorithm is measuring what we intend to but also to connect these measurements with other signs and symptoms clinically relevant.^35–37 In this way, AI measurements become more informative and useful for a full evaluation of a children’s healthy development.

There are some factors explaining the limited production of AI-related technologies for evaluating motor skills in children. There is a priority for using AI to assess other health problems in this and other populations. During the last two decades, most AI for health has been developed for the diagnosis and follow-up of physical problems such as cancer, cardiovascular diseases, or neurodegenerative disorders in adult subjects.^19,38 High costs slow the production of these AI-related technologies,^39,40 especially in low-and-middle-income countries. Rich countries promote the investment of significant amounts of money for developing new cutting-edge technology,⁴¹ although for a wide range of purposes. In low- and middle-income countries, AI development suffers from some extra limitations, such as insufficient economic and human resources, limited data, non-transparent AI algorithm sharing, and scarce collaboration between technological institutions.⁴²

The use of AI-related technologies in scientific research is also limited, and this is linked to other factors. As expected, developers focused on engineering and not research to facilitate the use of their technologies. For example, only one of our reviewed studies performed a usability and feasibility analysis,⁴³ which is important to make the technology friendlier and more accessible to future users.³³ This can be explained, in part, because most of them is still developed within the academia, and not yet in the private sector and for commercial purposes. However, considering how they can improve the speed and precision of BMS evaluation of children for doctors and teachers, these AI-related technologies have great commercial potential in the educative and clinical contexts.

Conclusions

Engineering work and technological features have been prioritized in the studies about AI-related technologies. The validation of these algorithms was strictly based on engineering criteria; it means, no substantive knowledge of the medical or psychological aspects of motor skills was integrated into the validation process. Technical features typically evaluated in psychometric instruments designed for assessing motor skills in children were also ignored (e.g., COSMIN criteria). The use of these AI-related technologies in scientific research is still limited.