Validity and reliability of the Thai version of the Autism Treatment Evaluation Checklist : A two-phase diagnostic accuracy study

This study aimed to evaluate the psychometric properties of the Background: Thai version of the Autism Treatment Evaluation Checklist (Thai-ATEC); a tool which has been developed for Thai parents and caregivers who have children with autism spectrum disorder (ASD). Approval for this study was first obtained from the appropriate Ethics Methods: committee and from the original Autism Treatment Evaluation Checklist (ATEC) developers. This was a two-phase study. Phase 1 consisted of the forward–backward translation of the ATEC and phase 2 included the testing of psychometric properties, i.e. the validity and reliability of the final draft of the tool. The validity of the tool was assessed by comparing Thai-ATEC scores of parents and caregivers of 160 children with ASD with the assessment of a child and adolescent psychiatrist using DSM-V criteria on the same group of children. The inter-rater reliability of the tool was tested using a two-way model of intra-class correlation coefficient (ICC) for two-parent/caregivers’ assessment of 50 children with ASD. The validity of the Thai-ATEC was moderate to high. A cut-off point of Results: ≤38 scores was used to distinguish between children with ASD with mild symptoms and the rest of the children (sensitivity = 94%, specificity= 61.9%, and the area under ROC curve = 90%). A cut-off point of ≥68 scores was used to distinguish between children with ASD with a severe degree of symptoms and the rest (sensitivity = 94%, specificity = 62.8%, area under receiver operating characteristic curve = 85%). The inter-rater reliability was very strong (ICC = 0.97). The Thai-ATEC has moderate to high validity and high reliability. Conclusions:


Introduction
Autism spectrum disorders (ASD) are a cluster of neurodevelopmental disorders that are characterized by abnormalities in social interactions and verbal and nonverbal communication, and restricted, repetitive and stereotyped patterns of behavior, interests and activities 1 .ASD symptoms begin in the early years after birth, generally before 3 years of age 2 .The prevalence of ASD has increased over recent years in many parts of the world 3 .In Thailand, it was estimated that there were 180,000 children with ASD nationwide (i.e.2.8 per 1,000 population or 15 per 1,000 children age <15) 4 .However, this may underestimate the true scale.Accurate diagnosis needs adequately trained health professionals and appropriate tools 5 .
Many assessment tools have been developed to screen and diagnose children with autism, such as the Autism Diagnostic Interview-Revised (ADI-R) 6 , the Pre-Linguistic Autism Diagnostic Observation Schedule (PL-ADOS) 7 , and the Childhood Autism Rating Scale (CARS) 8 .These tools are mostly used by intensively trained mental health professionals.These tools are time-consuming, however; for example, the ADI-R takes 2 h to complete and the PL-ADOS takes 30 min.Scores derived from the tools had low sensitivity with older children with ASD 9 and these tools are not readily available to caregivers.Although these tools are appropriate for research, because of these limitations, other tools have been investigated that have been standardized for evaluating the changes of symptoms of children with autism, one example of which is the ATEC 10 .
The ATEC was developed by Rimland and Edelson at the Autism Research Institute 11 .The tool has been designed to enable parents to assess their children with ASD.It has increased the effectiveness of home caring because it allows for confirmation of the effectiveness of treatments provided 10,12 .This tool can be used by parents, caregivers, teachers, professionals and researchers.The ATEC has been translated to 18 languages and is widely used globally 10,13 .
After testing 1,358 children, results showed a high internal consistency (Pearson correlation coefficient, split-half method) of the total ATEC scores (r=0.94) and high internal consistencies of the subscales (speech/language: r=0.92; sociability: r=0.84; sensory/ cognitive awareness: r=0.88; health/physical behavior: r=0.82) 14 .Compared with the CARS utilized by a health care professional, the result showed that ATEC scores and CARS scores had a significant correlation (ρ =0.71, p-value <0.0001) 15 .It also showed that the predictive value of the ATEC was good.This could be used to continuously follow up the response to treatments 10 .
In Thailand, there is a high need for an efficient assessment tool, and a high interest in the ATEC.However, no formal standardized translation of the original ATEC has been done and no research to assess its psychometric properties in the Thai context has been conducted.Therefore, this study aimed to translate the ATEC into Thai and evaluate its psychometric properties.

Methods
This was a two-phase study.Phase 1 consisted of the forwardbackward translation of the ATEC and phase 2 included the testing of psychometric properties, i.e. the validity and reliability of the final draft of the tool.The full study protocol can be accessed at the Khon Kaen University graduate school or from the corresponding author upon request.
Phase 1: Forward-backward translation of the ATEC The authors obtained permission from Dr. Stephen M. Edelson, the original developer of the ATEC at the Autism Research Institute (San Diego, CA, USA), for the use of the ATEC in this study.The forward-backward translation included six processes: 1) studying and gaining understanding of the assessment tool, after which a child psychiatrist work at Child and Adolescent Mental Health Rajanagarindra Institute, Bangkok and clinical psychologist work at Psychiatric Hospital Nakhon Ratchasima Rajanagarindra translated it into the Thai language; 2) synthesis of the first translation; 3) backward translation into the English language by a British translator who had been living in Thailand for over 10 years(Private business at his home in Bangkok), and a professional translator Director and Tutor at Samarthome, Nakhonratchasima, both employed for this particular task; 4) comparing the back translation with the original version for content, sematic, idiomatic, and cultural equivalences by an expert committee comprising six members, including the child psychiatrist, the clinical psychologist, two backward translators, an expert in communication with children with ASD work at Faculty of Medicine Ramathibodi Hospital, Mahidol University, Bangkok, and a parent of a child with ASD with a high English proficiency, Vice president Autism Society of Thailand, Bangkok; 5) trying out the assessment tool, during which the processes were undertaken iteratively until the tool had reached acceptable understandability by stakeholders; and 6) submitting the final tool and documentations to the original ATEC developers.The complete Thai-ATEC can be found in Supplementary File 1.

Phase 2: Testing of the validity and reliability psychometric properties of the final draft tool
The validity of the tool was assessed by comparing the Thai-ATEC scores from parents and caregivers of 160 children with ASD, with the assessment of the same child and adolescent psychiatrist using DSM-V criteria 2 of the same group of children with ASD.The psychiatrist was blinded to the results of the parallel measure.
The validity indexes included sensitivity, specificity, predictive value, likelihood ratio positive (LR+), likelihood ratio negative (LR−), accuracy, and area under receiver operating characteristic (ROC) curve.These were calculated by exploring an appropriate cut-off point score that was able to distinguish between children with ASD with mild degree of symptoms and the rest of the children.Another appropriate cut-off point score was assessed for its ability to distinguish children with ASD with a severe degree of symptoms from the rest of the children.The tool's inter-rater reliability was tested by comparing the Thai-ATEC scores from 50 pairs of parents/caregivers of 50 children with ASD.Each pair of parents/caregivers was independently assess the child with ASD.

Participants
Inclusion criteria were parents/caregivers of children with ASD who were able to read and write.The age of the children with ASD was ≥3 years old.The sample size was calculated based upon data from a pilot study with expected sensitivity = 0.81, expected specificity = 0.60, prevalence = 0.4, desired pecision = 0.10 and a confidence level = 95%.Therefore, 160 parents and caregivers of children with ASD were needed.They were recruited simple random sampling during an appointment with the psychiatrist at the outpatient clinic at the North Eastern Institute of Child and Adolescent Mental Health, Khon Kaen province, Thailand, during May-August 2017.The tool's inter-rater reliability was tested using a two-way model of intra-class correlation coefficient (ICC) rater assessment of the same children with ASD.The participants of the inter-rater reliability test consisted of 50 pairs of parents and caregivers who were subsets of the 160.Each pair was selected for this test because both were caring for the same child and accompanied the child on the day of the assessment.Therefore, a child with one parent or one caregiver on those days at the hospital was not selected.After a child was examined by the psychiatrist, each pair of parents and caregivers was invited into a separate room and informed about how to complete the Thai-ATEC.The pairs were not allowed to meet and talk to each other.There was no time limit to fill in the checklist.Parents and caregivers spent an average of 9.21 minutes completing the tool.

Statistical analysis
The validity indexes including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, likelihood ratio (LR+/−), area under receiver operating characteristic (ROC) curve and the appropriate cut-off point of the tool were calculated by comparing scores obtained from the parents' and caregivers' Thai-ATEC assessment with the results of the psychiatrist's assessment using DSM-V criteria.The 95% confidence intervals of these indexes were also estimated.An appropriate cut-off point score was explored based on its ability to distinguish between children with ASD with mild degree of symptoms and the rest of the children.Another appropriate cut-off point score was assessed for its ability to distinguish children with ASD with a severe degree of symptoms from the rest of the children.The tool's inter-rater reliability was calculated using a two-way model of intra-class correlation coefficient (ICC) rater assessment of the same children with ASD.The SPSS statistical software version 19.0 was used to perform these data analyses.

Ethical approval and consent
Approval for the study was obtained from the Khon Kaen University Ethics Committee in Human Research (HE601001).Parents of children with ASD provided written informed consent for their participation in the research.
At a cut-off score of ≥68 to distinguish between children with ASD with a severe degree of symptoms and the rest of the children with mild to moderate symptoms, it was found that the tool's sensitivity was 91.3% (95% CI, 0.67-0.98), the specificity was 62.8% (95% CI, 0.55-0.74), the PPV was 28.8% (95% CI, 0.17-0.39), the NPV was 95.7% (95% CI, 0.91-0.99), the accuracy was 68.1%, the LR+ was 2.40, the LR− was 0.26, and the area under ROC curve was 85% (Table 2).Therefore, the above findings suggest that children with ASD can be assessed using the Thai-ATEC and the severity of ASD symptoms can be grouped according to the cut-off points found in this study as follows: Level of mild symptoms: Thai-ATEC scores in the range of 1-38 Level of moderate symptoms: Thai-ATEC scores in the range of 39-68 Level of severe symptoms: Thai-ATEC scores in the range of 69-179 The flow of participants can be seen in Supplementary File 2.

Reliability of the Thai-ATEC
ICCs were calculated to evaluate the inter-rater reliability of the tool.It was found that ICC for the total score was 0.97 (95% CI, 0.97-0.98).When we analyzed each subset of the tool, that is speech/language/communication, sociability, sensory/cognitive awareness and health/physical/behavior, it was found that the ICCs were 0.94, 0.86, 0.89 and 0.97, respectively.These were all more than 0.75, meaning that the assessment was highly consistent 16 (Table 3).

Table 3. Inter-rater reliability of the Thai-Autism Treatment Evaluation Checklist (ATEC).
The first observer was one of the parent/caregiver pair, and the second observer was the other person in the parent/caregiver pair.

Discussion
To our knowledge, this was the first study of its kind that attempted to formally translate the ATEC into Thai.Standardized and rigorous translation processes were employed.Beaton et al. proposed guidelines for the process of cross-cultural adaptation of self-report measures, including forward and backward translation to ensure that tools are appropriate and comparable 18 .This study has intentionally followed this guideline and employed additional processes which were needed.A child and adolescent psychiatrist, a Thai professional translator, a professional ASD communicator, health professionals responsible for the care of children with ASD, a British professional translator and a parent with a high English proficiency who had a child with ASD were all involved in the translation process to ensure that the final tool was of high quality and that the meaning of the Thai-ATEC was consistent with the original ATEC tool.
The psychometric properties of the Thai-ATEC were moderate to high.The sensitivity of the tool was 94%, its specificity was 62%, and the area under ROC curve was ≥85%.The discovered cut-off points were used to divide the severity of ASD symptoms into three levels: mild, moderate and severe.These levels mild were similar to the original ATEC, However, the results showed that moderate and severe levels were different from the original 14 .(Table 4).
Parents and caregivers can use the ATEC to monitor children with ASD to assess the severity of impairment in different domains and the progress in response to intervention over time.The ATEC covers not only behavioral issues, but also health and systemic issues, such as sleep problems, seizures, eating, gastrointestinal problems, hyperactivity, self-injuries and sleep disturbance.
Health and systemic issues are of high concern among parents 19 .
A study in India comparing the CARS and clinical diagnoses found that at a CARS score of ≥33, the sensitivity of the scale was 81.4%, the specificity was 78.6% and the area under the curve was 81% 16 .Thus, the Thai-ATEC had higher sensitivity but lower specificity than the Indian CARS.Geier et al. found that the CARS correlated well with the ATEC 15 .
The inter-rater reliability of the Thai-ATEC was very strong.The ICC of the total score was 0.97 and the ICCs of subscale 1, 2, 3 and 4 were 0.94, 0.86, 0.84 and 0.97, respectively.The Indian study that compared the CARS with clinical diagnoses by psychiatrists 16 found that inter-rater reliability was only 0.74.An ICC of >0.9 is perceived as indicating high reliability 20 .
Limitations of this study included the time consuming nature of the processes, owing to the limited number of child and adolescent psychiatrists involved in the study.Child and adolescent psychiatrists are rare in Thailand.
In this study, the experienced child and adolescent psychiatrist who acted as the gold standard, followed the DSM-V criteria and spent 5-10 minutes per person.This relatively short length of time might have caused some mistakes and may have led to misclassification of symptoms that could have affected sensitivity and specificity.However, the single-psychiatrist approach taken in this study may have minimized subjective differences in scoring methods and provided superior consistency to a multi-psychiatrist approach.
This study was conducted in the north-eastern part of Thailand.Therefore, further studies are needed to explore the usability of the new Thai-ATEC in other parts of Thailand, where local dialects are present.

Conclusion
The Thai-ATEC has moderate to high validity and high reliability.The cut-off points at scores of ≤38 and ≥68 were appropriate to distinguish the level of severity between the mild, moderate, and severe ASD symptoms.Parents and caregivers of children with ASD can efficiently use the tool to assess their children at home and communicate the results with health professionals. 1.

Open Peer Review
Current Peer Review Status:

Manote Lotrakul
Department of Psychiatry, Mahidol University, Bangkok, Thailand The advantages of the ATEC are that it is shorter than other measures, easy to understand and designed to be completed by parents, teachers, or caregivers, making it a suitable tool for a country where there is a shortage of psychiatrists.It provides quantitative measurements on important domains of ASD severity which increased the effectiveness of home-base care for children with ASD.
The authors have translated the ATEC into Thai following Beaton's Guidelines for the cross-cultural translation and adaptation of self-report instruments.The psychometric properties of the Thai-ATEC have been tested and reported as acceptable to good.The methods section and results of the study have sufficient and clear details.The authors are skeptical whether the Thai-ATEC can be used in other parts of Thailand but this may not be the case because the wording in each item is natural, clear and easy to understand.

Comments:
Please change "DSM-V" to "DSM-5" It would be better if the author also present the other type of reliability, i.e., the internal consistencies of the ATEC scores and subscales.Page 6 paragraph 4: It is not relevant to compare the Thai-ATEC with the CARS study in India as the purpose of that study was to develop the screening tool whereas the ATEC is designed to measure ASD severity.Page 6 paragraph 6: This research is time consuming or slow-paced but it is not the limitation of the study.

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound?

Are sufficient details of methods and analysis provided to allow replication by others? Yes
If applicable, is the statistical analysis and its interpretation appropriate?Yes Are all the source data underlying the results available to ensure full reproducibility?Yes

Are the conclusions drawn adequately supported by the results? Yes
No competing interests were disclosed.

Competing Interests:
I have read this submission.I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com

Dataset 1 .
The Thai-ATEC scores for each child from each parent/ caregiver and the corresponding diagnosis from the child psychiatrist http://dx.doi.org/10.5256/f1000research.14537.d20217517 /doi.org/10.5256/f1000research.15824.r34245© 2018 Lotrakul M. This is an open access peer review report distributed under the terms of the Creative Commons , which permits unrestricted use, distribution, and reproduction in any medium, provided the original Attribution Licence work is properly cited.

©
2018 Geier M. This is an open access peer review report distributed under the terms of the Creative Commons , which permits unrestricted use, distribution, and reproduction in any medium, provided the original Attribution Licence work is properly cited.Mark R GeierInstitute of Chronic Illnesses, Inc., Silver Spring, MD, USAThe researchers have undertaken an important and timely study to evaluate the validity and reliability of the Autism Treatment Evaluation Checklist (ATEC) for diagnosis and evaluate children diagnosed with autism in Thailand.They have demonstrated that they were able to translate the ATEC from English into the Thai language, and then utilize the ATEC as a means to identify and evaluate the severity of children diagnosed with autism in Thailand.They have utilized appropriate methods for data collection and data analyses.The study results are consistent with previously cited published studies examining the validity of the ATEC on different populations using different metrics.The researchers appropriately consider study limitations, and the study conclusions are supported by the data.Is the work clearly and accurately presented and does it cite the current literature?YesIs the study design appropriate and is the work technically sound?YesAre sufficient details of methods and analysis provided to allow replication by others?YesIf applicable, is the statistical analysis and its interpretation appropriate?YesAre all the source data underlying the results available to ensure full reproducibility?YesAre the conclusions drawn adequately supported by the results?YesNo competing interests were disclosed.Competing Interests:I have read this submission.I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.Page 10 of 11 F1000Research 2018, 7:538 Last updated: 17 MAY 2019

Table 1 . Validity of the Thai-Autism Treatment Evaluation Checklist (ATEC) at the cut-off score of ≤38.
PPV, positive predictive value; NPV, negative predictive value; LR+, likelihood ratio positive.

Table 2 . Validity of the Thai-Autism Treatment Evaluation Checklist (ATEC) at the cut-off score of ≥68
.

Table 4 . Symptom severity levels of the original Autism Treatment Evaluation Checklist (ATEC) and the Thai-ATEC
.A comparison of mild, moderate and severe symptoms between the original and new Thai version of the ATEC.