Establishing Reference Values for a New Computerized Cognitive Function Test Program for Children
Article information
Abstract
Objective
To establish reference values for the computerized cognitive test and evaluate cognitive function improvements across different age groups, we introduce the computerized Cognitive Function Test program (eCFT), specifically designed for children. We aimed to establish eCFT reference values and assess cognitive function improvements across different age groups.
Methods
We included children aged 3–6 years with confirmed normal cognition based on the Korean Developmental Screening Test for Infants and Children and Kaufman Assessment Battery for Children-II. The eCFT consists of 8 subtests for visual perception, attention, memory, and executive function.
Results
A total of 66 participants (36 males and 30 females) with an average age of 4.4 years participated. The age 6 group consistently outperformed both age group 3 and 4 in terms of correct responses. With regard to the completed stage, the “selective auditory stimulus” test findings were 2.0 and 3.9 for the age 3 and age 6 groups, respectively (p<0.05). The “trail-making” test findings were 1.7, 2.1, 2.6, and 2.8, respectively (between ages 3 and 6, p<0.01; between ages 4 and 6, p<0.05); moreover, the age 5 group surpassed the age 3 group (2.6 and 1.7, respectively, p<0.05).
Conclusion
The eCFT is an easily accessible tool to evaluate cognitive function in young children. We introduce reference values with a cutoff range for preschool-aged children, enabling early intervention for those with cognitive impairment. Given its accessibility and relatively short evaluation time, the eCFT has potential for clinical use.
INTRODUCTION
Cognitive function plays a crucial role in children’s independence in daily activities and social engagement [1]. Cognitive rehabilitation is necessary to treat children with cognitive impairment or developmental delays because it can prevent cognitive decline and reduce social costs [2,3]. The effectiveness of cognitive rehabilitation has been demonstrated in patients with cognitive impairments [4,5] and it has even been shown to improve cognitive function in children without overall learning difficulties [5]. Although cognitive rehabilitation can be achieved through pharmacological or non-pharmacological treatments, more careful consideration is needed when administering medications for children owing to the negative perception of caregivers and patients towards medications, and concerns about potential side effects. Additionally, there is more substantial evidence supporting the use of pharmacological intervention in adults, while the data regarding their efficacy and safety in children is insufficient [6-10]. However, traditional programs such as one-on-one or group therapy may be unavailable or impractical because of resource limitations and health conditions [11].
Computerized cognitive training (CCT) can compensate for these shortcomings, and several CCT programs, including RehaCom (HASOMED), ComCog (neofect), and Bettercog (M3 Solutions), have demonstrated effectiveness in enhancing cognitive function in individuals with cognitive impairment. RehaCom has shown its efficacy in improving cognitive function in children with ADHD and chronic stroke patients [12,13]. ComCog has demonstrated cognitive improvement in individuals with mild cognitive impairment (MCI), stroke, traumatic brain injury (TBI) patients [14,15]. In addition, ComCog has exhibited greater cognitive enhancement in a young TBI group than in an old TBI group [16].
The accurate evaluation of cognitive function is essential prior to rehabilitation. Computerized cognitive function tests serve as practical and accessible tools for this purpose, as they help reduce the examiner’s workload, minimize the variability introduced by human assessors, enable automated data storage, and capture reaction time data. Several Computerized Cognitive Test Batteries are available for children [17,18]. Cambridge Neuropsychological Test Automated Battery is the most frequently used one and it is only applicable for children aged 4 years and above [18]. The National Institutes of Health Toolbox Cognition Battery and Amsterdam Neuropsychological Test are the only tests that can be applicable from the age of 3, while others can be employed for children aged 4–5 and above [18].
The computerized Cognitive Function Test program (eCFT) is a newly developed online cognitive function test that is part of the Bettercog (computerized cognitive rehabilitation program) developed in Korea. The eCFT is specifically designed to assess cognition in Korean children aged 3 and above, using words and images which are familiar to Korean children. It utilizes a touchscreen-based interface for user interactions. We, therefore, aimed to establish reference values for the eCFT and to investigate whether there is a trend of increasing cognitive function with age.
METHODS
Participants
We included children aged 3–6 years who were identified as having normal cognition through the Korean Developmental Screening Test for Infants and Children (K-DST) and Kaufman Assessment Battery for Children-II (K-ABC-II) test. Based on previous studies’ methodology, we aimed to recruit 15 subjects per age group [19, 20].
The participants were recruited via poster advertisements at the National Traffic Injury Rehabilitation Hospital and the study was conducted from November 2021 to September 2022. The exclusion criteria were hearing difficulties, visual impairments, hemi-neglect, and other musculoskeletal or neurological disorders that could impede following the instructions. The participants visited the research lab once to complete the K-DST, K-ABC-II, and eCFT. The assessments were conducted by a single occupational therapist, with a total duration of approximately 2 hours, including approximately 40 minutes for the eCFT.
The K-DST is a standardized questionnaire completed by caregivers to identify developmental delays in children. It evaluates various domains, such as motor skills, language, social and personal skills, and problem-solving ability [21,22]. It consists of six domains, each with a scoping range from 0 to 24 points. If a domain score is more than 1 standard deviation (SD) above the mean within the age group, it indicates a potentially higher-than-average developmental level. Scores within ±1 SD represent normal development. Scores below -1 SD and greater than -2 SD indicate a need for continuous observation. Scores below -2 SD suggest developmental delay, requiring further evaluation. The revised 2017 version was used in this study [23]. The K-ABC-II is a standardized test designed to measure cognitive ability and achievement in children and adolescents, to assess cognitive domains, including memory, planning, attention, and problem-solving abilities [24]. It comprises five domains and 18 subtests, with each subtest having a maximum scale score ranging from 1 to 19 points. Depending on the child’s age, subtests are selectively administered rather than conducting all 18 subtests on each child. Test results are interpreted using standard scores, with a mean of 100 and a SD of 15, and a maximum score of 160. Standard scores within ±1 SD are considered “average,” scores below -1 SD and greater than -2 SD are “low,” scores below -2 SD are “very low,” scores higher than +1 SD and below +2 SD are “high,” and scores higher than +2 SD are “very high.” The Korean version of the test was used to assess the cognitive function in this study [25].
Computerized program description
The eCFT consists of 8 subtests designed to evaluate various cognitive domains, including visual perception, attention, memory, and executive function. These subtests are: “finding half,” “visual and auditory stimulus,” “flip sequence,” “delayed recall,” and “trail-making.” Table 1 presents the instructions for each subtest. The “finding half” test was used to assess visual perception. The “visual and auditory stimulus” test evaluates attention and reaction time, while the “selective visual” and “selective auditory” stimulus tests measure complex attention and reaction times. The “flip sequence” test is designed to assess immediate memory and the “delayed recall” test focuses on delayed memory. The “trail-making” test is a tool to evaluate working memory.
Computerized Cognitive Function Test program (eCFT) task description: 8 subtests
Each subtest has unique characteristics and requirements. The “finding half” test requires participants to respond to each of the 20 questions within a 10-second timeframe. The “visual stimulus,” “selective visual stimulus,” “auditory stimulus,” and “selective auditory stimulus” tests are structured across 5 stages each. The “visual” and “auditory” stimulus tests present 5 stimuli at each stage, for a total of 25 questions, while the “selective visual” and “selective auditory stimulus” tests feature 3 true and false stimuli in each stage, resulting in a total of 30 questions. False stimuli were presented randomly, and the stimulation and interval times were modified to adjust the difficulty level of each stage. For the “visual” and “selective visual stimulus tests,” the stimulation times per stage were set to 4, 2.5, 2, 1.5, and 1 second. Unlike that in visual stimuli, it was challenging to maintain a constant stimulation time for auditory stimuli and it was adjusted only by the interval time. The interval time for all four tests was consistently set to 3, 2.5, 2, 1.5, and 1 second [26]. The “flip sequence” test comprises five stages, with a total of 20 questions. The interval time ranged from 3 to 5 seconds. Lower and higher stages were evaluated with a 3- and 5-second interval, respectively. This means that participants were required to remember the “flip sequence” for 3 to 5 seconds before recalling it, and they were required to answer each question within 10 seconds [27]. The “delayed recall” test consists of a single stage comprising 10 questions. Participants are shown two pictures for 5 seconds, followed by playing “Whack-A-Mole” for 10 seconds. Following this, they are required to recognize the previously displayed pictures from a set of five within the given 10 seconds. The trail-making test comprised 3 stages. In the first stage, the participants were instructed to click on the order of the rainbow colors. In the second stage, they were required to click from 1 to 7, and, in the final stage, they alternated between rainbow colors and numbers. The participants were required to respond to each stage within 2 minutes. The test time was shortened if the participants responded earlier than the given time or failed a stage.
Outcome measures
The data collected from the eCFT included the number of correct answers, incorrect answers, incomplete responses, completed stages, and reaction times.
Statistical analysis
Statistical analyses were performed using the IBM SPSS statistics 23 (IBM Corp.). Descriptive statistics, such as mean, SD, 15th percentile, and 85th percentile of the number of correct answers and reaction time, were presented. An analysis of variance was used to compare mean differences across age groups, and a post hoc Games–Howell test was conducted to determine specific mean differences. Statistical significance was set at p<0.05.
Ethics approval
This study adhered to the Declaration of Helsinki (2013 revision) and ICH-GCP guidelines. The study received approval from the Institutional Review Board of Seoul National University Hospital (IRB No. 2107-171-1236) and from the Institutional Review Board of National Traffic Injury Rehabilitation Hospital (IRB No. 2021-10-022). Informed consent was obtained from legally authorized representatives.
RESULTS
We included 66 participants (36 males) aged 3–6 years old (mean, 4.4 years old). None of the participants had a significant medical history and information regarding their socioeconomic status was not collected for this study.
The results of the K-DST and K-ABC-II are presented in Table 2. For the K-DST, the average score was 127.2±10.8. The total scores per age group were 128.9, 126.4, 124.9, and 137.5, respectively. In the K-ABC-II, the average score was 158.4±50.2. The total scores for each age group were 104.3, 143.9, 188.0, and 204.4. All participants scored within the normal range on the K-DST and K-ABC II tests.
Demographic and clinical characteristics
Table 3 presents the results of the 8 eCFT subtests, including the number of participants, SD, 15th percentile, and 85th percentile for both the number of correct answers and the reaction time. A cutoff value at the 15th percentile, equivalent to 1 SD, was employed to identify abnormal performance.
Test results according to age group
The results demonstrate a progressive improvement in the number of correct answers with increasing age (Fig. 1). Improvements in the “selective visual stimulus,” “auditory stimulus,” “selective auditory stimulus,” “flip sequence,” and “trail-making” tests were observed in children aged 3–6 years. The age 6 group exhibited significantly superior performance compared to the age 3 and 4 groups in the “visual stimulus” test (age 3 group: 17.9±4.6, age 4 group: 21.4±4.0, and age 6 group: 24.6±1.1; between ages 3 and 6, p=0.001 and between ages 4 and 6, p=0.011). The age 6 group outperformed the age 3 group and the age 5 group achieved a higher score than the age 4 group in the “auditory stimulus” test (age 3 group: 14.5±5.8, age 4 group: 16.9±6.4, age 5 group: 19.9±5.2, and age 6 group: 23.8±1.9; between ages 3 and 6, p=0.001 and between ages 4 and 6, p=0.001). No disparities in performance across age groups were observed in the “selective visual” test, while the age 6 group outperformed the age 3 group in the “selective auditory” test (age 3 group: 4.4±2.3 and age 6 group: 13.3±8.6, p=0.010). In the “flip sequence” test, the age 6 group performed significantly better than the age 3 and 4 groups (age 3 group: 9.5±7.8, age 4 group: 11.5±8.0 and age 6 group: 22.8±7.6; between ages 3 and 6, p=0.001 and between ages 4 and 6, p=0.001), and in the “delayed recall” test, the age 6 group surpassed the performance of the age 4 group (age 4 group: 6.0±3.2 and age 6 group: 9.3±1.0. p=0.001). The age 6 group achieved higher scores than the age 3 and 4 groups in the “trail-making” test (age 3 group: 6.0±6.6, age 4 group: 10.2±8.7, age 5 group: 15.1±8.2, and age 6 group: 22.5±8.0; between ages 3 and 6, p<0.001 and between ages 4 and 6, p=0.002), while the age 5 group outperformed the age 3 group (p=0.017).
Comparison of cognitive function scores among age groups per subset. (A) Attention. (B) Memory. (C) Working memory. *p<0.05, **p<0.01, and ***p<0.001.
Although the reaction time tended to decrease as the children’s age increased, the difference was not statistically significant. Regarding the completed stages, the age 6 group completed more stages than the age 3 group in the selective auditory stimulus test. The average completed task stage was 2.0 for the age 3 group and 3.9 for the age 6 group (p<0.05; Fig. 2). The task completion rates for passing up to stage 5 were as follows: 10% for the age 3 group, 11% for the age 4 group, 14% for the age 5 group, and 57% for the age 6 group. In the “trail-making” test, the age 6 group completed more stages than the age 3 and 4 groups did, while the age 5 group completed more stages than the age 3 group did. The average completed task stage was 1.7 for the age 3 group, 2.1 for the age 4 group, 2.6 for age 5 group, and 2.8 for the age 6 group (between ages 3 and 6, p<0.01; between ages 4 and 6, p<0.05; and between ages 3 and 5, p<0.05). The task completion rates for passing up to stage 3 were as follow: 29% for the age 3 group, 47% for the age 4 group, 79% for the age 5 group, and 92% for the age 6 group.
Comparison of completed stages among age groups by subsets. (A) Attention. (B) Memory. (C) Working memory. *p<0.05 and **p<0.01.
DISCUSSION
We evaluated age-specific differences in eCFT results among children without cognitive impairment and proposed cutoff values based on the 15th percentile of the number of correct answers and reaction time for each subtest. The eCFT results indicated a pattern of increasing correct responses and decreasing reaction times with age. Notably, age 6 group consistently outperformed both age groups 3 and 4 in terms of correct responses. In the “selective auditory stimulus” test, the age 6 group demonstrated higher task completion rates compared to the age 3 group. In the “trail-making” test, the age 6 group completed more task stages than both the age 3 and age 4 groups. To the best of our knowledge, this is the first study to establish reference values for computerized cognitive assessments tailored specifically to preschool-aged children with typical cognitive development in Korea.
Our findings revealed that the mean performance scores in each cognitive domain showed slight variations depending on the child’s age, with older children tending to achieve higher scores. Notably, the age 6 group consistently outperformed the age 3 and 4 groups across various subtests. Furthermore, older groups tended to exhibit shorter reaction times in the attention domain, although this trend was not statistically significant. However, this pattern of shorter reaction times with increasing age was not observed in the findings of the “flip sequence” test. In other words, in challenging tasks such as memory function tests, younger children can exhibit shorter reaction times with lower accuracy than those exhibited by older children.
Furthermore, reaction time is directly related to attention. In a previous study, the average reaction time in the visual Continuous Performance Test (CPT) for typical Korean adults was reported as 0.42 seconds, while in the auditory CPT, it was 0.59 seconds [26]. In our study, the 6-year-old participants exhibited mean reaction times of 0.58 seconds in the “visual stimulus” test and 0.91 seconds in the “auditory stimulus” test, which differs from adults. The lower reaction time in the age 6 group compared to that of adults could be derived from either motor control or attention-related issues. Examining the results of the auditory stimulus test can further elaborate this. The average reaction time for visual and auditory controlled CPT (CCPT) in adults was reported as 0.41 and 0.57 seconds, respectively. Additionally, the average reaction times for the “selective visual stimulus” and “selective auditory stimulus” tests were 0.44 and 1.27 seconds, respectively, in the age 6 group. This suggests that inhibitory responses to auditory stimuli are somewhat more challenging than those to visual stimuli, and it implies that the difference observed in visual stimuli may be attention-related and attributed to processing speed rather than motor control [26]. In another study [19], a computerized attention assessment for children aged 5 to 15 years was developed, which was used to evaluate auditory CCPT and visual CCPT, dividing the total assessment time into three phases (early, middle, and late). The test provided target ratios of 22% in the early phase, 50% in the middle phase, and 78% in the late phase, and it maintained a fixed stimulus interval of 2 seconds and a stimulus presentation time of 0.1 seconds. The results revealed that the average reaction time for visual CCPT in 5- and 6-year-olds was 0.67 and 0.61 seconds, respectively, whereas our eCFT “selective visual stimulus” reported 0.61 and 0.44 seconds, respectively. For auditory CCPT, the average reaction time in 5- and 6-year-olds was 1.07 seconds, and our eCFT’s “selective auditory stimulus” test reported 1.40 and 1.27 seconds, respectively [19].
Regarding the completed stages, statistically significant differences were observed only in the “selective auditory stimulus” test and “trail-making” test. There were no differences in other attention-related subtests or the “flip sequence” test. This could be interpreted as indicating that executive function in 6-year-olds significantly outpaces that in other age groups, and that they exhibit superior sustained attention and better impulsivity control.
In addition, accurate responses to auditory stimuli are often lower than those to visual stimuli. In particular, the “selective auditory stimulus” test is challenging because of inhibition [19]. Our findings have corroborated this trend, revealing significant differences in the “selective auditory stimulus” among the completed stages.
The “trail-making” test assess motor speed, speed of mental processing and mental flexibility. It was used for children with learning disabilities and attention problems as well as those affected by brain injury [28]. In a study involving children aged 9 to 14, the alternative (number-letter) “trail-making” test was conducted, dividing participants into three groups: a group with brain injuries, a group with academic difficulties, and a control group. The test involved connecting 15 numbers and 15 letters alternatively. The completion times differed significantly, with the brain-damaged, academic difficulties, and control groups requiring 111.44, 68.38, and 33.56 seconds, respectively, to finish the test. [29]. As our study targeted children aged 3 to 6 years, the “trail-making” test used in our study utilized numbers 1–7 and colors (red, orange, yellow, green, blue, indigo, and violet). The average completion times of step 3 for the age 3, age 4, age 5, and age 6 groups were 68.54, 53.71, 44.73, and 36.17 seconds, respectively (not presented in Table). However, the number of individuals who completed the step 3 varied. In the age 3, age 4, age 5, and age 6 groups; 4/14, 9/19, 11/14, and 12/13 participants, respectively, completed the test. Even when examining normally developing children, there were instances where individuals did not successfully complete the alternative “trail-making” test. Therefore, considering these challenges, the use of the “alternating selection (step 3)” test might be more suitable for older age groups than that of the other test.
Computerized tests offer several advantages, including reducing the burden of examiners, minimizing variation by examiners, automatically storing data, obtaining information about reaction time, and allowing testing to be done at home [19], and have, therefore, gained popularity. The BMT-i, computerized Adaptable Test Battery to assess children’s academic skills and cognitive function has been validated [17].
Measuring reaction time is a good screening tool for normal aging and MCI, and it is more accurate than measuring correct answers in adults aged >60 years of age [30]. Previous studies established a correlation between processing speed and overall cognitive performance [31]. Notably, individuals diagnosed with ADHD exhibit challenges in sustaining attention and show significant impairments in selective attention as they have higher rates of omission and commission errors. In the case of unpredictable stimulus, children with ADHD tend to make a substantial number of those errors [32,33]
This study has certain limitations. First, it was conducted within a specific region of Korea, which potentially limits the generalizability of our results to other populations. Second, the small sample size may have contributed to the not significant differences between the groups. For instance, in the “selective visual” test, the age 6 group obtained considerably higher scores than the other groups (scores: 22.9, 24.6, 26.3, and 32.1 for ages 3, 4, 5, and 6, respectively), although this difference did not reach statistical significance. It is plausible that a larger sample size would reveal significant differences. Third, we did not collect detailed information about the patients’ characteristics, such as their socioeconomic status, number of siblings, and parental education levels. These factors may have influenced our results. Forth, we used colors (red, orange, yellow, green, blue, indigo, and violet) instead of letters in the “trail-making” test. This was done to avoid the possible unfamiliarity of very young children with letters. However, the test may not have been accurately conducted for children with color blindness, considering that we did not conduct a screening test for color blindness.
Using the eCFT, we established reference values for each subtest within the age range of 3–6 years, providing insights into the cognitive development of preschool children without cognitive impairment. Our findings indicated a progressive improvement in cognitive performance with age, which was particularly evident in specific subtests. The eCFT, known for its accessibility and cost-effectiveness, holds great promise for clinical application in identifying cognitive impairment and facilitating early intervention in preschool-aged individuals.
Future research should explore the potential of the eCFT as a tool to distinguish between children with and without cognitive impairments, such as those with low academic skills or attention deficits. Additionally, further investigations may seek to identify the age at which children’s scores on the eCFT begin to plateau in a specific cognition domain.
Notes
CONFLICTS OF INTEREST
The funding for this research was provided by M3 SOLUTIONS, CO., LTD, with Seoul National University Hospital serving as the lead research institution. This work falls under the category of commissioned research and the M3 Company played no role in the research design, data collection, analysis, or interpretation of results. The funding covered research expenses, equipment acquisition, and publication costs, with no influence on the research findings. The research was conducted independently, and the researchers maintained academic independence regarding the interpretation of the research results.
FUNDING INFORMATION
The funding for this research was provided by M3 SOLUTIONS, CO., LTD, with Seoul National University Hospital serving as the lead research institution (grant number: 0620214740). This work falls under the category of commissioned research.
AUTHOR CONTRIBUTION
Conceptualization: Lee GJ. Data curation: Kim H, Lee S. Methodology: Lee GJ. Formal analysis: Lee H. Funding acquisition: Lee GJ. Investigation: Lee H, Kim H. Project administration: Lee GJ. Resources: Lee GJ. Supervision: Lee GJ. Validation: Lee H. Visualization: Lee H. Writing – original draft: Lee H. Writing – review and editing: Lee H.
