Executive functioning in children with ADHD Investigating the cross-method correlations between performance tests and rating scales

Executive functions (EF) represent higher-order neurocognitive processes involved in the regulation and direction of thoughts and behaviors, relevant to problem-solving, goal setting, and following plans (1). From the multitude of specific EF subtypes, three core EFs are identified: 1) Inhibition: the ability to exert behavioral self-control and cognitive interference control when confronted with external/internal stimuli. 2) Working memory: the ability to retain relevant information in one’s mind while using it to solve a present problem. 3) Cognitive flexibility: the ability to change perspective and/or approach when confronted with changes such as novel information, rules, or changes in relevance (1).

EF is of central importance to a person’s daily functioning and is an important factor in the organization of actions to reach present and future goals. EF plays a central role in the current understanding of ADHD, and the basic dysfunctions underlying the disorder, as well as in other psychiatric disorders (2,3,4,5).

Even though EF play a central role in ADHD, and individuals with ADHD score significantly lower than control groups on measures of EF (6, 7), research also shows that the relationship between ADHD and EF is not well understood. Evidence shows that not all children with ADHD express deficits in areas of EF, although children with ADHD show a higher risk of these deficits (8,9,10). High levels of cognitive heterogeneity are present in children with ADHD (11, 12). Evidence indicates only weak correlations between different methods of evaluating EF. Toplak et al. (13, 14), Krieger et Amador-Campos (6), Soto et al. (15), and others have found some modest, primarily non-significant or weak correlations between cognitive tests and rating scales measuring EF. In a meta-analysis conducted by Toplak et al. (14) examining the overall cross-method correlation in 20 individual studies, the overall median correlation between EF test and rating scales was only r=.19.

Toplak et al. (13) find that the rating scale Behavior Rating Inventory of Executive Function (BRIEF) expresses a superior level of predicting ADHD core symptoms. They argue that the cause of the non-significant levels of correlations between cognitive tests and rating scales may be related to the gap in ecological validity, which refers to differences in the assessment methods ability to predict behavior in real-world settings. Toplak et al. (13) argue that the clinical setting of cognitive tests produces poor ecologically valid information compared to information gathered from rating scales which are evaluated by parents and teachers, who experience the behavior of the child in their daily life. Toplak et al. (14) argue that cognitive tests cannot be used to predict performance in goal-directed behaviors in real-life settings, and go on to hypothesize that rating scales, on the other hand, might reflect mental constructs pertaining to the successful pursuit of goals.

Ledochowski, Andrade & Toplak (16) further argue that the implementation of a test design that seeks to emulate naturalistic EF tasks and settings might produce more ecologically valid EF test results. Ledochowski, Andrade & Toplak (16) propose the use of Unstructured Performance-Based Task (UPT), a test design in which the test material and examiner impose minimal structure on the test subject.

Soto et al. (15), on the other hand, argued that the research design of earlier studies, such as Toplak et al. (13, 14), has an impact on the interpretation of these studies. Parent ratings of ADHD symptoms were used to evaluate the predictive qualities of rating scales and cognitive tests, and the outcome was determined by mono-informant and mono-method biases. Soto et al.’s (15) research indicates that:

EF tests outperform EF ratings in predicting academic performance (regarding academic test results defined by The Kaufman Test of Educational Achievement), while EF tests and EF ratings show similar levels of correlation with academic rating results (defined by The Academic Performance Rating Scale).

Based on these complex results, Soto et al. (15) conclude that EF tests and rating scales must reflect different underlying constructs or expressions of EF (cognitive and behavioral aspects). Soto et al. (15) argue that the use of rating scales, which adopt alternative terms and constructs that refer to specific behavioral skills, such as organizational, time management, and planning skills, may help to distinguish the different but related areas of executive function (i.e., cognitive, and behavioral expressions of EF skills).

As presented in previous studies of EF and ADHD, the results showed weak or no correlations between cognitive tasks and ratings of EF in children with ADHD. However, the studies included children with various degrees of EF dysfunctions, as shown children with ADHD show great heterogeneity of cognitive functions. The present study aims to investigate the correlations between EF cognitive tests and parent rating scales in a group of children with ADHD, who also specifically show significant impairments of EF.

Therefore, the current study aims to investigate a clinically meaningful subgroup of children with both ADHD and significant executive dysfunctions defined by high ratings on the Behavior Rating Inventory of Executive Function (BRIEF) organize subscale. Organizational skills are a relatively new area of neurocognitive and clinical research but shows promise as an area of intervention research. The current study employs cognitive tests and rating scales that measure a broad array of EF, and questionnaires designed to measure organizational and planning skills specifically. The study may clarify how best to design and interpret the results when using different modalities to measure EF.

To investigate cross-method correlation between cognitive tests and parent rating scales of EF in children having both ADHD and significant executive dysfunctions in planning and organization skills.

This study is based on baseline data from a randomized controlled trial (RCT) investigating the effect of “Organizational Skills Training for Children with ADHD” (17). The number of participants was determined with a sample size calculation based on the primary outcome of the RCT, for more details please see the protocol (17). The data was collected between June 2017 and November 2022. A child could only participate in the trial if the written consent of all legal guardians was obtained. Danish law requires written consent by the child above the age of 15 years. All participating children gave verbal assent for participation. The trial has been approved by the Danish Data Protection Agency (ID 17/7467) and the Regional Scientific Ethical Committees for Southern Denmark (S-20160180).

The current study investigates a relatively homogeneous sample of children diagnosed with ADHD. All participants were recruited from outpatient clinics in the Southern Region of Denmark in an intervention trial of Organizational Skills training (17) using the following inclusion criteria: All ADHD diagnoses were verified using the Kiddie-Schedule for Affective Disorders and Schizophrenia (K-SADS) diagnostic screening interview (18). All children show an IQ >80 using the Reynolds Intellectual Screening Test (RIAS) (19) and BRIEF parent rating scale scores on or above the clinical cut-off of 60 on the plan/organize subscale. Children with an autism spectrum disorder were excluded from the study. Only data collected at baseline were used in this paper.

Behavior Rating Inventory of Executive Function - BRIEF is a rating scale using a three-point rating system (never/sometimes/often), designed to measure the executive function of children (age 5 to 18). It contains 86 items, eight clinical scales, and is comprised of a parent and a teacher questionnaire. BRIEF’s internal consistency (αs = .80–.98) and test/retest reliability (rs = .82 for parents, .88 for teachers) is good, the inter-tester reliability between parent and teacher rating (r = .32) is, however, low (20,21,22).

Children’s Organizational Skills Scale - COSS (23) is a rating scale designed to measure specific domains in children’s behavioral abilities at home and school life concerning skills in Organization, Time Management, and Planning (OTMP skills). The COSS has 58 items (parents) and 35 (teachers) using a 4-point scale (hardly ever or never/sometimes/much of the time/just about all the time). COSS shows excellent internal consistency alphas for Parent (αs = .98) and Teacher (αs = .97) items alike, as well as test/retest reliability (r = .99 for parents and .94 for teachers). COSS shows good interrater consistency (mean parent to teacher being r = .69, p < .001). The COSS scale furthermore measures specific organizational behavior, in line with Soto et al.’s (15) argument for the need of rating scales that target EF behavior specifically.

Cambridge Neuropsychological Test Automated Battery – CANTAB is a computer-administered test battery containing high levels of precise, sensitive, and objective measurements concerning cognitive functions, and is often used in the research of EF (6, 7, 13). In the present study, the following tests were used: One Touch Stockings of Cambridge (OTS) an executive functions measure addressing planning. The Spatial Working Memory Test (SWM) a nonverbal working memory measure and, the Stop Signal Task (SST) measuring impulsivity.

To take the complex and interconnected structure of EF into account, the present study applies a correlation analysis that encompasses both subscale levels and composite levels of EF – both in respect to the rating scales and cognitive test data. Subscales and composite variables from cognitive test (CANTAB): from the Spatial Working Memory (SWM) test, which measures working memory, the variable SWM Between Search Errors (SW) is applied. From the One Touch Stockings of Cambridge (OTS) test, which measures planning, the variable OTS Problems Solved on First Choice (OT) is applied. From the Stop Signal Task (SST) test, which measures inhibition, the variables SST Stop Signal Reaction Time, SST Directions Errors: Go Trials, and SST Directions Errors: Stop Trials are all weighted and applied as a single measure: SST weighted variable (SS). This was done by adding z-standardized scores of the OTS CANTAB items with weights 2:1:1. A Global cognitive composite score has been created by combining the present subscales into one combined score: CANTAB Composite Total Score (TO). The composite CANTAB TO scale was computed as a sum score of z-standardized OT, SW and SS scores.

From the BRIEF rating scale, the following subscales and composite variables are applied: Global Executive Composite (GE), Emotional Control (EM), Shift (SH), Initiate (IN), Organization of Materials (MA), Inhibition (IH), Working Memory (ME), Monitor (MO), Plan/Organize (PL).

From the COSS rating scale, the following variables are applied: Memory and Materials management (MM), Task Planning (TP), Organized Actions (OA) and Total score (TO).

As primary analyses, pairwise Spearman correlations were calculated between the three composite scales, BRIEF GE, COSS TO, and CANTAB TO, and for the same-variable subscales. In secondary analyses, pairwise Spearman correlations were conducted between all composite and subscales of BRIEF, COSS, and CANTAB. In the interpretation of the secondary analysis, a p-value of .01 indicates a significant finding, to take the multiple testing into account. Further secondary analyses considered pairwise Spearman correlations stratified on child sex, and child ADHD subtype (excluding children with hyperactive subtype due to an insufficient number of cases). The t-scores of COSS scales are only defined for children aged 8 years or above, however, the study sample included 10 children of age 6–7 years. In a sensitivity analysis, children of ages 6–7 years were allocated COSS t-scores corresponding to age 8 years, and pairwise Spearman correlations between all composite and subscales of BRIEF, COSS, and CANTAB were computed, including COSS t-scores of children aged 6–7 years. The analysis found no significant differences between children ages 6–7 and children ages 8–13 on the COSS scale results.

All analyses were performed in STATA 17.0 (Statacorp, Texas, USA).

Descriptive statistics were computed on the participant characteristics: child sex, child age, ADHD subtype. See Table 1 for more details.

Comparing EF cognitive tests and EF rating scales, the results are primarily non-significant and unclear for the entire group of children with ADHD and executive dysfunctions. Results concerning the subgroup of participants with a primary inattentive subtype indicate that the correlation between results from EF tests and EF rating scales are more significant and more negatively associated. Cross-methodological biases such as differences in ecological validity, respondent bias, and differences in the underlying constructs measured by tests and rating scales can play a vital role in explaining the present findings. Further research in these cross-methodological biases is important to better understand the complex nature of EF as both cognition and behavior, how EF is examined in the clinic, and in order to understand and research the potential ways EF is expressed in subgroups of children and adolescents with ADHD.