Supplementary scales for the school-age forms of the Achenbach System of Empirically Based Assessment rated by adolescents, parents, and teachers: Psychometric properties in German samples

The Achenbach System of Empirically Based Assessment (ASEBA) represents the most widely used and cross-culturally established rating scales internationally (3–4). The questionnaires for school-age children and adolescents are also available in a German version for all three informant perspectives: parent report (CBCL/6-18R), youth self-report (YSR/11-18R), and teacher report (TRF/6-18R) (1). These are well established in Germany in child- and adolescent psychiatry as well as psychotherapy for inpatient and outpatient facilities and practices, recommended as basic instruments for broadband multimodal diagnostic of behavioral and emotional problems (2). Factor analyses generated eight empirically based cross-informant problem scales (Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, Rule-Breaking Behavior, and Aggressive Behavior) (4,5,6,7). Additionally, six DSM-oriented scales were developed, which are based on the Diagnostic and Statistical Manual of Mental Disorders (DSM-5: Depressive Problems, Anxiety Problems, Somatic Problems, Attention Deficit/Hyperactivity Problems, Oppositional Defiant Problems, and Conduct Problems (8)). The scale Attention Deficit/Hyperactivity Problems in the teacher report version is further divided into two subscales: Inattention and Hyperactivity-Impulsivity (3). Expanding on these “2001 scales”, Achenbach and Rescorla integrated the following four supplementary “2007 scales” into their Multicultural Supplement to the Manual for the ASEBA School-Age Forms & Profiles (9): Based on previous findings, Achenbach & Rescorla proposed a 14-item scale as a supplement to the CBCL, initially named Posttraumatic Stress Problems scale (PTSP) (9,10,11) and renamed as the Stress Problems Scale (SPS) by Achenbach, Rescorla & Ivanova in 2015 (9, 10) to reflect that the scale measures general stress rather than specific posttraumatic stress symptoms (12, 13). Furthermore, a CBCL Obsessive-Compulsive Problems scale (OCP) (8, 13) and a TRF Sluggish Cognitive Tempo scale (SCT)(9, 15) were developed and implemented in the Multicultural Supplement to the Manual for the ASEBA School-Age Forms & Profiles (9). Besides these problem-oriented scales, a fourth “2007 scale” was generated: the YSR Positive Qualities Scale (PQS) (9). The PQS covers 14 positive items from the YSR (9), with significantly higher scores being found in girls compared to boys and in older compared to younger adolescents (16). More recently, other research groups have developed the following additional supplementary scales, which have not yet been integrated into the ASEBA family: CBCL and TRF Dysregulation Scale (DRS1) (17,18) as well as a weighted and rounded Dysregulation Scale (DRS2) (19), different types of CBCL and TRF Autism Spectrum Disorder scales (20, 22) and the CBCL Mania Scale (23).

In 1995, a dysregulation profile was identified, which is characterized by high scores on the following rating scales of the 1991 CBCL version: Aggressive Behavior, Anxious/Depressed, and Attention Problems (18). Other researchers (17) calculated a sum score from the 41 items of these three subscales (2001 version of the CBCL). Rescorla et al. (24) conducted a cross-cultural comparison of the sum score of the dysregulation profile and found mostly minor differences between gender and age across informants. Based on the previous research, in 2018, McQuillan et al. (19) suggested a weighted and rounded score in order to improve the conceptual precision.

Furthermore, two data-driven specific autism spectrum disorder (ASD) scales were constructed - a 9-item CBCL Autism spectrum disorder scale (ASD1) (20) and a 10-item TRF Autism spectrum disorder scale (ASD2) (21) each developed both in community samples and in heterogeneous clinical samples. The two scales differ in three items. More recently, a third, data-driven ASD scale with 15 items was developed, which overlaps in six items with the ASD1 and in nine items with the ASD2 (22).

Overall, in most analyses, these supplementary scales have shown at least acceptable internal consistencies (≥.70) for the informant perspectives analyzed in the respective studies (9, 13, 14, 19, 21,22,23, 25,26,27,28,29,30). Moreover, validity has been demonstrated for several of the scales. While low correlations between the respective informants were reported for SCT (CBCL and YSR) and OCP, correlations were moderate for SPS (r = .38) and even stronger for DRS (r>.50) (9, 24). Further validity analyses revealed that very high SPS scores in the YSR were associated with a moderate increase in the likelihood of a posttraumatic stress disorder diagnosis (25). Additionally, children and adolescents with a diagnosis of obsessive-compulsive disorder obtained significantly higher scores on the OCP scale compared to a psychiatric control sample and a population control sample (14).

To the best of our knowledge, no study has been conducted examining the supplementary scales of the Achenbach system for school-age across informants in a community sample, a clinical sample, and disorder-specific subsamples for aspects of their convergent/divergent and discriminant validity as well as their internal consistencies. This is the aim of the current analyses. In view of the already confirmed intercultural reliability of the instruments, we hope our analyses may contribute to the canon of results on these instruments. Interculturally validated instruments enable the transferability of results of both epidemiological and clinical-interventional studies to the target group. In addition, such instruments can also be used reliably for patients with a migration background, who in modern societies have their natural share among those seeking advice from help facilities.

The present study analyzed the supplementary scales of the German school-age versions of the Achenbach questionnaires regarding internal consistency and aspects of validity across informants and samples. The discussion focuses first on the scales that showed at least adequate internal consistency (SPS, DRS, MAS, PQS, ASD3) and then on the scales which do not show this cross-informantly. As our community sample was part of the multicultural analysis by Achenbach & Rescorla, our findings in the CBCL and YSR show similarity to the authors’ findings concerning the 2007 scales (SPS, OCP, SCT, PQS) (9).

The internal consistency of the Stress Problems Scale (SPS) across CBCL and YSR and samples of different origin was at least adequate (Ω>.70) with a narrow exception in TRF (Ω=.69). This is in line with previous multicultural findings reporting Cronbach’s α in the community sample for all three informant types (9) as well as with the results of You et al. (25) regarding their TRF (α=.79) and the YSR (α=.85) community sample. Moreover, our results also revealed adequate internal consistencies in the clinical sample across parents and self reports, whereas Zandberg et al. (13) found a somewhat lower internal consistency (α=.68) in self-reports in a group of 14–19-year-old girls with a primary diagnosis of posttraumatic stress disorder. Cross-informant correlations in the community sample were higher than in the multicultural analysis by Achenbach & Rescorla (9), who reported low or no correlations between parents and teachers as well as between teachers and adolescents in a multicultural community sample (9), as was the case in our clinical sample. Correlations for INT and EXT were high across all informants, and the mean difference in the SPS between the clinical and community sample was the highest of all supplementary scales (dCBCL = 1.75; dYSR = 1.03).

Both versions of the Dysregulation Scale (DRS1 and DRS2) achieved good (Ω≥.80) to excellent (Ω≥.90) internal consistencies, which was consistent with previous findings in a community sample reported by McQuillan et al. (19). However, upon closer inspection, it emerged that some items, such as Fears, Fears school, or Fearful, did not reach rit>.30 across informants. Furthermore, the item-total correlation indicated higher values for the YSR and particularly for items describing covert behavior (Cries a lot, Dreams, Feels guilty, Talks about suicide, Worries). While Rescorla et al. found minor effects of age and gender in a CBCL and YSR community sample (36), we confirmed low effects (if at all) in the clinical samples. For the TRF, boys scored slightly higher than girls, which is in line with the US findings (23). Across the CBCL and YSR samples, high correlations with EXT as well as INT emerged, with higher values for EXT, whereas TRF correlations with INT were only in the medium range for clinical ratings. Cross-informant correlations were at least moderate, with the exception of the TRF with the YSR. Discriminant validity was demonstrated across informants. All in all, both DRS performed good concerning internal consistency and the aspects of validity analyzed, with only minor differences between the unweighted and the weighted, rounded scale. Nevertheless, in the interest of parsimony, a reduced scale would be advantageous, especially since a look at the item-related analyses support this hypothesis. One possible approach would be a data-driven procedure, comparable with the optimizing of the ASD3 (22). Another approach could be the strategy used by Evans et al. (38) analyzing a selected theoretically item-set based on previous research by confirmatory factor analyses.

The internal consistencies of the Mania Scale (MAS) are at least adequate (Ω≥.80). But our results were lower than in previous studies (23, 30), though we were able to confirm the internal consistencies for a broader age range and across genders. Across the informants and samples, high correlations with EXT were found, whereas high correlations with INT were only found in the community samples. Cross-informant correlations were at least moderate, except for the TRF with the YSR. Mean differences between the clinical and community sample were large for the CBCL and medium for the YSR.

In the community sample, the YSR-specific scale Positive Qualities Scale (PQS) showed good internal consistency, in line with the findings of Achenbach & Rescorla (9), the clinical sample it was adequate. The correlations with INT as well as EXT, as far as significant, were at least low, due to a low variance of PQS. While no linear correlation between PQS and INT was found in the clinical sample, positive correlations with both EXT and INT were found in the community sample. PQS may therefore be interpreted as an enriching concept because the positive scale stands out from all other - problem-oriented - scales. On the item level, a need for further research seems to be apparent, as some items had a rather low item-total correlation. In contrast to Rescorla et al. (16), PQS was only slightly, if at all, correlated with gender. Age effects within the community sample were small and in line with the literature (16). The effects were also small in our clinical sample, but in the opposite direction. The discriminant validity was rather low, with the clinical sample unexpectedly achieving slightly higher values.

Of the three Autism Spectrum Disorder scales (ASD), Offermans et al.’s (22).data-driven ASD3 showed at least adequate internal consistencies across all samples and informants, supporting and extending the findings of the original research group. Although the ASD3 was originally developed based on parent reports, we found even better internal consistency for the TRF. The analyses of convergent/divergent validity for all samples and perspectives concerning ASD resulted in high correlations between all three scales and INT. Additionally, ASD2 and ASD3 correlated highly with EXT, but correlations with INT were significantly higher in all cases. Moreover, cross-informant correlations were consistently moderate, with the exception of the TRF with the YSR. Discriminant validity was proven for all ASD scales, whereas convergent/divergent validity was high for parent and teacher reports but at least low for self-reports. Gender and age effects were only slightly pronounced, if at all.

With regard to internal consistency, the Obsessive-Compulsive Problems scale (OCP) showed predominantly weak internal consistency in the community sample, which was in line with the findings of Achenbach & Rescorla (9) but contrary to those of Nelson et al. (14) and Geller et al. (26). In the present study, OCP only in the clinical YSR sample achieved an adequate internal consistency. Therefore, the further results should only be interpreted as meaningful for this informant perspective. The high correlation of OCP with INT was significantly higher than that with EXT, in accordance with clinical expectation. The cross-informant correlations between the CBCL and YSR in the clinical and community samples were at best moderate, although higher overall compared to Achenbach & Rescorla (9). This is unsurprising from a clinical perspective, as OCP symptoms are unlikely to be fully communicated by youth, and are more likely to be hidden. Discriminant validity was given across informants: Large mean differences were found between the respective nearest diagnosis group and the remaining clinical sample with the CBCL and YSR, and medium ones with the TRF. The correlation analyses of OCP with gender and age revealed only one moderate effect, which was a gender effect in the clinical YSR sample.

For the Sluggish Cognitive Tempo scale (SCT), internal consistencies were good in the clinical TRF sample (Ω=.81) but poor in both CBCL samples. This is in line with the results of Achenbach and Rescorla (8), while Bauermeister et al. reported good internal consistencies for their clinical CBCL and TRF samples (9, 27)). Furthermore, Achenbach and Rescorla (9) found adequate internal consistencies for TRF in a community sample (not available for our presented analyses). Since the SCT was developed based on the TRF, our results support the assumption that this scale might be more, if not exclusively, suitable for the TRF. As also reported by Bauermeister et al. (39), we detected strong correlations between SCT and INT in our clinical sample. Despite higher item overlaps between SCT and EXT, the correlations were low, and indeed significantly lower than those with INT and not even significant for the TRF (clinical sample). Becker et al. (28) likewise found no significant correlations of SCT with the DSM-oriented scales Attention Deficit/Hyperactivity Problems or Oppositional Defiant Problems in the CBCL. However, our low to moderate results should be interpreted with caution due to poor internal consistency. In contrast, in our TRF sample, SCT did not even correlate with any Hyperactivity-Impulsivity scale, with Aggressive Behavior, or with the DSM-oriented scale Oppositional Defiant Problems.

Surprisingly, the following scales correlated notably strongly with SCT across the samples: Withdrawn/depressed (one item in common), the DSM-oriented scale Depressive Problems (two items in common), as well as moderately with the DSM-oriented scale Inattention (no items in common) in the TRF. On the one hand, the strong correlation between SCT and Depressive Problems might imply that daydreaming from SCT is associated with rumination (28, 40); a moderate correlation with the dimension Inattention may thus also be rooted in this. On the other hand, the strong correlation between SCT and Withdrawn/depressed can be explained by the associated social withdrawal (15, 28, 40). In view of the lack of the internal consistency of the CBCL scale, the moderate correlation between the CBCL and TRF and the large effect of discriminant validity for the CBCL should be interpreted with caution.

The Mania Scale and the Autism Spectrum Disorder scale according to Offermans et al. (22) can be recommended for clinical use. Inspired by Offermans et al.’s successful strategy and in the interest of parsimony, instead of using the current versions of the Dysregulation Scale, we suggest a new approach of data-driven analyses with the aim of focusing on the most relevant items in terms of content. Construct and discriminant validity suggest the Stress Problems Scale as a higher-order dimension. If at all, the Obsessive-Compulsive Problems scale should only be used in self-assessment, while valid and reliable results on the Sluggish Cognitive Temposcale are more likely in teacher assessment. The Positive Qualities scale seems to be an enriching concept but needs further research.