Further evidence of the Diagnostic Utility of the Child Behavior Checklist for identifying pediatric Bipolar I Disorder

The sample was derived from four independent studies using identical assessment methodology: 1) and 2) were prospective controlled family studies of boys and girls 6 to 17 years of age with and without DSM-III-R ADHD (Boys Study: n = 140 ADHD and n = 120 Controls; Girls Study: n = 140 ADHD and n = 122 Controls) (23, 24); 3) was a prospective controlled family study of youth 10 to 18 years of age with (n = 105) and without (n = 98) DSM-IV pediatric BP-I disorder (25); and 4) was a prospective family study of youth 6 to 17 years of age of both sexes with active symptoms of DSM-IV BP-I disorder (n = 105) (4). The ADHD studies recruited subjects from pediatric and psychiatric clinics. The BP-I disorder studies recruited subjects from referrals to the Clinical and Research Programs in Pediatric Psychopharmacology at the Massachusetts General Hospital and through advertisements in the community. Controls were recruited from pediatric clinics, advertisements to hospital personnel and community newspapers and internet postings. All studies excluded adopted subjects, subjects where the nuclear family was not available, subjects with major sensorimotor handicaps, autism, inadequate command of the English language or full scale IQ < 70 (< 80 for ADHD studies). Potential subjects were also excluded from the ADHD studies if they had a primary psychotic disorder and from the BP-I disorder studies if their BP-I disorder was due solely to a medication reaction. For all four studies, parents provided written informed consent to participate. Children and adolescents provided written assent to participate. The Partners Human Research Committee approved these studies.

In all four studies, psychiatric assessments of subjects were made with the Kiddie Schedule for Affective Disorders – Epidemiologic Version (KSADS-E) (26–28). The K-SADS-E is a semi-structured interview tool used in research that assesses for the presence or absence of past and current symptoms within diagnostic categories. Extensively trained and supervised psychometricians with undergraduate degrees in psychology conducted all interviews. For the ADHD studies and the controlled BP disorder study, raters were blind to the ascertainment status of the families. For the BP disorder family study, raters were blind to the study assignment and whether the subject was a proband or sibling.

Diagnoses were based on independent interviews with parents and direct interviews with children older than 12 years of age. Data were combined such that endorsement of a diagnosis by either reporter resulted in a positive diagnosis. For the sample used in this study, 46.9% (n = 310) of the assessments were based on the parent interview only. For the assessments where parents and children were both interviewed there was agreement between reporters in 30.4% of subjects (n = 201), disagreement between reporters in 4.8% of subjects (n = 32), and no information available on the presence or absence of agreement between reporters for 17.9% of subjects (n = 118). In the subjects where there was disagreement between parents and children, 28 diagnoses of BP disorder were made based on parent report, and four diagnoses of BP disorder were made based on child report.

To assess the reliability of our overall diagnostic procedures, we computed kappa coefficients of agreement by having experienced, blinded, board-certified child and adult psychiatrists and licensed experienced clinical psychologists diagnose subjects from audiotaped interviews made by the assessment staff. Based on 500 assessments from interviews of children and adults, the median kappa coefficient was 0.98 for the ADHD studies and the controlled BP disorder study and 0.99 for the BP disorder Family study.

Socioeconomic status (SES) was measured using the 5-point Hollingshead scale. A higher score indicates being of lower socioeconomic status (29).

The parent of each participant completed the 1991 version of the CBCL for ages 4 to 18 years. The CBCL queries the parent about the child’s behavior in the past six months and aggregates this data into behavioral problem t-scores (8, 30). A computer program calculates the t-scores for each scale. Raw scores are converted to gender and age standardized scores (t-scores having a mean of 50 and SD of 10). A minimum t-score of 50 is assigned to scores that fall at percentiles of ≤ 50 on the syndrome scales to permit comparison of standardized scores across scales. t-scores above 70 (2 SD) indicate clinical disorder. Subscales include Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, Rule-Breaking Behavior and Aggressive Behavior.

We first compared demographic characteristics between subjects with and without BP-I disorder in each of the studies separately and then in the combined sample using Student’s t-test for continuous outcomes, Pearson’s χ² test or Fisher’s exact test (for expected counts < 5) for binary outcomes and Wilcoxon rank-sum test for ordinal outcomes. Next, we calculated conditional probabilities for each of the studies separately using a conservative cut-off point of ≥ 180 for the CBCL-BP profile. We subsequently combined the data from the four studies and used receiver operating characteristic (ROC) curves to examine the ability of the CBCL-BP profile to identify those with and without a structured interview diagnosis of BP-I disorder. ROC analysis uses each value across the entire range of scores for the CBCL-BP profile as the cutoff for defining a case and compares this classification to the “true” diagnosis, as defined by the clinical interview. The ROC analysis then plots the false positive rate (1-specificity) and the true positive rate (sensitivity) for each value of the CBCL-BP profile on the x- and y-axis, respectively, to create the ROC curve. Starting in the lower right-hand corner of the plot, each successive point corresponds to an increase in one point of the CBCL-BP profile. ROC analysis summarizes diagnostic efficiency with the AUC statistic. An AUC of 0.5 means the test does not predict the disorder in any way and an AUC of 1.0 means the test predicts the disorder perfectly. We used conditional probabilities to examine the diagnostic utility of various cutoff points. For each cutoff, we calculated sensitivity, specificity, PPV, NPV and the percent correctly classified.

Furthermore, using the same methods as described above, we performed a sensitivity analysis in a restricted sample consisting of subjects for whom there was no disagreement regarding the presence or absence of BP-I disorder (n = 511). The restricted sample included subjects where the K-SADS-E assessment was based on parent interview only and subjects where there was agreement between parent and child report.

All analyses were performed using Stata^® (Version 14).

Subjects from the four original independent samples were only included in this sample if a CBCL was completed. Table 1 shows the demographic details from the four contributing samples. In examining differences between children with and without BP-I disorder in the controlled family study of BP-I disorder, children with BP-I disorder were more likely to be Caucasian (p ≤ 0.001). There were no meaningful differences between children with and without BP-I disorder in the ADHD studies. In the combined group that included the four contributing samples, children with BP-I disorder were more likely to be male (p = 0.002). No other meaningful differences were found within the combined sample in age, socioeconomic status or race.

As shown in Table 2, similar values of sensitivity, specificity and percent correctly classified with BP-I disorder were observed in each individual study using the conservative cut-off point of ≥ 180 for the CBCL-BP profile (Table 2).

Since all the studies used nearly identical methodology and assessments and had similar conditional probability analysis results, we combined data from the four samples for this analysis to improve statistical power. Thus, our combined sample consisted of 661 subjects, of which 130 (19.7%) had BP-I disorder. Figure 1 depicts the combined t-scores from the four studies for the CBCL-BP profile that yielded an AUC of 0.91.

FIGURE 1.

Examination of the performance of specific cut off t-scores that correspond to 0.5 SD increases on the CBCL-BP profile to correctly identify subjects with a diagnosis of BP-I disorder showed that a cut-point of 195 had the best properties as determined by the AUC with 80% sensitivity, 87% specificity, 61% PPV, 95% NPV, and 86% correctly classified with BP-I disorder (Table 3).

We found similar findings in our sensitivity analysis using the restricted sample as we did in the full sample. In the combined sample there were 87 subjects (17%) who had BP-I disorder. There was no significant difference (p = 0.06) in mean age between subjects with BP-I disorder (10.3 ± 3.5 years) and those without BP-I disorder (11.0 ± 3.2 years). The ROC analysis of the CBCL-BP profile yielded an AUC of 0.93. A t-score of ≥ 195 on the CBCL-BP profile was the best cut-off score and correctly classified 87% of subjects with BP-I disorder with 86% sensitivity, 87% specificity, 58% PPV and 97% NPV.