Discriminability of the Beck Depression Inventory and its Abbreviations in an Adolescent Psychiatric Sample

The data for the current investigation were derived from the ongoing REAL-SMART project (“Recognition and early intervention for alcohol and substance abuse in adolescence and systemic metabolic alterations related to different psychiatric disease categories in adolescent outpatients”). The participants were referred patients aged 13–20 years who attended the adolescent psychiatric outpatient clinic of Kuopio University Hospital (KUH) in Finland between June 2017 and March 2022, with breaks in clinic operations due to the COVID-19 pandemic. The reasons for non-participation were not recorded but included declining to participate, dropping out or being transferred to inpatient treatment before being approached, and not being presented with the study by clinical personnel for various reasons. Of the adolescents having at least one appointment in the outpatient clinic (n = 2853), a total of 754 (26.4%) participants were enrolled in the study, of whom the majority (73 %) were female, which is a typical gender distribution for these clinics. Previous or current diagnoses did not affect recruitment. Two patients were excluded for not filling in the BDI-IA at baseline.

The participants were first interviewed and then autonomously completed a multi-measure questionnaire containing the BDI-IA on a tablet computer later in the same session. All items were presented one at a time, with automatic advancement after giving a response. Although it was possible to return to change earlier responses by swiping right on the screen, this functionality was not advertised, and only a few respondents used it for only a few responses.

Prior to undertaking the investigation, the Research Ethics Committee of Kuopio University Hospital confirmed the study procedures and written informed consent was obtained from all the participants. In addition, the study complied with the ethical principles set by the 7th revision of the Declaration of Helsinki (30).

To enhance the transparency of the present research, we adhered to the STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies) checklist (Supplementary material Table S1), which consists of 30 items (31).

All statistical analyses were performed in the R software environment (version 4.4.0) (37). When calculating BDI sum scores, missing BDI responses were replaced with the participant’s mean for the other responses of the respective form version. Overall, missingness was minimal (0.5% of all responses).

For the nonparametric comparison of distributions, we used the stochastic superiority index, p ̂ = (X > Y) + 0.5 × (X = Y), also known as the common language effect size, which indicates the probability that a random member of subgroup X has a higher score than a random member of subgroup Y. To test the statistical significance of p, we used the appropriate Brunner–Munzel test (38,39), implemented in the brunnermunzel R package (version 2.0) (40). All statistical tests were considered significant at p < 0.05 unless stated otherwise.

The sample was not collected specifically to test the BDI's diagnostic accuracy, and this study was conceived after the data had been gathered. Thus, data collection was not informed by the present study's needs. However, previous studies on the single-test accuracy of the BDI (26) had a median sample size of 316, indicating that our study was adequately powered (see Table S2 for detailed information).

We estimated the required sample size for comparing two binary diagnostic tests (the pre-specified cut-off applied to two BDI versions) in a paired design with the calculations suggested by Akoglu (2022) (41), with a Type I error rate (α) of 5%, 95% power, and conservatively applying Yates’ continuity correction. For detecting a clinically meaningful difference of .05 in either sensitivity or specificity between versions, the required sample size is 249 when the test agreement is at maximum and 1908 at minimum. Since the compared BDI versions were various summations of nearly the same responses, test agreement can be expected to be close to the maximum, and the present sample size was, therefore, more than sufficient.

As we secondarily report differences in BDI distributions between the depressed and non-depressed groups, we also estimated the required sample size for these comparisons. We are not aware of a specific power calculation formula for the Brunner–Munzel test, but it is at least as powerful as the Wilcoxon–Mann–Whitney U-test in most situations (39). We, therefore, estimated the two-sided power of the U-test with the formulas of Noether (1987) (42), as implemented in the rankFD package (version 0.1.1) (43) for R. At an α of 5% and 95% power, detecting even a modest effect of p ̂ = 0.6 required only 217 participants each in the two compared groups. The sample was thus clearly sufficient, even though the estimation was slightly optimistic in not accounting for ties.

The latent structure of the BDI-IA was investigated with a confirmatory factor analysis of the a priori single-dimensional model, treating the items as ordinal (DWLS estimator), with the cfa function in the lavaan package (version 0.6-17) (44), using all pairwise-available data and otherwise standard settings. Factor scores were computed with the lavPredict function using the default empirical Bayes modal approach. Factor scores for the abbreviated forms were calculated using the parameters from the full model, thus keeping parameters identical across the form versions and conceptually treating missing items in an abbreviated version as missing responses of the full BDI.

Basic accuracy ratios, Cohen’s kappa (κ), the diagnostic odds ratio (DOR), and the number needed to diagnose (NND; reciprocal of the Pierce skill score, a.k.a. Youden’s J index) for each BDI version in detecting gold-standard depression diagnoses were computed with the epi.tests function of the epiR package (version 2.0.74) (45). The NND can here be interpreted as the number of individuals that need to be screened to correctly detect one person with a diagnosis (46). The primary full-length BDI cut-off of 0.76 for these analyses was the mean score equivalent to a sum of 16 or greater, as described above, and the secondary mean score cut-off was 0.52. For the abbreviated forms, apparent prevalences were higher at these cut-offs due to higher scores for the included items. For comparability, we therefore matched cut-offs with the closest equivalent in sensitivity; when two cut-off candidates were equal in their sensitivity difference, the cut-off with the smallest difference in specificity was selected.

To compare the diagnostic performance of the BDI-IA with the BDI-SF, BDI-PC, and BDI-6 in a paired design, the discriminability of the mean scores of the shortened forms was compared with those of the full 21-item form using the statistical procedure of Roldán-Nofuentes (2020) (47) implemented in the R software package testCompareR (version 1.0.3) (48). First, a global test jointly compared sensitivity and specificity for detecting the gold-standard diagnosis, and if this test was statistically significant, sensitivity and specificity differences were tested separately. Multiple tests within the same form pair were corrected with Holm’s method, the package default. Due to cut-off matching by sensitivity, this procedure mainly compared specificity but considered the sensitivity discrepancies arising from imperfect matching due to coarse sum score distributions.

As sum scores have a simpler measurement model, which is potentially less accurate than factor scores, we compared the diagnostic categorization by factor scores with categorization by the full BDI-IA sum score, in the same manner as the comparisons with the mean scores of the abbreviations. Again, the factor score cut-off was selected to match the sensitivity of the sum score cut-off, and accuracy was compared with testCompareR as above. The strength of association between the binary categorizations by mean scores and factor scores was expressed as a correlation (equivalent to Stuart’s τc).

As an exploratory addition to the main results, we determined sample-optimal cut-offs for all the BDI versions using the R package cutpointr (version 1.1.2) (49) with the bootstrapped version of the cutpointr function using default settings, the chosen cut-off being validated in 1000 out-of-bag samples.

The main criterion for comparison was Cohen’s unweighted κ, estimated jointly and for genders separately, with secondary criteria being misclassification cost with a) false positives and false negatives being weighted equally or b) with triple weight on the latter. Discriminability across the whole score range of each form was assessed with receiver operating characteristic (ROC) curves and accompanying area under the curve (AUC) values, as recommended in the STARD 2015 guidelines (50). For details of the interpretation of AUC values, see Mallet et al. (2012) (51).

The fit of the one-dimensional factor model of the BDI-IA was sufficient, as the scaled comparative fit index (CFI) was 0.949, the scaled root mean square error of approximation (RMSEA) was 0.089, and the standardized root mean square residual (SRMR) was 0.060. Items 19 Weight loss and 20 Somatic preoccupation had the weakest factor loadings (0.24 and 0.42, respectively), while the core items 1, 3, 4, 5, 7, and 8 pertaining to depressive mood and negative self-view had loadings of 0.80 or above. Thresholds (standard scores corresponding to the cumulative response probability within an item) had a wide range: for example, the highest score of 3 on item 10 Crying was as frequent as the score of 1 on item 19 Weight loss. See the standardized parameters of the factor model in Table 3 and BDI factor score distributions by diagnostic category in Table S3.

The main screening discriminability results are presented in Table 4 and Table 5. The primary cut-off (16 or higher on the full BDI) turned out to be relatively low in this highly symptomatic sample, with a sensitivity of 0.94 but a specificity of only 0.64. In the global test, the mean scores of all the abbreviated versions were as good at discriminating between MDD and no depression as the full BDI-IA. Although the differences were not statistically significant, the BDI-6 had a higher negative predictive value (NPV) than the BDI-IA at the same positive predictive value (PPV), which was also reflected in having the lowest NND and a higher DOR. The 7-item BDI-PC was the only abbreviation to have a lower κ and worse NND than the BDI-IA, although, again, these differences were not statistically significant, and the DOR was the highest of all the versions. These differences were largely attributable to a higher threshold due to a lack of perfectly matched cut-off values. The 13-item BDI-SF performed nearly identically to the BDI-IA. The results of the robustness analysis using the two alternate diagnostic categorizations were largely the same (Tables S4 and S5).

At the secondary cut-off of 11 or higher, the diagnostic accuracy of the form versions did not differ, probably due to the sensitivities being extreme (0.98 for the BDI-IA) at such a low cut-off (Tables S6 and S7). ROC curves for the BDI-IA version are displayed in Figures 2a and 2b, and all the form versions are provided in Supporting information Figure S1.

FIGURE 2a.

Sample-optimal cut-offs for the main diagnostic condition, MDD vs. no depression diagnosis, criterion Cohen's kappa, all participants together.

FIGURE 2b.

Sample-optimal cut-offs for the main diagnostic condition, MDD vs. no depression diagnosis, criterion Cohen's kappa, by gender.

Factor scores were extremely strongly correlated (r 0.98 to 0.99) with mean scores in all BDI versions (Figures 3 and S2). Screening assignments were also highly similar (r 0.96 to 0.98 in the primary condition), with only 1–1.5% of respondents classified differently with the two methods (Table S8). Factor scores did not differ in discriminability from the BDI-IA sum/mean scores.

Figure 3.

BDI mean scores versus factor scores with fit lines. Note: The higher and lower horizontal cut-off lines correspond to BDI-IA sums 16 and 11, respectively, and the factor score cut-offs to the matched sensitivities.

The BDI cut-offs optimized for agreement with diagnoses (κ) in the whole sample were slightly higher than the primary a priori value and approximately corresponded for the full BDI-IA to a sum score of 19 or greater (Figure 2a). This difference was mostly due to females, as the value was 17½ when estimated in the male subsample (Figure 2b). When the optimization criterion was an equal misclassification cost for false positives and negatives, the BDI-IA sum score equivalent was 18, and when false negatives were deemed three times as costly, this cut-off was approximately 13 (Figure S3). As with matched cut-offs, optimized cut-offs were higher for the abbreviated versions due to the higher scores for the included items than the excluded ones (Table 6).

Our study had several strengths. The sample was representative of patients in the adolescent patient population, thus enabling the generalizability of the findings to adolescent outpatient psychiatric care. Sensitivity and specificity are independent of prevalence, and the high values suggest that the abbreviated questionnaires are also suitable for screening in primary care. The use of gold-standard research diagnoses over register diagnoses maximized the reliability of the screening reference. In addition, our diagnostically naturalistic and heterogeneous sample was relatively large, making it sufficient in statistical power. Moreover, a paired comparative diagnostic accuracy design was used, providing greater statistical power than unpaired designs.

Regarding the limitations of this study, although SCID is widely used in clinical settings, including in Finland, and a portion of our sample consisted of older adolescents (i.e., ages 17–20), we acknowledge that this instrument may not adequately capture the developmental nuances, particularly in younger adolescents. Furthermore, the BDI-IA, being an older version, may have limitations, such as failing to differentiate between increases and decreases in depression-related vegetative symptoms (i.e., sleep and appetite) (14), despite its high correlation with the BDI-II (15). However, recent psychometric studies continue to support the utility of older versions of the BDI (56,57).

Additionally, our results naturally depend on the employed definition of depression and its operationalization as diagnostic criteria. However, there is an ongoing debate on these definitions, etiology, and the relevant pathological period. Thus, if the diagnosis had a greater emphasis on somatic symptoms, the full BDI might prove superior. Moreover, according to Finnish treatment guidelines, people with mild depression should be treated in primary health care. However, there were a few participants with mild depression in our sample who might have been referred to the outpatient clinic not primarily because of their depression but due to the presence of other comorbid conditions. Finally, although sensitivity and specificity are, in principle, independent of prevalence, our findings cannot necessarily be generalized to primary health care or general population samples. Future studies replicating our results in primary healthcare settings are therefore recommended.