Mind the Gap: A Retrospective Study of Discrepancies in Self-Reported and Administrative Database-Identified Mental Health Issues in Slovenia

Mental health is an area of growing public health importance. Despite successful global efforts in reducing the burden of other non-communicable diseases, the burden of mental disorders and self-harm behaviour remains stable (1). The persistent burden of mental disorders and self-harm underscores the importance of valid and reliable population prevalence data for public health decision-making.

Population surveys and health administrative databases are common sources for mental disorder prevalence data (2). Existing research indicates significant differences in the populations identified as having mental disorders by surveys and those identified through health administrative databases (3,4,5,6). For example, studies show substantial discordance between data sources: Edwards et al. (3) found only 19.4% agreement between survey and administrative diagnoses of mood and anxiety disorders in Ontario. Davis et al. (5) reported limited agreement (Cohen’s kappa 0.24–0.46) even between different self-report methods in the UK Biobank. While O’Donnell et al.’s (6) trend analysis in Canada noted a convergence over time, administrative prevalence remained consistently higher than self-report. Even when self-report is used in online registries, Sordo Vieira et al. (4) found that while it can indicate general psychopathology, in-depth interviews are still needed for specific disorder identification.

However, this evidence is limited to a small number of studies, each applying a different methodology. These discrepancies may result due to variations in population coverage, types of health services included, classifications used to register data and classification algorithms or the way mental health disorder is defined (7). In this study, we use the term ‘mental health issues’ to encompass a broad range of mental health experiences, including diagnosable mental disorders (as defined by clinical criteria) as well as subthreshold manifestations and transient distress that may not meet diagnostic criteria but still impact wellbeing. This distinction is important because it allows us to examine the full spectrum of mental health in the population, rather than focusing solely on diagnosed conditions.

With the aim of assessing the discrepancies between self-reported and administrative data sources in identifying mental health issues in Slovenia, the objectives of the present study were: to assess the discrepancies in self-reported and database-identified groups of cases with mental health issues in Slovenia, linking data from a large population-based survey and three health administrative databases; and to gain insight into which socio-demographic factors contribute to the identification of a case in either of the data sources or in both of them.

The study was designed as a retrospective study using data from the 2019 Slovenian European Health Interview Survey (EHIS) (8), conducted by the National Institute of Public Health (NIPH), and from three national health administrative databases, also maintained by the NIPH. This approach allowed us to compare individual-level data on mental health service utilisation with self-reported mental health status, providing a comprehensive assessment of the agreement between these data sources.

The analysis revealed pronounced disparities in mental health issue prevalence estimates obtained from self-reported data compared to health administrative databases, and only a minority of individuals identified as experiencing mental health issues were found in both data sources.

This finding highlights a significant measurement challenge in public mental health research. Nearly a third of individuals self-reported mental health issues but were not identified in the administrative data sources, while almost half that were identified in the administrative data did not report mental health issues in the EHIS. The discordance in two case groups derived from administrative and survey data sources was expected, as it was already identified in previous similar research (5,6,7,8).

What separates our research from previous studies is that we have not focused on specific mental health disorders (i.e. mood and anxiety disorders), yet we have still found a substantial part of the population using mental health services while not self-reporting any mental issues in the same time window. While administrative data is generally expected to underestimate the prevalence of mental health conditions, our finding of a higher prevalence in administrative data compared to the EHIS survey, consistent with observations from a similar study (6), underscores the complexity of mental health measurement and the need to further investigate the factors contributing to this discrepancy.

The sizeable group of individuals identified in the healthcare databases yet not self-reporting mental health issues in the EHIS presents an intriguing facet of the measurement discrepancy. We found higher age is associated with being identified as having mental health issues in administrative data sources. The majority of cases identified from administrative databases were captured in OPD. A study on antidepressant prescribing trends in Slovenia showed substantial differences in the prevalence of antidepressant recipients between those aged 54 or lower and those older than 55 years, with prevalence reaching 20% in individuals aged 80 and over (16). Even though neither the referenced or present study delves into the interpretation of possible causes of high consumption of psychoactive drugs among the elderly, the results indicate higher healthcare usage with increasing age. In parallel, older adults are less likely to recognise mental health issues as such, (17) and might be treated for mental health symptoms caused by physical illness, therefore not identifying as having mental health issues in the surveys, because the primary source of poor mental health is associated with a physical illness (18). They could also be unaware of the connection between their symptoms and a diagnosable condition, or this information might not have been effectively communicated to them by healthcare providers. Consequently, while their conditions are reflected in administrative data, their self-reports may not include mental health concerns. Recall bias could also influence this discrepancy (19). Participants in the EHIS might have difficulty remembering experiencing mental health issues or they might underestimate their past negative experience in the function of coping, although research indicates the latter might not be a common phenomenon (20). Lastly, we have to consider stigma as an important factor in using healthcare services and self-reporting mental health issues. Stigma appears to have a greater impact on self-reporting of mental health issues in surveys compared to the actual usage of mental health care services (21,22,23). Individuals may be more willing to seek professional help than to disclose mental health problems. Individuals with higher education were less likely to be identified only in administrative data, potentially suggesting a greater comfort level with self-disclosure of mental health issues in this population group. While our study design cannot definitively measure the impact of stigma, these findings hint at a potential interplay between stigma and data source discrepancies. Higher prevalence observed in administrative data might appear counterintuitive, however it highlights the potential influence of prescription drug data and the varying pathways individuals take to access mental healthcare. This underscores the importance of using multiple data sources to capture a more complete picture of mental health in the population.

It is also important to note that while the overall prevalence of mental health issues was higher in the administrative databases, a subset of individuals reported these issues in the EHIS survey but did not appear in the administrative data. For these individuals, stigmatising attitudes towards help-seeking behaviour, such as negative expectations of professional help or fear of being stigmatised for having a mental disorder, may be a significant barrier to seeking help, as documented in previous research (24, 25). One key factor is likely the individual’s perception of their need for healthcare services. Individuals who do not perceive a need for professional help, perhaps due to mild symptoms, minimal functional impairment, or other personal beliefs, are less likely to engage with the healthcare system and thus be recorded in administrative databases. Some individuals might seek help outside of the traditional healthcare system, through alternative therapies or support groups, which would not be reflected in administrative data (17). However, these individuals may still acknowledge and report experiencing mental health issues in surveys. Furthermore, limitations inherent in the health data collection methods themselves might play a role. Incomplete data coverage within the administrative databases is a possibility. For example, individuals using outpatient services without claiming psychoactive medications were not captured in the administrative data sources we used in this study. Additionally, under-diagnosis by healthcare providers could contribute to the discrepancy. Stigma surrounding mental health or a lack of awareness among providers could lead to missed diagnoses, resulting in individuals experiencing mental health issues but not being recorded in administrative databases (26, 27). Also, younger participants of the EHIS survey, students specifically, were less likely to be identified only in administrative and more likely to be identified in only survey data sources. This could be explained by a number of factors previously researched within this age group, such as preference towards informal sources of help and favouring counselling compared to psychiatric or psychological services, (17) or even interpreting and reporting milder forms of distress as mental health issues (28).

This study has some limitations. First, the administrative databases used capture only a part of mental health service utilisation within the formal healthcare system, potentially underestimating service use. Additionally, the accuracy of administrative data relies on consistent and reliable coding practices. Second, the EHIS mental health assessment used in this study is less structured than instruments such as the Composite International Diagnostic Interview and is not commonly used to assess mental disorder prevalence. This approach may encompass a broader range of mental distress, potentially including subthreshold experiences, and may not be directly comparable to studies using validated mental disorder screening tools. However, a broader scope is valuable for understanding the full spectrum of mental health within a population, not solely clinical diagnoses.

The inclusion of subthreshold experiences offers insights that have value in contributing to better understanding of help-seeking behaviour as well as identification of at-risk populations (28, 29). A single-item question on experiencing symptoms or receiving a diagnosis of mental disorder during a certain time period in the past is an approach often used in population surveys in the field of mental health (30) as well as in other fields (31). Secondly, the goodness-of-fit of our multinomial logistic regression model, as indicated by the Pearson test, was less than ideal. While the Deviance test suggested adequate fit, the Pearson test indicated suboptimal goodness-of-fit for our multinomial logistic regression, possibly due to large sample size and outcome class imbalance. A secondary analysis with reduced imbalance improved the Pearson test. Although statistical fit has limitations, the model’s face validity and meaningful insights remain valuable. Readers should consider fit limitations when interpreting results. Finally, someone may dispute lack of sensitivity analysis. In fact, this analysis was done, but due to the limits of the journal it could not be presented. However, the results can be obtained from the authors. A key strength of this study lies in its unique access to both self-reported data from the large, representative EHIS survey and comprehensive administrative health databases, linked at the individual level. This approach allows for a more comprehensive understanding of the discrepancies and complexities inherent in identifying mental health issues using different data sources. Finally, by focusing on the Slovenian population within its unique healthcare context of universal coverage and a mixed public-private provider model, this study offers insights relevant to similar health systems where diverse factors may influence mental health help-seeking and reporting.

Integrating diverse data sources and refining measurement instruments are crucial steps towards a more comprehensive and accurate picture of mental health in our communities. Additionally, a deeper understanding of the factors influencing help-seeking behaviour, and how this relates to mental health and health outcomes, is essential for developing effective interventions and support systems. Future research should further explore the factors influencing discrepancies between data sources and develop strategies to bridge the gap between self-reported experiences and objective health records.

To improve agreement between these data sources, collaborative efforts are needed. Improving the accuracy and completeness of administrative data requires implementing standardised coding practices and encouraging the use of validated clinical assessment tools. This necessitates cooperation between public health researchers, healthcare providers and policymakers.

Enhancing the validity of self-reported data relies on using validated questionnaires with clear instructions, while ensuring respondent privacy. Finally, reconciling discrepancies requires implementing data linkage initiatives and analysing the reasons for discrepancies between data sources.

These efforts, along with initiatives such as the European Health Data Space, which opens possibilities for large-scale, multinational longitudinal studies, are crucial for ensuring data accurately reflect the mental health needs of the population.