Validity and reliability of Korean version Competency Assessment Tool-Mental Health
Article Category: Original article
Online veröffentlicht: 14. Juli 2020
Seitenbereich: 143 - 152
Eingereicht: 14. Juni 2019
Akzeptiert: 12. Sept. 2019
DOI: https://doi.org/10.2478/fon-2020-0017
Schlüsselwörter
© 2020 Hyun Mee Cho et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
According to a status report on mental diseases in Korea published by the Korean Ministry of Health and Welfare, approximately 4.7 million adults, or 11.9% of the total adult population in Korea, experience at least one mental health issue in a year, with a prevalence rate of 25.4% for mental diseases within their lifetime.1 In addition, a number of people in Korea either directly or indirectly experience mental health issues. As of 2015, the number of patients who received inpatient mental health treatment reached approximately 81,000, whereas the number of outpatients was approximately 45,000/day. Although the number of patients in 2015 decreased slightly compared with 2014, it has shown a consistent upward trend over the last decade.2,3,4
Symptoms of mental health issues include auditory hallucination, delusion, depression, and asthenia, which lead to self-harm, suicide, drug addiction, and falling. In the United States, One out of eight inpatients in emergency ward is admitted to general ward due to suffering from mental health issues.5 Moreover, when patients have mental health issues, they often struggle to realize quickly and make a judgment regarding their physical diseases or do not voluntarily seek medical assistance on time, leaving diseases untreated or missing out on the proper time for treatment. Many patients come to hospital or seek professional medical assistance only after their condition has deteriorated.6
Given such circumstances, nurses at general wards often encounter a situation where they need to care the patients with mental health issues.5 Nurses are voicing difficulties in caring for such patients as they lack with appropriate knowledge and experience in mental health care. This problem can act as an obstacle for the effective communication between nurses and patients, thereby increasing at least twofold the probability of preventable physical adverse reactions.7
Meanwhile, mental health care is an essential nursing service for providing necessary health care to patients in need by identifying any actual or potential mental health issues and for maintaining and promoting their mental health.8 To implement mental health care adequately, profound understanding and professional knowledge and skills are required, particularly those related to the growth and developmental tasks of human beings. The competency of nurses with relevant education, training, and experience is an important prerequisite to the provision of care for patients with mental health issues.7 Nurses at the Department of Psychiatry consider themselves as a tool for nursing; they control the pace of their speech, vocal tone, and level of activity among other aspects, when conversing or interacting with patients to reduce and control stimulation, and ultimately ensure the well-being of patients and foster a therapeutic environment.9 In addition, an accurate assessment of patients during nursing enables nurses to predict unconventional and at times destructive behaviors of patients with mental health issues and helps them to keep the patients safe, enhancing the efficiency of treatment.10 For the well-being and recovery of patients and reduction in treatment cost, nurses at general wards need to develop competency in mental health care.7 However, compared with nurses at the Department of Psychiatry, nurses at general wards have relatively less knowledge and interest as well as experience in mental health care.11,12 Moreover, nurses at general wards focus more on the physical conditions than mental conditions of patients10 which may lead to an extension of hospital stay, additional hospitalization cost, and increased risk of disease recurrence,7,13,14 ultimately decreasing the treatment satisfaction level of patients. Such situation has given rise to the increasing need for interest in and education on mental health care by nurses at general wards.12
Nurses at general wards feel stress or burden from patients with mental health issues,5,15 and it is reported that nurses experience “care with no effect,” “bias,” “avoidance of mental health care,” “difficulties in care owing to lack of knowledge or environmental restrictions,” and other problems when caring for patients with mental health issues. To address these problems, nurses need to enhance their competency in mental health care; it is important to provide regular education and training related to mental health care for nurses at general wards to that end.16 In addition, in a study on the mental health care competency of nurses at general wards and patients with mental disorders conducted in Korea, 91.5% of incumbent nurses at general wards of hospitals call for refresher courses on mental health care.17
To develop educational programs and intervention methods on mental health care for general nurses, their actual competency in mental health care need to be assessed by identifying their skills and level of knowledge in mental health care. A number of studies conducted in Korea have used the Behavioral Health Care Competency (BHCC) tool.7 Another measurement tool is the Competency Assessment Tool-Mental Health (CAT-MH).11 There has been no development of similar tools in Korea.
To evaluate the mental health care competency of general nurses in Korea in a reasonable and systematic manner, there is a need for the development of mental health care tools that adequately consider Korean culture, the increasing demand for mental health care, and characteristics of the nursing practice environment. In this context, this study was conducted to provide essential information on the development of educational programs and intervention methods for mental health care by verifying the reliability and validity of CAT-MH and establishing basic data for identifying the level of mental health care competency of general nurses. The findings are expected to contribute to future development of an effective measurement tool of mental health care competency in Korea. The specific objectives of this study are as follows:
To confirm the content validity of CAT-MH, To confirm the construct validity of CAT-MH, To confirm the concurrent validity of CAT-MH, and To confirm the internal consistency for reliability verification of CAT-MH.
This study aimed to establish basic data for identifying the level of mental health care competency of general nurses through verification of the reliability and validity of CAT-MH in the development of a measurement tool for mental health care competency in Korea.
This methodological study sought to test the validity and reliability of the translated Korean version of CAT-MH, a competency measurement tool for mental health.
This study was conducted on nurses working at general wards, excluding those working at the Department of Psychiatry, in five hospitals with 200 beds or more located in Busan metropolitan city, Gyeongsang-nam-do in Korea. In this study, factor analysis was used as a primary analysis method. To conduct a factor analysis for validity analysis, the number of respondents is required to be at least 5–10 times greater than the number of the expected items in the questionnaire.18 We calculated the sample size based on the guidelines given by Comrey and Lee,19 which stipulated that a sample size of 200 participants or more is adequate for an exploratory factor analysis. Therefore, considering the 25 items expected in this study and a withdrawal rate of 10%, the total number of study participants was set at 275.
A total of 275 questionnaire copies were distributed, and 275 responses were received (collect rate: 100.0% and response rate: 98.9%). Excluding three insincere responses, a total of 272 responses were included in this study. All study procedures were conducted after obtaining approval from the Institutional Bioethics Committee of Kosin University. The study participants were recruited in cooperation with the nursing center of each hospital, and only participants who consented to participate in this study were selected. The participants were informed of the study purpose and that the collected data would be used solely for academic purposes. They were assured of the privacy of data used in validation, and of their anonymity throughout the analyses and statistical processing. Collection of data was conducted outside the working hours of the participants to ensure no disturbance to their work. The participants were informed of their discretion to withdraw from this study at any time and to request for the partial or complete deletion of their data. The person in charge of the collected data was the researcher of this study. The data were stored at a designated place; electronic records were stored and managed in the researcher's personal computer, which required a password to access for security. The data obtained from the surveys were not used for purposes other than academic research. Anonymity was maintained in the study results. The data used in this study will be stored for 3 years; electronic copies will be permanently deleted; and all other forms of documentation, printouts, letters, and records will be shredded or incinerated.
All data collected throughout this study were analyzed using SPSS 20.0 for Windows (SPSS Korea Data Solution Inc.) and AMOS 21.0 (SPSS Korea Data Solution Inc.). The general characteristics of the participants were calculated using real numbers, and percentages, mean, and standard deviations were obtained. Content, construct, and concurrent validity were confirmed. The content validity index (CVI) was used to estimate the content validity. Exploratory and confirmatory factor analyses were conducted to test the construct validity. In addition, the BHCC for nonpsychiatric nurses and CAT-MH were used to confirm the Pearson's correlation coefficient, and Cronbach's α was used to confirm internal consistency.
This study used the translated Korean version of BHCC (2016) developed in the United States for non-psychiatric nurses. The tool comprises the four subfactors: assessment, practice/intervention competency, recommendation of psychotropics, and resource adequacy. It is a 5-point scale assessment with 1 indicating “strongly disagree” and 5 indicating “strongly agree.” A higher score indicated greater BHCC of the nonpsychiatric nurse. In the study by Han and Woo,20 the reliability of the tool was defined according to the following Cronbach's α values: 0.85 for assessment, 0.86 for intervention, 0.86 for recommendation of psychotropics, and 0.87 for resource adequacy. In this study, the Cronbach's α values for reliability were as follows: 0.91 for assessment, 0.90 for intervention, 0.78 for recommendation of psychotropics, and 0.78 for resource adequacy.
The CAT-MH for assessing the mental health care competency of medical professionals was translated into Korean, amended, and complemented on the approval of the developers. The Korean version CATMH underwent validity and reliability tests before use. The tool comprised a total of 26 items (5-point scale) that were classified into sections based on importance and benefits. At the time of the development, the tool reliability was as follows: Cronbach's α = 0.93 for knowledge and Cronbach's α = 0.95 for benefits.
Before proceeding with the study,21 approval for use of the tools was obtained from the developers of the BHCC for nonpsychiatric nurses and CAT-MH via e-mail, in accordance with the guidelines of the World Health Organization. The primary translation of the tool from English to Korean was performed by a professor at Seoul National University with a doctorate from a nursing graduate school who is fluent in both Korean and English, and with experience of living in an English-speaking country for over 5 years and of working as a nurse. The translated Korean version was then reverse-translated from Korean to English by a professor at the same university who has been a professor of English literature for over 20 years, and with experience of living in the United States for over 10 years; the professor reviewed the sentence structure of the items and the similarity in meaning between the original and translated versions. Subsequently, a panel of experts checked the content validity of the tool. A total of 25 items were included in the tool for the use of this study, and one item with low content validity was excluded.
After obtaining approval from the Institutional Bioethics Committee of Kosin University (1040549-180611-SB-0062-01), a preliminary study was conducted from August 1 to 10, 2018 on 10 clinical nurses working at general wards to identify issues with the questionnaire items and other potential issues that may arise during the main study. The participants of the preliminary study were selected based on the same criteria as that of the tool and were asked to respond to the Korean version CAT-MH in a self-reported method. The general characteristics of the participants were as follows: average age of 34.24 ± 2.41 years; possessing a bachelor's degree (six participants); average years of work experience of 9.01 ± 3.68 years; religious faith (four participants); and general nurse as position at work (six participants). No participant had difficulty in interpreting or reading the questionnaire, and 5–10 min was consumed for responding to the questionnaire.
The report showed that women accounted for the largest portion of gender with 93.1%, followed by 47% in their 20s, 38.4% in their 30s, 12.1% in their 40s, and 2.6% in their 50s. The number of unmarried people was 59.9%, compared with 39.7% of married people. Religion (70.7%) outnumbered absenteeism (29.3%), followed by bachelor's degrees (65.1%), professional college graduates (26.7%), and graduate schools (8.25%). The number of general nurses (70.3%) was the largest, followed by those who were charge nurses (24.1%) and those who were nurses (5.6%). The total clinical experience was 28% for <3 years, 39.2% for <9 years and >4 years, and 32.8% for >10 years. The total average year of clinical experience was 8.09 years. The working years at the current department had the most experience with <3 years (64.7%), followed by <9 years (31.5%) and >10 years (3.9%). The average working year at the current department was 3.44 years. There were 55.6% more nursing experience than there were none (45.4%) and 66.8% more psychiatric education experience (33.2%). Psychological nursing learning experience (84.1%) was higher than none (15.9%).
CVI was measured to test the validity by two professors in nursing, one doctoral student in nursing with >18 years of experience as a psychiatric nurse, one doctoral student in nursing with >6 years of clinical experience, and two master's students in nursing with >5 years of clinical experience. The result showed that all items showed a CVI of ≥0.80, except for Item 26 that had a CVI of ≥0.8, with Scale-CVI (average) = 0.95 and Scale-CVI (universal average) = 0.85.
A Kaiser–Meyer–Olkin test was conducted for an exploratory factor analysis, and the result was 0.95. A Bartlett's test of spherecity confirmed the adequacy of the correlation matrix for the factor analysis, and the result indicated the adequacy of the data for the analysis, χ2 = 3,907.85 (
Factor analysis of the Competency Assessment Tool-Mental Health (the section on importance of skills and knowledge).
| Items | Factor loading |
|---|---|
| 1. Knowing the symptoms/characteristics of mental illnesses | 0.51 |
| 2. Knowing the uses and side effects of psychiatric medications | 0.59 |
| 3. Knowing the biological nature of mental illnesses | 0.42 |
| 4. Knowing and using the best types of therapy for general mental illnesses | 0.58 |
| 5. Knowing and using the best types of therapy for people with a forensic status | 0.57 |
| 6. Knowing and using the best types of therapy for people with a substance abuse/mental illness diagnosis | 0.71 |
| 7. Knowing and using the best types of therapy for people with a mental illness/mentally retarded diagnosis | 0.71 |
| 8. Knowing and using the best types of therapy for children/adolescents affected by severe emotional disorders | 0.66 |
| 9. Knowing and using crisis interventions | 0.69 |
| 10. Knowing and using recovery principles | 0.73 |
| 11. Developing and putting into practice individual service plans and individual treatment plans | 0.55 |
| 12. Using community resources effectively | 0.50 |
| 13. Knowing about legal issues, client rights, and ethical issues | 0.57 |
| 14. Working well as a member of an interdisciplinary team | 0.60 |
| 15. Working in a professional way | 0.65 |
| 16. Evaluating one's own work performance | 0.63 |
| 17. Keeping accurate work records | 0.53 |
| 18. Knowing and being able to use first aid, cardiopulmonary resuscitation (CPR), and other ways to keep clients and workers safe | 0.47 |
| 19. Providing basic nursing care, including taking vital signs and helping clients use the bathroom | 0.43 |
| 20. Using different ways to reduce worker stress | 0.51 |
| 21. Knowing basic facts of good nutrition | 0.54 |
| Eigenvalue | 12.15 |
| % of variance | 57.87 |
| % of cumulative | 57.87 |
| Cronbach's α | 0.96 |
Factor analysis of the Competency Assessment Tool-Mental Health (the section on benefits of additional education).
| Items | Factor loading |
|---|---|
| 1. Treating clients with respect and dignity, and as equal partners in their treatment | 0.48 |
| 2. Including family members and other supportive people/groups in the client's treatment | 0.42 |
| 3. Knowing the uses and side effects of psychiatric medications | 0.50 |
| 4. Knowing the biological nature of mental illnesses | 0.50 |
| 5. Knowing and using the best types of therapy for general mental illnesses | 0.58 |
| 6. Knowing and using the best types of therapy for people with a forensic status | 0.57 |
| 7. Knowing and using the best types of therapy for people with a substance abuse/mental illness diagnosis | 0.69 |
| 8. Knowing and using the best types of therapy for people with a mental illness/mentally retarded diagnosis | 0.66 |
| 9. Knowing and using the best types of therapy for children/adolescents affected by severe emotional disorders | 0.64 |
| 10. Knowing and using crisis interventions | 0.64 |
| 11. Knowing and using recovery principles | 0.57 |
| 12. Developing and putting into practice individual service plans and individual treatment plans | 0.54 |
| 13. Using community resources effectively | 0.50 |
| 14. Knowing about legal issues, client rights, and ethical issues | 0.52 |
| 15. Working well as a member of an interdisciplinary team | 0.58 |
| 16. Working in a professional way | 0.66 |
| 17. Evaluating one's own work performance | 0.54 |
| 18. Keeping accurate work records | 0.57 |
| 19. Knowing and being able to use first aid, CPR, and other ways to keep clients and workers safe | 0.54 |
| 20. Providing basic nursing care, including taking vital signs and helping clients use the bathroom | 0.50 |
| 21. Using different ways to reduce worker stress | 0.51 |
| 22. Knowing basic facts of good nutrition | 0.44 |
| Eigenvalue | 12.16 |
| % of variance | 55.27 |
| % of cumulative | 55.27 |
| Cronbach's α | 0.96 |
An additional confirmatory factor analysis was conducted using the confirmed items in the exploratory factor analysis. The goodness of fit of the model for the section on importance of skills and knowledge was checked: χ2 = 530.07 (
Figure 1
Confirmatory factor analysis (The Section on Importance of Skills and Knowledge).
Figure 2
Confirmatory factor analysis (The Section on Benefits of Additional Education).
To test the concurrent validity, the BHCC for nonpsychiatric nurses was used to identify the correlation between the tool and the Korean version CAT-MH in this study. The results showed the correlation coefficient between the tools as 0.13–0.68 (
Correlation among variable.
| Variables | X1 | X2 | X3 | X4 | X5 | X6 |
|---|---|---|---|---|---|---|
| X1 = Assessment | 1 | |||||
| X2 = Practice/intervention competency | 0.68* | 1 | ||||
| X3 = Recommendation of psychotropics | 0.39* | 0.44* | 1 | |||
| X4 = Resource adequacy | 0.40* | 0.49* | 0.24* | 1 | ||
| X5 = Importance of skills and knowledge | 0.32* | 0.31* | 0.33* | 0.14* | 1 | |
| X6 = Benefits of additional education | 0.36* | 0.26* | 0.13* | 0.14* | 0.64* | 1 |
The internal consistency of CAT-MH was confirmed as follows: Cronbach's α = 0.96 for the section on importance of skills and knowledge, and Cronbach's α = 0.96 for the section on benefits of additional education (Tables 1 and 2).
This study aimed to provide basic data for the development of a mental health care competency measurement that is appropriate for the Korean clinical environment after translating the CAT-MH into Korean. We verified that the Korean version CAT-MH is a valid tool for evaluating the mental health care competency of Korean nurses. The internal consistency of CAT-MH was confirmed (given in Cronbach's α values): 0.96 for the section on importance of skills and knowledge and 0.93 for the section on benefits of additional education. Thus, the present results showed no significance differences from those given by Clasen et al.,11 which showed Cronbach's α values of 0.96 and 0.95 for the sections on importance of skills and knowledge and on benefits of additional education, respectively.
In addition, the exploratory factor analysis showed that the data were adequate for the factor analysis. The factor analysis revealed that four items in the sections on importance of skills and knowledge (Item 1: “I treat study participants with respect and consider them as equal partners”; Item 2: “I include to the self-help group family members or other supporters in the process of patients’ treatment”; Item 4: “I respond to culture, race, and gender issues that affect participants”; and Item 16: “I support patients”) and three items in the section on benefits of additional education (Item 3: “I’m aware of mental disorder symptoms and characteristics”; Item 4: “I respond to culture, race, and gender issues that affect patients”; and Item 16: “I support patients”) had a communality of ≥0.4 and were removed. This outcome indicates the cultural and background differences between the participants in the present study and those given by Clasen et al. The study by Clasen et al.11 was conducted on mental health care medical personnel in Ohio, USA, whereas the present work was conducted in Korea on general nurses with no experience in psychiatric care. In particular, Item 4 was about culture, race, and gender issues. Given the difference in culture and working environment between the United States and Korea, there might have been disparities in the perspective of nurses toward treating patients with mental health issues, resulting in the difference between the two studies.17 With regard to Items 1, 2, and 16, general nurses in Korea focus on assessing present symptoms and the rapid relief of such symptoms in clinical practice.10 However, the primary causes of mental issues include the growth and personal environment of patients, involving emotional influence by family or particular incidents.7,8 Therefore, it is considered important to recognize that the relief of symptoms through medication should not be the only focus and that patient's family members or supporters who might have influenced the mental health issues or recovery of the patients should be included in the treatment process of patients with mental health issues.
When comparing the confirmatory factor analysis with the exploratory factor analysis conducted in this study, the former proved a more important analysis process in testing the construct validity, confirming that the Korean version CAT-MH was an adequate model. Not only this result is evident that the tool adequately measures the nurses’ mental health care competency but it also reconfirms the need for adequate education and training to enhance mental health care competency geared toward satisfying the care needs of patients on top of patients’ physical issues. In this context, to enhance the mental health care competency of nurses, hospitals must provide education and training on concepts in mental health care and care practice for patients with mental health issues. To this end, the Korean version CAT-MH, in which reliability and validity were tested in this study, needs to be used to identify the mental health care competency of general nurses to provide necessary education and training.
To confirm the concurrent validity of the Korean version CAT-MH, its correlation with the BHCC for nonpsychiatric nurses was identified. The results showed that it had statistically significant correlation with the two measurement tools. Furthermore, this CAT-MH version can be assumed to be a valid tool for assessing the BHCC of nurses based on the correlation coefficients of 0.60–0.80 (indicating “high validity”) and 0.80–1.00 (indicating “very high validity”) obtained in the evaluation of validity through correlation coefficient.22 In this context, the BHCC of nonpsychiatric nurses in Korea and actual shortcomings in their competency need to be identified. Subsequently, concrete measures to enhance the CAT-MH of clinical nurses can be formulated.
This study aimed to provide basic data for enhancing the mental health care competency of nonpsychiatric nurses by using and verifying the CAT-MH for nurses in Korea. This study also verified the high validity and reliability of the tool in assessing the mental health care competency of nurses and is believed to be significant as basic data for enhancing such competency. However, additional studies are needed on the questionnaire items for mental health care competency to assess whether they take into account the nursing environment in Korea. In addition, the predictability of the Korean version of CAT-MH needs to be verified.