Validity and reliability of the evaluation tool of TCM confidence in students with TCM learning experience
Article Category: Original Article
Publicado en línea: 18 dic 2023
Páginas: 465 - 469
Recibido: 17 mar 2023
Aceptado: 18 may 2023
DOI: https://doi.org/10.2478/fon-2023-0049
Palabras clave
© 2023 Rui-Min Jia et al., published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
Traditional Chinese medicine (TCM) is one of the great medical achievements, which has been developed more than 5000 years in China.1 It was a medical science formed and matured empirically during Chinese people’s daily life and against diseases. TCM has its own special characteristics assimilating from Chinese culture. It uses a holistic view to diagnosis and treatment of diseases by combining individuality and natural and social environments together.2
TCM has been focusing on disease prevention and health promotion, and been making great contributions to the reproduction and survival of the Chinese nation. Its medical achievements are not just because it is an effective treatment method but also because of emphasis on regulating people’s life pattern.3 Recently, COVID-19 has been rampant worldwide. TCM is acknowledged globally as an effective method for treatment and prevention.4
In light of these contributions, the Chinese government has issued a large series of policies and laws to inherit and advance TCM.5 One of the important strategies is setting up TCM institutions and TCM subjects in medical institutions to play a key role in cultivating TCM servicers. The number of graduated and current students has been increasing gradually.6 However, there is no evaluation tool to evaluate the study results – the confidence level of TCM. Therefore, this current study aims to design an evaluation scale and then evaluate its validity and reliability.
The participants of this current study were nursing undergraduates who have studied TCM in the Shanxi University of Chinese Medicine. Three levels of students were included – sophomore, junior, and senior, and first-year students were excluded because of lack of learning experience of TCM. Three parallel samples were included: 200 students for pretesting and identifying the original entry, 500 students for factor analysis and validity and reliability analyses, and 200 students for verifying validity and reliability.7 Every participant was voluntarily recruited by tutors notification and confidentially maintained throughout the present study. Ethical approval was obtained from the Shanxi University Human Ethnics Committee (No. LL038).
The questionnaire was designed based on talent training objectives of TCM and documented studies of TCM education,8–11 including 46 items. Seven TCM specialists were involved to evaluate the content validity of the items, and after the assessment, 34 items were selected.
Data were collected online through the Questionnaire Star Website from August to November 2020. Responses to the examined questions were rated on a Likert scale with 5 degrees, with values ranging from 1 to 5. The mean score was positively related to the confidence level of TCM.
Results were described and analyzed using Statistics Package for the Social Sciences (SPSS) version 22.0 (IBM Corporation, Armonk, New York, United States). Qualitative data were represented by frequency and percentage (%), and quantitative data were represented by mean ± standard deviation (M ± SD). Exploratory factor analysis (EFA) was performed by using the principal component method with maximum variance factor rotation, and the characteristic root >1 was used as the standard for extracting common factors. The reliability was evaluated by Cronbach’s coefficient for internal consistency, and Pearson’s intraclass correlation coefficient (ICC) for test–retest reliability. Confirmatory factor analysis (CFA) was performed by AMOS 24.0 software (IBM Corporation, Armonk, New York, United States) for evaluating structure validity and content validity.12 The evaluation of the fitting model was achieved by χ2/freedom degree (DF), root mean error of approximation (RM-SEA), and goodness-of-fit index (GFI), comparing fitting index (CFI) and normed fit index (NFI).13 A
The participants were nursing students recruited voluntarily. The majority were female, and first-year students were not included because of lack of experience of Chinese lectures. The specific information is presented in Table 1.
Demographic characteristics of participants.
| Item | Pretesting acquisition entry (N = 200) | Factor analysis and reliability analysis (N = 500) | Evaluating structure/content validity (N = 200) | |||
|---|---|---|---|---|---|---|
| N | % | N | % | N | % | |
| Male | 35 | 17.5 | 57 | 11.4 | 38 | 19 |
| Female | 165 | 82.5 | 443 | 88.6 | 162 | 81 |
| Sophomore | 57 | 28.5 | 153 | 30.6 | 63 | 31.5 |
| Junior | 89 | 44.5 | 187 | 37.4 | 78 | 39.0 |
| Senior | 54 | 27.0 | 160 | 32.0 | 59 | 29.5 |
The EFA results showed that the Kaiser–Meyer–Olkin (KMO) value was 0.704, and the Bartlett spherical test value was 1442.52 (
Factor loading of observed items in the questionnaire.
| Cognition of TCM culture and theory | Advantages and characteristics of TCM diagnosis and treatment | Feelings about TCM achievements | Personal behaviors related to TCM | Attitudes towards the association between TCM and Western medicine | Attitudes towards the future of TCM | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Items | Loading | Items | Loading | Items | Loading | Items | Loading | Items | Loading | Items | Loading |
| A3 | 0.83 | B2 | 0.72 | C1 | 0.81 | D3 | 0.72 | E1 | 0.70 | F3 | 0.75 |
| A5 | 0.74 | B4 | 0.72 | C2 | 0.76 | D2 | 0.67 | E2 | 0.57 | F2 | 0.78 |
| A4 | 0.73 | B3 | 0.67 | C3 | 0.70 | D1 | 0.61 | F4 | 0.78 | ||
| A2 | 0.67 | B6 | 0.65 | F1 | 0.64 | ||||||
| A1 | 0.65 | B1 | 0.63 | F5 | 0.52 | ||||||
| A6 | 0.50 | B5 | 0.53 | ||||||||
Content validity was analyzed from 2 aspects, namely, Item Content Validity Index (I-CVI) and Scale Content Validity Index (S-CVI). The results showed that the I-CVI was 0.83–1.00 and the S-CVI was 0.92.
In this study, the differential validity was used to evaluate criterion validity. The differences in scores of the total content and of each dimension between 2 groups (group 1: willing do TCM related work in future; group 2: do not willing do TCM-related work in future) were analyzed by using the independent sample t-test. The results are provided in Table 3.
Results of discriminant validity between 2 groups (M± SD).
| Group | No | Scores: cognition of TCM culture and theory | Scores: advantages and characteristics of TCM diagnosis and treatment | Scores: feelings about TCM achievements | Scores: personal behaviors related to TCM | Scores: attitudes towards the association between TCM and Western medicine | Scores: attitudes towards the future of TCM | Scores: total content |
|---|---|---|---|---|---|---|---|---|
| 1 | 412 | 23.11 ± 4.17 | 28.13 ± 6.17 | 20.45 ± 5.23 | 21.07 ± 4.71 | 18.78 ± 6.57 | 33.19 ± 7.22 | 142.70 ± 18.61 |
| 2 | 88 | 20.18 ± 5.21 | 14.21 ± 5.71 | 17.38 ± 4.67 | 10.61 ± 6.18 | 16.32 ± 4.36 | 23.62 ± 7.54 | 103.28 ± 25.37 |
| T-value | 5.36 | 14.16 | 4.34 | 19.76 | 3.76 | 17.18 | 29.25 | |
| <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 |
Cronbach’s a coefficient of the questionnaire was 0.95 and that of the 6 examined aspects was 0.92, 0.92, 0.94, 0.96, 0.93, and 0.97, respectively, and split-half reliability was 0.81. These results were obtained through analyzing 500 validated questionnaires. After 4 weeks, the retest reliability was performed by analyzing 200 questionnaires. Cronbach’s a coefficient of the questionnaire was 0.93 and that of 6 examined aspects was higher than 0.80 (i.e., 0.82, 0.90, 0.85, 0.93, 0.81, and 0.90, respectively).
The contents of the questionnaire are shown as follows (Table 4).
Contents of the questionnaire.
| Observed aspects | Items |
|---|---|
| The cognition of TCM culture and theory | A1 People foremost |
| Advantages and characteristics of TCM diagnosis and treatment | B1 “Syndrome differentiation and treatment” & “Holistic treatment” |
| Feelings about TCM achievements | C1 Tu You-you won the Nobel Prize in Literature |
| Personal behaviors related to TCM | D1 Engaged in TCM related work |
| Attitudes towards the association between TCM and Western medicine | E1 Chinese and western medicine should complement each other in the treatment of diseases |
| Attitudes towards the future of TCM | F1 The government policies enhance the development of TCM |
The great contributions have been made by TCM for the public health. The Chinese government and related departments have been putting efforts to enhance TCM and to facilitate greater understanding and acceptance of TCM, with the aim of providing medical services to global people. One of the strategies is cultivating TCM talents, who are the major contributors to TCM. It is essential for these individuals to have confidence in TCM themselves before they can make contributions to its development.5
However, the scientific research in this field is very few. There is currently no evaluation tool of TCM confidence. Therefore, it seems that this present research did a valuable work. The evaluation tool was designed based on talent training objectives of TCM, documented studies, and suggestions of TCM experts.8–11 Finally, the content was decided to include 6 aspects—the cognition of TCM culture and theory, advantages and characteristics of TCM diagnosis and treatment, feelings about TCM achievements, personal behaviors related to TCM, and attitudes toward Western medicine and the future of TCM.
In the current study, the validity and reliability of the evaluation tool of TCM confidence were tested. Generally, the reliability is evaluated using Cronbach’s coefficient and test-retest reliability. Validity is evaluated mainly based on 3 aspects, namely, content validity, structure validity, and differentiation validity. Cronbach’s
In this study, the evaluation tool of TCM confidence includes 6 dimensions and 25 entries demonstrating good validity and reliability. The results suggest that the designed tool can be used to assess TCM confidence levels of students with TCM experience and can also evaluate outcomes of TCM education when studying TCM lectures. Hence, this evaluation tool may not only reflect the TCM confidence level of TCM students but also show weaknesses of TCM education. This helps educators find problems, devise strategies to improve it, and finally contribute to TCM. However, this evaluation tool of TCM confidence is a subjective evaluation scale, which may limit to show objective abilities of TCM. Therefore, it may be necessary to develop both subjective and objective evaluation scales in future.