Development and validation of an instrument to assess knowledge, attitudes, and practices on digital health among nursing officers†

Digital health is a concept that has continuously evolved and has transformed almost all healthcare industries in the world lately.^1,2 Although digital health technologies have significant potential to revolutionize healthcare delivery and improve health outcomes, numerous challenges hinder the rapid adoption of these technologies by healthcare professions.³ These adoptions need collective efforts from multiple parties, such as healthcare providers, manufacturers, and health authorities.⁴ In low- and middle-income settings such as Sri Lanka, the lack of digital health capacity of healthcare providers is one of the considerable challenges in this adaptation.⁵

Nurses are playing an important role as care providers in every sphere of healthcare systems worldwide.⁶ With the recent digital health revolution, nurses are forced or pushed to adapt to innovative technological devices, machinery, and equipment and also exchange knowledge, skills, and education in nursing care by the health industry market forces.^7,8 Therefore, nurses should be able to perform electronic documentation, record-keeping, and effective decision-making in a digital technology-rich environment.^6,7 However, the lack of digital health competencies among nurses is a considerable concern that needs to be addressed.^9,10 Therefore, the current study aimed to develop and validate a questionnaire to assess the digital health capacity among nursing officers.

Validation of an instrument is a highly important aspect of an instrument.¹¹ The researchers use study instruments to collect information in research studies; questionnaires are the most widely used tools to collect data.¹² Furthermore, researchers ensure to always collect the required information in the most reliable and valid manner.¹³ Therefore, the validity and reliability of an instrument are significant aspects of a methodology in a research study.^11,14

Validity is a vital aspect that needs to be assessed prior to the data gathering. Validity refers to the extent to which a measuring instrument accurately assesses the specific behavior or quality it is intended to measure.¹⁴ It is important to use validated instruments to collect data for ensuring that the findings obtained as a result of the analyses are valid. There are different types of validity forms that have been suggested in the literature.^11,15 However, 2 types of validity have been generally accepted to have particular importance in the literature, namely content validity and construct validity.^11,15 Content validity is a qualitative form of validity that expresses whether the measuring items of the instrument represent the phenomena intended to be measured and to what extent that instrument items measure the phenomena.^16,17 Although there are several methods to measure the validity of an instrument, obtaining an expert opinion and statistical methods are the most frequently used methods. Even though face validity is considered a weak form of structural validity, many researchers have assessed it during the validation of instruments.^16,18

Reliability refers to the ability of an instrument to measure the phenomenon and give similar results when applied at different times.¹⁹ Here, a strong positive correlation between the results of the measuring instrument at different times is an indication of the reliability of the instrument.^11,15 There are different methods used to determine the reliability of an instrument.²⁰ The test–retest reliability is the most frequently used method among them.²¹ Moreover, internal consistency measures the consistency among the questions within the instrument and also makes an assessment of how well the measuring instrument measures a particular phenomenon; it measures by assessing the correlation between each item that constitutes the instrument.^16,17

2.

Methods

A standardized methodology was followed in the process of the development and validation of the questionnaire.^11,15 The study was divided mainly into 3 steps, the initial development of the questionnaire through a literature review, obtaining content and face validity through an expert review process, and conducting a survey to assess the test–retest reliability.^16,17

A summary of the development and validation of the instrument consists of the following steps (Table 1).

Table 1.

Steps involved in the study.

Step	Nature of activity	Methods	Number of domains	Number of items	Addition or subtraction
1	Development of construct	Literature review Synthesize literature review	-	60	-
2	Item generation	Develop items	04	55	Subtraction of 05 overlapping items
3	Establishment of face validity and content validity	Expert review process	04	51	Subtraction of 05 items Addition of 01 item
4	Assessment of reliability	Test–Retest reliability	03	51	-
5	Assessment of stability	Internal consistency	03	51	-

2.1.

Item generation

A comprehensive literature review was carried out to determine the concepts necessary for inclusion in the questionnaire. “MeSH” terms and other keywords such as “digital health,” “nursing,” “knowledge, attitudes, practice,” and “surveys and questionnaires” were used in PubMed, Google Scholar, and other medical search engines to examine studies done over the past 15 years. Relevant research articles were selected and identified key areas in digital health and nursing for item generation. Article selection process is shown in the Figure 1. Further questions were identified from previous related questionnaires. The following steps were followed to develop the items in the instrument.

Following the gathering of relevant information from the literature review, the construct of items was developed. Items were developed considering appropriateness, structure, and clarity of items and simple language was used so that it is easily understood by the participants; items were nonoverlapping. The final construct of items was developed following a discussion with 3 experts in the fields of telehealth, health informatics, and nursing.^11,16 A Likert-type scale was used to grade the responses, assuming equal intervals between the response options, where possible.

2.2.

Expert review

The developed instrument was reviewed by an expert panel, which consisted of 7 experts, for obtaining content and face validity.^22,23 The expert panel includes experts from the fields of nursing, health informatics, and telemedicine. After providing information and obtaining their consent to the expert review process, the set of questions was distributed among experts along with the instructions to review. They were provided with a questionnaire for obtaining their opinions on the relevancy, clarity, ambiguity, and simplicity of the questions by using a Likert-type scale, as well as allowed to express their free comments and suggestions for further improvement of the instrument.^24,25 The following table was added to the instruction sheet to guide experts on the scoring method of the review (Table 2).

Table 2.

Scoring method.

Items	Scoring
Relevancy	1 = Not relevant
	2 = Item needs some revision
	3 = Relevant but need minor revision
	4 = Very relevant
Clarity	1 = Not clear
	2 = Item needs some revision
	3 = Clear but need minor revision
	4 = Very clear
Simplicity	1 = Not simple
	2 = Item needs some revision
	3 = Simple but need minor revision
	4 = very simple
Ambiguity	1 = Doubtful
	2 = Item needs some revision
	3 = No doubt but need minor revision
	4 = Meaning is clear

After collecting expert responses, the content validity index (CVI) was calculated separately for relevancy, clarity, ambiguity, and simplicity of items.²⁵ The expert responses for CVI were obtained using the 4-points ordinal scale and they are allowed to rate each item on a scale ranging from “very relevant” (score 4) to “irrelevant” (score 1). CVI was reported using the item level-content validity index (I-CVI) and the scale level-content validity index (S-CVI). The value range of I-CVI >0.79 was considered to be the relevant items, CVI between 0.7 and 0.79 items remained with a revision, and <0.7 item was distraction.^24,25 Their comments and suggestions were taken into consideration when revising the questions. Based on the expert review process, some items were removed and some items were added to the questionnaire.

2.3.

Reliability tests

The test–retest reliability test was conducted for assessing the reliability of the instrument using 20 in-ward nursing officers at Colombo South Teaching Hospital.^18,26 The revised instrument was distributed among participants indicating that the subjects would be required to fill in the same questionnaire twice. Based on the standard recommendations for the test and retest, the questionnaires were administered again after 2 weeks (14 days) of the first administration to the same participants.^11,27 Data were collected during their working hours without interfering with their hospital routines. Information regarding the study was provided via an information sheet, and written informed consent was obtained preliminary to the study. After completion of data collection, data were analyzed using IBM Statistical Package for Social Sciences (SPSS) software version 25 (IBM Corporation, Armonk, NY, United States). The intra-class correlation coefficient was calculated for knowledge, attitudes, and practices (KAP) and the overall KAP calculated to assess the test–retest reliability.^28,29 P-values<0.05 were considered statistically significant. When the correlation coefficient is >0.7, the acceptable reliability and coefficient values lying between 0.7 and 0.8 are considered good reliability and a correlation coefficient >0.9 is considered excellent reliability.²⁹

Meanwhile, the internal consistency of the instrument was assessed by calculating the Cronbach’ α value for the overall KAP. When the Cronbach’s α coefficient value <6, the internal consistency of the scale is considered weak and the value between 0.6 and 0.7 is considered acceptable, 0.7–0.9 is considered good internal consistency, and ≥0.9 is considered high internal consistency.^11,29

3.

Results

3.1.

Development of the questionnaire

Comprehensive literature search yielded 95 articles. After the removal of duplicates and screening for titles and abstracts, a total of 62 articles were reviewed for eligibility. In this stage, 19 articles were removed based on the inclusion criteria. Finally, a total of 43 articles were selected to develop questionnaire items. Articles were reviewed and 60 items were generated for the questionnaire. Five items of less relevant and overlapping questions were removed following an initial discussion with the experts. The remaining 55 items were reviewed through the expert review process. Knowledge-assessing questions were developed with 3 options “Yes,” “No,” and “Uncertain.” Attitudes-assessing questions were developed with a 5-point Likert scale from “Strongly Agree” to “Strongly Disagree” to obtain responses. Although homogeneity was not maintained in the practice domain, the questions were developed with a Likert scale, where possible.

3.2.

Expert review

According to the expert review process, 5 items were removed (I-CVI <0.7), 8 items were revised (I-CVI 0.78– 0.8), and 1 new item was added to the questionnaire. S-CVI was calculated using both average (S-CVI (AV)) and universal agreement (S-CVI (UA)) methods. The results found that S-CVI (AV) 0.9 and S-CVI (UA) 0.8 indicate good validity. The final questionnaire consists of 51 items with 4 domains including socio-demographic data, knowledge, attitudes, and practices on digital health. Out of the 51 questions, sociodemographic data contain 13 questions, the knowledge domain contains 5 questions, the attitudes domain contains 25 questions, and the practice domain contains 8 questions.

3.3.

Reliability test

The test–retest study included 15 female (75%) and 5 male (25%) nursing officers, aged between 19 years and 30 years and all participants were grade III nursing officers. For determining the test–retest reliability, the intraclass correlation coefficient was calculated including the KAP domains. The correlation coefficient values were 0.76, 0.98, 0.99, and 0.99 for KAP and total questionnaire, respectively, indicating excellent reliability (P < 0.05). These results indicate that the developed instrument is reliable in assessing KAP on digital health among nurses.

Furthermore, to determine the internal consistency of the questionnaire, Cronbach’s α coefficient for the whole questionnaire was calculated and it was found to be 0.76. These values indicate good internal consistency of the questionnaire.¹¹

The following table shows items in the KAP domains in the final version of the instrument (Table 3).

Table 3.

Final version of the questions of the KAP domains.

Domains	Items
Knowledge
1. Digital technologies (use of computers, mobile phones and other electronic tools and communication technologies) are used in many fields worldwide	a. The term “Digital Health” means the provision of healthcare via digital technologies.
	b. Digital health has been used in all areas such as clinical, educational, managerial, administrative, training, and research.
	c. Digital health limits the access to healthcare for people in rural or remote areas.
	d. Digital health technologies are being used in both curative health and public health.
	e. Digital health facilitates managing data services such as data collection, management, use and exchange in health systems.
2. Your hospital is going to implement an eHIS. This implementation will facilitate	a. Direct access to all the information of patients through the computer.
	b. Easy access to laboratory system through the computer at any time.
	c. The maintenance of paper-based medical records.
	d. The management of patients through decision support tools.
	e. To provide all nursing care through electronic devices.
3. Telemedicine	a. Telemedicine is the use of telecommunication to provide medical education and services at a distance.
	b. Telemedicine needs sophisticated technologies.
	c. Patient management with prescribing drugs can be done through telemedicine.
	d. Follow-up of patients can be done through telemedicine.
	e. In telesurgery, a surgeon performs a surgical case without his/her being present in the operating room.
4. NI is an emerging specialty globally.	a. It is a combination of nursing science, computer science, information science, and cognitive science.
	b. NI is applicable only in areas of clinical, education, and nursing research.
	c. NI roles require all nurses to be computer literate.
	d. NI is concerned with assisting and processing nursing data and information only.
	e. The goal of NI is to support decision-making only in specialized roles.
5. Suppose you are assigned to do a health education session for a group of patients under the digital health facilities. Do you have knowledge and skills on	a. Using basic components of the computer and its peripheral devices? (e.g., mouse, keyboard, multimedia, printer)
	b. Using the Internet/search engines to gather information?
	c. Developing course materials using MS Word/MS PowerPoint/MS Excel?
	d. Sending a report of your teaching session via email?
	e. Using digital conference facilities such as Zoom/ Google Meet/Teams to conduct online sessions?
Attitudes
Relative advantages of digital health	1. Reduce clinical errors made by nurses when providing patient care.
	2. Enable me to accomplish my tasks more quickly while performing duties in the workplace.
	3. Makes my work easier due to the use of digital devices.
	4. Improves the quality of nursing care provided by me to the patients.
	5. Enables me to provide more comprehensive nursing care.
Compatibility of digital health	6. Compatible with all aspects of nursing activities that I conduct in the workplace during my duty hours.
	7. Fits well with the way I like to work in the workplace.
	8. It is compatible with the current demand in health care.
Acceptability of digital health	9. Nurses should adjust to the digital health environment (NIs).
	10. I am interested in practicing digital health in my daily duties as a nurse.
	11. I am interested in getting education/training on digital health.
	12. I am seeking proper digital health education/ training.
	13. Digital health reduces time spent with the patient.
	14. A nurse should spend more time with the patients, not work with computers.
	15. It negatively affects the nurse–-patient relationship in my workplace.
Complexity of digital health	16. Digital health technologies require large mental effort by nurses.
	17. It is difficult to adapt to digital health technologies in general.
	18. Digital health creates new responsibilities for nursing staff.
	19. The support from the other categories of healthcare staff to practice digital is not adequate.
	20. I am not competent in using digital health.
	21. I do not have facilities to adapt to/practice digital health.
	22. I do not have adequate education/training to practice digital health.
Observability of digital health	23. I have observed how other health professionals utilize digital health technologies.
	24. I notice that digital health technologies are being used for a variety of tasks at the hospital.
	25. In the hospital where I work, digital health is prominent.
Practice
1. Do you have any digital health implementations (e.g., HIMS, EHR System, Telehealth) in your hospital?	a. Client identification through digital devices
2. What are the digital health activities available in your hospital? (Select all that apply to you and put an [X] mark in the brackets given).	b. Client registration through digital devices
	c. Trace laboratory investigations through digital devices
	d. Client discharge plan through digital devices
	e. Help to maintain clients’ medical records digitally (EMR/EHR/PHR)
	f. Use nurses’ decision support tools (checklists, alert systems, reminders)
	g. Provide nursing services through digital technologies whenever physical distance exists between client and nurses (telenursing practices)
	h. Communication through digital technologies
	(transmit routine news, workflow notifications, communication among staff)
	i. Referral coordination via digital technologies
	j. Activity planning and scheduling via digital devices (nurses notes)
	k. Nursing officer’s training (IT training, eLearning)
	l. Track client’s medication consumption and monitoring
3. How long (in months) you have been engaging in the above-mentioned digital health activities?	a. Client identification through digital devices
	b. Client registration through digital devices
	c. Trace laboratory investigations through digital devices
	d. Client discharge plan through digital devices
	e. Help to maintain clients’ medical records digitally (EMR/EHR/PHR)
	f. Use nurses’ decision support tools (checklist, alert systems, reminders)
	g. Provide nursing services through digital technologies whenever physical distance exists between client and nurses (Telenursing practices)
	h. Communication via digital devices (transmit routine news, workflow notifications)
	i. Referral coordination via digital technologies
	j. Activity planning and scheduling via digital devices (nurses notes)
	k. Nursing officer’s training (IT training, eLearning)
	l. Track client’s medication consumption and monitoring
4. Which of the following digital devices do you use in your digital health practice in the workplace? (Select all that apply to you and put an [X] mark in the box given)	a. Desktop computer
	b. Laptop/notebook computer
	c. Tablet computer (e.g., iPad, Galaxy)
	d. Smartphone
	e. Landline phone
	f. Fax machine
5. How often do you use above-mentioned digital devices in the workplace?	a. Desktop computer
	b. Laptop/notebook computer
	c. Tablet computer (e.g., iPad, Galaxy)
	d. Smartphone
	e. Landline phone
	f. Fax machine
6. Which of the following digital devices do you use in your activities not related to work? (Select all that apply to you and put an [X] mark in the box given)	a. Desktop computer
	b. Laptop/notebook computer
	c. Tablet computer (e.g., iPad, Galaxy)
	d. Smartphone
	e. Mobile phone (normal)
	f. Landline phone
	g. Fax machine
	h. Wearable devices
7. How often do you use the above-mentioned digital devices for activities not related to work?	a. Desktop computer
	b. Laptop/notebook computer
	c. Tablet computer (e.g., iPad, Galaxy)
	d. Smartphone
	e. Mobile phone
	f. Landline phone
	g. Fax machine
	h. Wearable devices
8. What is your main purpose in using the above-mentioned digital devices in your daily life?	b. Find health information
	c. Access to health care
	d. For health education
	e. Communication
	f. Digital health games

Note: eHIS, electronic Health Information System; HER, Electronic Health Record; HIMS, Health Information Management System; EMR, Electronic Medical Record; HER, Electronic Health Record; PHR, Patient Health Record; KAP, knowledge, attitudes, and practices; NI, nursing informatics.

4.

Discussion

This study was conducted mainly on content validity, face validity, and reliability analyses to ensure that the items assessing digital health capacity in each domain (knowledge, attitude, and practice, KAP) were appropriately developed. Certain systematic methods have been employed by researchers to develop KAP questionnaires such as literature reviews, expert reviews, focus group discussions, and interviews.¹⁶ We followed a comprehensive literature review and expert review to develop this questionnaire.

The developed questionnaire consists of 3 domains assessing KAP on digital health. The domain of knowledge consists of 5 major questions assessing knowledge on electronic Health Information Systems (eHIS), Telemedicine, Nursing Informatics (NIs), e-health, and computer literacy. The domain of attitudes consists of 25 major questions assessing attitudes under relative advantages, compatibility, acceptability, complexity, and observability of digital health. The domain of practice consists of 8 major questions assessing their usage of digital health in the workplace and personal life. This questionnaire has been developed to cover the KAPs in almost all aspects of digital health indicating good laterality.

Researchers in the field of nursing have conducted KAP studies to assess nurses’ KAP in certain areas such as diseases, care modalities, infection control and nursing education in Sri Lanka.^30,31 However, there is a lack of evidence regarding KAP surveys conducted to assess KAP on digital health or health informatics in Sri Lanka.^5,10 Moreover, the conducted studies have not provided KAP questionnaires to be utilized by other researchers.

Certain factors can create risks to the validity and reliability of the findings of a researcher. One of the common factors is measurement errors. It can happen due to a lack of validity and reliability of the measuring instrument. Although researchers cannot totally eliminate these errors, they can minimize them by employing valid and reliable instruments.¹¹ The KAP questionnaire developed by the current study has excellent validity (S-CVI for relevancy, clarity, ambiguity, and simplicity were 0.93, 0.91, 0.94, and 0.92, respectively) and good reliability (correlation coefficient for all items 0.99) and good internal consistency (0.76) that can be yielded by any researcher. Subsequently, this instrument has good content validity and good reliability.

This questionnaire can be used by researchers with confidence to assess KAP on digital health in health care professions locally and internationally. The findings from subsequent research will help to assess nurses’ KAP on digital health. The educators will be able to develop effective teaching approaches and training programs in NIs. It will serve as a pillar in fulfilling the requirement of a qualified nursing professional with informatics knowledge and digital health competencies for strengthening health care in the country.

5.

Conclusion

This study presents the development and validation of a questionnaire assessing KAP on digital health. Content validity and reliability were assessed. Results showed that the face validity and content validity of the instrument were established. Further, results convey that the instrument has good test–retest reliability and good internal consistency. This is a feasible questionnaire that can be employed to assess KAP on digital health in health care professions confidently.

Language:: English

Publication timeframe:: 4 times per year
Journal Subjects:: Medicine, Assistive Professions, Nursing

Journal RSS Feed

Development and validation of an instrument to assess knowledge, attitudes, and practices on digital health among nursing officers†

Maheshika Madhuwanthi

Sunethra Jayathilake

Neranga Liyanaarachchige

Rohana Marasinghe

Article Category: Original article

Published Online: Sep 16, 2024

Page range: 275 - 284

Received: Jan 15, 2024

Accepted: Mar 28, 2024

DOI: https://doi.org/10.2478/fon-2024-0030

Keywordsassess, development, digital, health, instrument, validations

© 2024 Maheshika Madhuwanthi et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Development and validation of an instrument to assess knowledge, attitudes, and practices on digital health among nursing officers^†

Keywords
assess, development, digital, health, instrument, validations