The development of a consensus definition for healthcare improvement science (HIS) in seven European countries: A consensus methods approach
Article Category: Original scientific article
Publicado en línea: 26 feb 2017
Páginas: 82 - 90
Recibido: 28 feb 2016
Aceptado: 29 ago 2016
DOI: https://doi.org/10.1515/sjph-2017-0011
Palabras clave
© National Institute of Public Health, Slovenia
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.
Improving healthcare quality has become a priority over the past years (1). While much of quality improvement work is unscientific (2), the adoption of a more scientific approach to improvement could enhance the ability of health systems to provide high-quality care and use their resources optimally (3). Healthcare improvement science (HIS) represents ‘the combined and unceasing efforts of everyone – healthcare professionals, patients and their families, researchers, payers, planners and educators – to make the changes that will lead to better patient outcomes (health), better system performance (care) and better professional development (learning)’ (4, p. 2). In this respect, quality improvement and change making should become an essential part of healthcare professionals’ work at all levels of the healthcare system (5). Implementation science and quality improvement efforts share the ultimate goal of improving healthcare quality. However, there are differences. Quality begins with a specific problem in a given healthcare system, while implementation science begins with evidencebased practice (EBP) that is not used effectively, then identifies and addresses quality issues at the level of providers, clinics, or the healthcare system (6). Topic reviewers reported a diversity of outcomes used for quality improvement evaluations, flaws in study design, and incomplete reporting (7).
The Health Foundation (8) developed an explanation of what improvement science should be. They defined it as ‘building a knowledge base for improving health services and translating this knowledge into practice to deliver the best possible patient care’ (8, p. 7). Moreover, it has been stressed that the implementation science community should provide guidelines for reporting the use of theoretical frameworks within implementation studies and their efficacy (9). This is important because few hospital-based interventions are essentially theoretical (10). The Health Foundation definition (8, p. 7) states clearly that improvement science ‘requires a systematic, scientifically rigorous approach to close the gap between current and best practice.’
In the case of quality improvement, theory needs development, which draws from a better understanding of practice (11). Many quality improvement practitioners believe that quality improvement is implemented in complex adaptive systems. They sometimes feel limited by research findings that fail to recognize the complexity and the need for adaptation. Quality improvement practitioners could help researchers to see the larger patterns specific to complex systems and draw upon possible solutions (12). Organizational learning framework could be used to explore the factors influencing improvement feasibility (13). Organizations worldwide have adopted interprofessional education and interprofessional collaborative practice in an effort to improve healthcare delivery systems (14), which fits well with the Health Foundation definition (8, p. 7) that ‘improvement science embraces disciplines across health services research, from sociology to statistics, psychology to health economics.’ The next generation of healthcare professionals will require new skills to make sure that quality improvement in health care is successful (15). For this reason, it is important how different healthcare professionals understand HIS.
The article describes a part of the project ‘Improvement Science Training for European Healthcare Workers (ISTEW),’ which aimed to develop shared academic and practice-based programmes that would enable European institutions to build improvement capacity within their healthcare workforce. This article discusses a keystone component of the project, namely the development of a consensus definition for HIS, based on the interprofessional and multicultural consensus of healthcare professionals considered as experts from seven European countries.
Research design comprised of consensus methods, including the nominal group technique, the Delphi process, and the consensus group technique. The aim of consensus methods is to determine the extent to which experts or lay people agree about a given issue. The term ‘agreement’ takes two forms, which need to be distinguished: firstly, the extent to which each respondent agrees with the issue under consideration (typically rated on a numerical or categorical scale) and, secondly, the extent to which respondents agree with each other (16).
The nominal group technique uses a highly structured meeting to gather information from relevant experts (usually 9-12 in number) about a given issue. It consists of two rounds, in which panellists rate, discuss, and then rerate a series of items or questions (16). The nominal group technique was used twice. First, we implemented two structured Skype meetings with research group members from seven European countries. The goal was to develop a working definition of HIS, to increase consensus amongst participants about the meaning and content of HIS, and to establish the level of development and understanding of HIS in each country. Expert-based knowledge, literature and comprehension across a core group of seven experts was used. After two meetings, a working definition was developed and tested in the Delphi process. Second, the nominal group technique was used on the same team in order to develop an extended questionnaire for the 2nd Delphi iteration.
A Delphi study is a robust method that draws on expert opinions, and compares them with the combined opinions of other participating experts over several rounds, until a consensus on specified criteria is reached (16). A two-iteration Delphi technique was developed with participants from seven European countries. The Delphi technique enabled us to collect data from several domains related to healthcare in education, research, management and practice. Moreover, two questionnaires were designed for each iteration with quantitative and qualitative elements allowing us to achieve a merging of opinions from 87 respondents. The respondents were consulted in two anonymous Delphi rounds. A 5-point Likert scale (1–strongly disagree, 2–disagree, 3–neither agree nor disagree, 4–agree, 5–strongly agree) was used in both questionnaires to rate the importance of each HIS definition element and of the definition as a whole. Respondents could also offer suggestions for improving separate parts of the definition by replying to open-ended questions and providing HIS descriptions used in their countries. Furthermore, we were interested in obtaining some demographic data (the area of work, respondent connection with HIS and involvement in HIS) and data about the description of HIS in their countries. Reliability test results were good. Cronbach’s Alpha for statements (N=6) for 1st Delphi iteration was 0.867 and for statements (N=7) for 2nd Delphi iteration 0.843.
Over several rounds, a specific definition element was included in the next version of the definition if >80% of respondents judged it as important or very important, and excluded if >50% judged it as not important or moderately important (17). The suggested reformulations of the second HIS definition were analysed with the nominal group technique by using qualitative and quantitative data from the first Delphi iteration. These reformulated items were presented in the second Delphi iteration.
Following two Delphi iterations and two nominal group techniques, a consensus group technique (a combination of a focus group and a public meeting) was employed. The main difference between consensus groups and focus groups is that in the latter, the moderator or analyst decides, whereas in consensus groups, the participants themselves negotiate and decide the findings (18). We used this technique at a three-day meeting of all ISTEW project members held in Bled (June 2014), Slovenia. The consensus group included 18 participants from seven European countries. We wanted to provide everyone in the group the opportunity to work collaboratively, to be involved in the decision-making processes, to be valued as experts, and for the group to become a forum for change (19). Consensus group members were discussing quantitative and qualitative data of the 2nd Delphi iteration and the suitability of the developed HIS definition. The aim was to arrive at a final definition of HIS.
A non-random convenience sample was used for the Delphi process. Each project team invited 10–15 people they knew to be involved in HIS in their respective settings. ‘Participants’ came from seven European countries (England, Italy, Poland, Romania, Scotland, Slovenia, and Spain) and from different healthcare areas. The total number of respondents in the 1st iteration was 87; of these, 66 chose to participate in the 2nd iteration. HIS fields in which the respondents were involved are shown in Table 1.
Participants’ fields of work in the Delphi study.
| Education, research | Management | Clinical practice | Public policy | Patient organisations | |
|---|---|---|---|---|---|
| 1st iteration (n=87) | 37% (32) | 29% (25) | 23% (20) | 7% (6) | 4% (4) |
| 2nd iteration (n=66) | 27% (18) | 39% (26) | 23% (15) | 5% (3) | 6% (4) |
Ten members of the research group participated in the nominal group technique, and 18 members of the ISTEW project group participated in the consensus group technique. Members of both groups were experts in HIS from different professions.
The research was conducted from November 2013 to September 2014. We started with the nominal group technique, first conducted in November 2013 and then in March 2014, followed by the Delphi process (the first and second iterations in December 2013 and April 2014, respectively), and finally, the consensus group technique in June 2014.The research protocol was approved by the scientific ethical committee of the research coordinator, the Faculty of Health Care Jesenice (Slovenia). Partners also adhered to their ethical approval processes for research. Participants were assured about the anonymity and confidentiality of their collaboration. The questionnaires were available either in English or the language of the expert. Partners were responsible for translating the questionnaires, as required.
Quantitative data analysis was carried out with descriptive statistics (Mean, Standard Deviation, Frequency, Percentage) and bivariate statistics (Chi-Square (p) – Kruskal-Wallis H nonparametric test, Independent samples t-test, Paired samples t-test, Pearson correlation, Spearman correlation) in the statistical software programme SPSS, v. 20.0. The level of statistical significance was set at p<0.05.
Qualitative data was obtained from the open-ended questions of the first and second Delphi iterations. First, each partner team extracted the themes from the qualitative data from both Delphi iterations and exposed these themes in the nominal group technique and the consensus group technique. The software programme NVivo, v. 10.0 and content analysis were used.
We asked respondents to classify themselves in three ways: those who worked in the field of HIS daily (this group we called ‘expert’), those who worked in the field of HIS occasionally (this group we called ‘occasional worker’), and those who were informed about HIS (this group we called ‘informed layman’). In the 1st Delphi iteration, significant differences were established between countries in identifying respondents as experts of HIS (p<0.001), occasional workers (p=0.033) or informed laymen (p<0.001). The number of answers ‘Yes’ was 97; 10 respondents recognized themselves in more than one way. In the 2nd Delphi iteration, significant differences were established between respondents in identifying HIS ‘expert’ (p=0.002) and in classification as an ‘informed layman’ (p=0.020). The number of answers ‘Yes’ was 85; 19 respondents recognized themselves in more than one way. Results are in Table 2. We had more experts and occasional workers in the second iteration, and less informed laymen.
Respondents’ involvement in HIS and differences between countries.
| Groups of respondents | 1stDelphi iteration | 2stDelphi iteration | ||
|---|---|---|---|---|
| Yes f (%) | X2 (p) | Yes f (%) | X2 (p) | |
| 21 (28.0) | 30 (45.5) | 21.181 (0.002) | ||
| 44 (62.9) | 31 (47.0) | 11.172 (0.083) | ||
| 32 (48.5) | 24 (26.7) | 2.614 (0.856) | 15.048 (0.020) | |
X2 (p) is answer ‘Yes’ between countries.
A total of 50% of respondents agreed that their national bodies, commissions or government which include HIS play an important part in healthcare and social policy in their country (n=43, x2=37.502, p<0.001). A total of 25 respondents (35.2%) agreed that HIS has an accepted status in healthcare or social organizations, and 29 respondents (35.4%) reported that national scientific research in HIS has been conducted in their country. We did not collect this data in the 2nd iteration.
In the 1st iteration, we tested the definition of HIS which was developed with the nominal group technique. The definition is shown in Table 3. Results reveal a high level of agreement with the definition as a whole on a 5-point scale. Respondents from Scotland expressed a significantly lower agreement with the definition as a whole, but with the highest level of standard deviation of their answers (M=2.83, SD=1.27, p<0.001) and also with its separate parts (a) (M=2.25, SD=1.22, p<0.001), (c) (M=2.67, SD=0.99, p<0.001), (d) (M=2.57, SD=1.78, p=0.029), except for parts (b) and (e).
Results for the level of agreement with the definition as a whole and with its separate parts (a - e), and differences between countries in the 1st Delphi iteration (n=87).
| M (SD) | Chi-Square (p) | |
|---|---|---|
| Agreement with the definition as a whole, from the part a to the part e. | 3.93(0.861) | 21.533(<0.001) |
| Healthcare Improvement Science (HIS) is the framework for achieving efficiency, efficacy and quality in health and social care (a). | 3.85(1.070) | 30.278(<0.001) |
| HIS depends on knowledge from a wide range of sources, not just research (b). | 4.15(0.958) | 7.921(0.244) |
| HIS is a link (connection) between numerous variables in clinical work and management/leadership work, such as: multi-disciplinary approach, patient involvement, patient safety, total quality management, change management, personal involvement of healthcare/social workers, personal development and responsibility, team work/group work, connections between practice-theory-problems-research, etc. (c). | 4.15(0.907) | 27.436(<0.001) |
| HIS is thus an umbrella term for all actions (practice, education, science and policy) that can lead to better health treatment outcomes (health), better system performance (care), better professional development (learning) and healthier communities (d). | 4.01(1.101) | 14.017(0.029) |
| HIS also demands flexibility and responsibility in our understanding, in our theories of knowledge, and in our use of research evidence in health care improvement strategy (e). | 4.26(0.767) | 7.319(0.292) |
Legend: M - central value of a discrete set of numbers calculated by the sum of the values divided by the number of values; SD - Standard deviation; Chi-Square (p) - Kruskal-Wallis H nonparametric test for several independent samples; a, b, c, d, e - parts of the whole definition.
Comment: X2 (p) is mean value between countries.
The agreement with the definition increased with the level of respondents’ expertise. Correlation analysis showed us that respondents who identified themselves as ‘experts’ were in positive correlation with the agreement with the definition as a whole (r=0.264, p=0.023). Respondents’ fields of work were not in correlation with the level of agreement with the definition as a whole.
Qualitative responses of respondents (n=21) who rated themselves as experts were analysed. A review of responses to the question ‘Which areas are missing from the proposed definition?’ yielded 120 codes and 8 main categories, namely the content of the definition, the purpose of the definition, multidisciplinary approach, definition structure, healthcare professionals, negative consequences, innovative emergent field, and references. Word frequency of codes revealed that, in the opinions of HIS experts, the most frequent words to be included in the proposed definition are the following: Improvement (n=19), social (n=14), science (n=13), health (n=10), methods (n=10), healthcare (n=9), care (n=8), patient (n=8) and work (n=8).
In the second iteration, we tested the definition of HIS developed with the nominal group technique, where the results from the first Delphi iteration were used. The definition is worded in Table 4. The average level of agreement with the second proposed definition of HIS was higher (M=4.30) compared to the first iteration (M=3.93) (t=-3.006, p=0.004). The average portion of agreement in the first iteration was 78.5% and in the second 86%. For separate parts of the definition, the greatest agreement was expressed with the part (b).
Results for the level of agreement with the definition as a whole and with its separate parts (a - f), and differences between countries in the 2nd Delphi iteration (N= 66).
| M (SD) | Chi-Square (p) | |
|---|---|---|
| Agreement with the definition as a whole, from the part a to the part f | 4.30(0.61) | 21.227(0.002) |
| Healthcare Improvement Science (HIS) is focused on safety, effectiveness, efficiency, timeliness, equity and continuous patient-centred improvement in healthcare (a). | 4.29(0.86) | 7.727(0.259) |
| HIS depends on valid and reliable knowledge from a wide range of sources, in both the academic sector and service sector in healthcare (b). | 4.40(0.86) | 7.611(0.268) |
| HIS is multidisciplinary in its approach, drawing both on biomedical and on social sciences (c). | 4.33(0.81) | 11.468(0.075) |
| Different factors, such as methodological frameworks for improvement, knowledge of healthcare professionals, patient involvement and innovation, support organizational culture, are factors that interplay with the approach (d). | 4.14(0.96) | 14.904(0.021) |
| HIS is thus an umbrella term for any action (practice, education, science and policy) that can lead to improved healthcare outcomes for people, better health system performance and also healthier communities (e). | 4.26(0.83) | 11.226(0.082) |
| HIS also demands flexibility and responsibility in understanding, in theories of knowledge, and in the use of research evidence in health care improvement strategy (f). | 4.24(0.77) | 16.761(0.010) |
Legend: M – central value of a discrete set of numbers calculated by the sum of the values divided by the number of values; SD – Standard deviation; Chi-Square (p) – Kruskal-Wallis H nonparametric test for several independent samples; a, b, c, d, e – parts of the whole definition.
Comment: X2 (p) is mean value between countries.
Expertise in the field of HIS and respondents’ area of work did not significantly correlate with the agreement with HIS definition. Significant differences in the definition as a whole were established between partners (p<0.001). For example, respondents from Scotland expressed a significantly lower agreement with the definition as a whole (M=3.50, SD=0.53) and also with the part (f) (M=3.45, SD=0.93). Moreover, respondents from Scotland (M=3.36, SD=1.43) and England (M=3.20, SD=0.84) expressed a significantly lower agreement with the part (d). Detail results are in Table 5.
Results of countries look for the level of agreement with the definition as a whole and with its separate parts (a – f) in the 2nd Delphi iteration.
| N | M (SD) | England | Italy | Poland | Romania | Slovenia | Spain | Scotland | Chi-Square (p) | |
|---|---|---|---|---|---|---|---|---|---|---|
| Whole definition | 64 | 4.30 | 4.20 | 4.33 | 4.33 | 4.56 | 4.71 | 4.47 | 3.50 | 5.613 |
| 0.61 | (0.45) | (0.50) | (0.50) | (0.53) | (0.49) | (0.52) | (0.53) | (< 0.001) | ||
| Part a | 65 | 4.29 | 4.40 | 4.20 | 4.67 | 4.56 | 4.57 | 4.20 | 3.70 | 1.456 |
| 0.86 | (0.55) | (1.23) | (0.50) | (0.53) | (0.54) | (0.86) | (1.06) | (0.210) | ||
| Part b | 65 | 4.40 | 4.20 | 4.30 | 4.56 | 4.50 | 4.86 | 4.40 | 4.09 | 0.676 |
| 0.86 | (0.84) | (0.48) | (1.01) | (0.76) | (0.38) | (1.06) | (1.04) | (0.669) | ||
| Part c | 66 | 4.33 | 3.60 | 4.10 | 4.67 | 4.44 | 4.86 | 4.40 | 4.09 | 1.923 |
| 0.81 | (0.89) | (0.88) | (0.50) | (0.53) | (0.38) | (0.91) | (0.94) | (0.092) | ||
| Part d | 66 | 4.14 | 3.20 | 4.20 | 4.22 | 4.56 | 4.57 | 4.47 | 3.36 | 3.473 |
| 0.96 | (0.84) | (0.63) | (0.83) | (0.53) | (0.54) | (0.74) | (1.43) | (0.005) | ||
| Part e | 66 | 4.26 | 4.20 | 4.30 | 4.11 | 4.44 | 4.71 | 4.53 | 3.55 | 2.389 |
| 0.83 | (0.45) | (0.48) | (0.93) | (0.53) | (0.76) | (0.52) | (1.29) | (0.039) | ||
| Part f | 66 | 4.24 | 4.20 | 4.20 | 4.33 | 4.44 | 4.86 | 4.40 | 3.45 | 3.648 |
| 0.77 | (0.45) | (0.42) | (0.50) | (0.73) | (0.38) | (0.83) | (0.93) | (0.004) |
The consensus group technique started with a discussion of 18 experts from institutions involved in the ISTEW project. It was moderated by the principal investigator and research fellow for the project, with notes being taken. First, the aim was to increase the understanding of the first and second Delphi iteration results and to present partners’ qualitative analysis results. The consensus group discussion yielded 14 key themes of the HIS definition, which were merged into 3 content categories, namely ‘country differences knowledge’, ‘science and HIS,’ and ‘new dimension of HIS for practice’. The strongest extracted elements were knowledge, change, outcome, population, safety, efficiency, person-centred.
All partners agreed that the HIS definition had to be short and concise. The emphasis of each definition part and the theme from qualitative data was discussed until a consensus was reached. Based on this, the final consensus on the definition of Healthcare Improvement Science is as follows:
‘Healthcare improvement science is the generation of knowledge to cultivate change and deliver personcentred care that is safe, effective, efficient, equitable and timely. It improves patient outcomes, health system performance and population health.’
HIS is incorporated into every phase of healthcare proceedings and is, consequently, a complex concept to define. Coining a definition that encompasses the multifaceted nature of HIS with all of its variables is very challenging, particularly because the use of the term HIS and its synonyms in the existing literature and other educational documents (grey literature) varies depending on the country and language.
By applying consensus methods, the definition was developed gradually. According to the results of two Delphi iterations, a high consensus among participants was achieved. Consensus increased from the first iteration to the second, until 86% agreement was reached. It is very important that the definition was developed through interprofessional interactions. Only a third of respondents in the Delphi survey perceived themselves as experts in HIS, although the Delphi study coordinators identified them as such. This, we suggest, reflects the extent to which HIS is developed in the partner countries involved, as we found that HIS was not equally developed in all partner countries. Indeed, we found that the term itself is not used at all in Spain, and that the concept is not well developed in Romania, where the term implementation science with its different connotations tends to be more common. Respondents coming from countries where HIS was not broadly developed may have found it challenging to respond in an informed way. Respondents from Scotland, where HIS is a political priority, expressed a significantly lower level of consensus with the entire definition compared to non-English speaking respondents, but the level of agreement was still high at 70%. These differences may be related to a number of issues. Firstly, most respondents from Scotland defined themselves as experts in HIS. Secondly, in Scotland, HIS is a more crucial driver of healthcare institution policies compared to the other countries. Finally, research and accessibility of scientific and professional HIS literature is greater in the United Kingdom compared to other non-English native speaking countries (20).
The definition formed in two Delphi iterations was a broadened one. The consensus group technique greatly contributed to reaching a compromise between experts, so that a short and concise version of the definition was developed. The definition includes a wide range of key themes, such as knowledge for cultivating change, personcentred care, improvement of patient outcomes, health system performance and population health. These themes are also included in definitions proposed by other authors (4, 8). The employed method for developing a definition leads to a better understanding of practice (11).
Only half of respondents agreed that HIS-related national bodies play an important role in health and social care policies in their own country. However, a closer look at the data reveals that the situation is not comparable across countries. Only one third of respondents felt that HIS has an appropriate status in their national healthcare system. As a result, HIS indicators are not developed well and HIS is an uncommon term in some countries. This is a major issue because, as Booth et al. (21) stated, changing population demographics, disease patterns and new research findings demand change in healthcare and require new ways for ensuring continuous improvement. Robson (22) believes that in making healthcare safer, one should join all levels of the profession in an effort to clearly work with their working environment and patient safety.
For a continuous improvement, an understanding and recognition of the need for HIS are crucial. Our definition underpins the need for developing interprofessional study programmes and other materials. Hall and Zierler (23) stressed that, with the growth of interprofessional education and practice in healthcare schools, faculty members have to assume new roles in designing and delivering interprofessional curricula. Senior leadership support is also vital in ensuring sustainable change, and it will be a key requirement in healthcare organisations and universities to make healthcare improvement a priority.
The sample was not equally weighted for the nationalities of the HIS experts, and this may have impacted the Delphi process. Knowledge of the English language and familiarity with it may have also played a role in providing answers, in questionnaire translation, and in discussions within the nominal and consensus groups. In countries where English is not spoken as the official language, most partners had to first translate the questionnaire, then obtain answers from experts in their languages, and translate the responses back into English. This may have resulted in the loss of understanding and different interpretations. This study has only included seven European countries; consequently, it would be interesting to explore the ways in which HIS is understood, practised and taught across Europe as a whole.
A new consensus definition of HIS was developed in a European context. The consensus method approach was pertinent to unify the new concept from multidisciplinary, interprofessional and multicultural perspectives. Having a shared definition of HIS is important to boost healthcare culture and professional training processes. The definition enables higher education institutions in health and social care to address this gap in their curricula, and ensure that improvement science becomes a core competency for all healthcare graduates.