Patient Reported Indicator Surveys (PaRIS): Methodological Considerations of a Field Trial in Slovenia
, , e | 03 gen 2024
Article Category: Original scientific article
Pubblicato online: 03 gen 2024
Pagine: 30 - 37
Ricevuto: 11 ott 2023
Accettato: 27 nov 2023
DOI: https://doi.org/10.2478/sjph-2024-0005
Parole chiave
© 2024 Matija AMBROŽ et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.
Healthcare systems that place greater emphasis on the quality of primary care have better clinical outcomes and are more economically efficient and sustainable [1–5]. The patient’s experience with the healthcare system is being recognised as an important dimension of the quality of healthcare. It is thus an increasingly common strategy to monitor the quality of healthcare at all levels [6–9].
Tools that assess patient experience enable patients to provide feedback and potentially strengthen a patient-centred health system [10].
A standardised data collection procedure is preferred in international surveys. Due to a wide variety of legal, ethical and cultural differences in primary care, a data collection strategy is adapted for each country in agreement with the consortium [11]. To date the largest international research on the quality of primary care (Quality and Costs of Primary Care in Europe (QUALICOPC)) was carried out in 31 countries between 2010 and 2013, and it included assessment of the patients’ experiences [12]. This study was crucial in developing the methodology for future large-scale international studies such as the Patient Reported Indicator Surveys (PaRIS).
In 2017, the OECD launched the PaRIS initiative to systematically measure the outcomes and experiences of healthcare that matter most to people [13]. The flagship project within the PaRIS initiative is a survey of people living with chronic conditions who are managed in primary care, namely the PaRIS survey [11]. The PaRIS survey has seen three different phases: 1) design and development phase (2017-2020), including the study design and the development of PaRIS questionnaires; 2) field trial (2021-2022), including the testing of the PaRIS survey design and PaRIS questionnaires in participating countries; and 3) main survey (2022-2023), including the full implementation of the PaRIS survey in participating countries and data collection [14].
In this paper, we aimed to evaluate the feasibility of the methodology employed in the field trial of the PaRIS survey in Slovenia and propose adjustments to enhance sampling in the main survey. While the manuscript does not present content-related field trial results, it provides a comprehensive assessment of the survey’s methodology.
The PaRIS survey is a cross-sectional observational study, and the study protocol is described in detail elsewhere [11]. The study presented here took place in Slovenian family medicine practices. Family medicine physicians work either as private contractors or in primary care centres, and both are part of national public primary care system. One family medicine team consists of 1 full time equivalent (FTE) family physician, 1 FTE practice nurse and 0.5 FTE registered nurse. Practice nurses are mainly involved in tasks including administration, appointments, and clinical work, while registered nurses work as nurse practitioners and carry out preventive activities and care for stable chronic patients [15, 16].
We defined providers as primary care practices in line with the study design of the PaRIS survey. In Slovenia one family physician works in one family medicine practice, and the registry of family medicine practices is available online, with the system led by the Health Insurance Institute of Slovenia (ZZZS). We used this registry for random sampling, by assigning consecutive numbers to every physician in the registry and then using a random number generator to select them. The registry contains all family physicians working primarily with registered patients. Physicians that provide healthcare only to institutionalised patients or children, or who practice occupational healthcare services as their primary area of work, were excluded. There were 978 eligible family medicine physicians in Slovenia in February 2022.
As this was a field trial, we aimed for 25 family medicine practices in a final sample. Based on previous studies, we estimated at least a 50% positive response rate from the contacted family physicians, and thus to achieve the target we invited 50 practices to participate.
All patients aged 45 years old or more, living in the community and having had any contact (including via phone or email) with the practice in the six months preceding the study, were eligible to participate. People living in a long-term care facility, healthcare or other residential institution were excluded, as were those in the terminal phase of illness or with an advanced cognitive impairment.
Sampling of patients was random, and we aimed at 70 patients per practice. Practices autonomously generated a list of eligible patients, utilising either a random number generator or systematically selecting every Nth patient. Each eligible patient was first contacted by the practice and given the necessary information about the study, and could then provide informed consent and contact data if they wanted to participate. After that, the information gathered by the practice was sent to the researchers.
To collect data, we used the PaRIS provider and patient questionnaires.
The provider questionnaire includes 40 questions and defines the structure and characteristics of the team providing primary care and of the provider responsible – the family physician. The questionnaire was developed by the OECD and an international consortium (PaRIS-SUR consortium) in collaboration with patients and provider organisations, including the World Organization of Family Doctors (WONCA), and other stakeholders [11]. The questionnaire covers domains related to care delivery such as urbanisation, practice type, care model, information and administration systems, remote consultations, skillmix, and remuneration. The estimated time for completion is 10-15 minutes.
The patient questionnaire includes 121 questions addressing individual and sociodemographic factors, health and healthcare capabilities, health behaviours and lifestyle, Patient Reported Outcome Measures (PROMs) such as symptoms and functioning, and Patient Reported Experience Measures (PREMs) such as coordination of care and safety. The PaRIS patient questionnaire is based on the PaRIS conceptual framework and consists of validated and established tools, already described elsewhere [11]. The estimated time to complete the patient questionnaire is 25 minutes.
The PaRIS questionnaires has two source questionnaires in English and French. The base for developing the Slovenian questionnaires was the original English versions developed and cognitively tested by the PaRIS consortium. We first translated the original English questionnaires into Slovene using the TRAP-D approach, as recommended by the guidelines [17]. The resulting questionnaires were then cognitively tested using the four-stage model of cognitive testing developed by Tourangeau [18].
Cognitive testing was only performed on questions that were identified as necessary for testing by the consortium. Changes were incorporated in the final version of the field trial questionnaires.
After acquiring the sample of providers those who were selected were contacted via post in which they received information about the study and an invitation to participate. Within a week after sending the invitation letter, all providers were contacted again via phone by the research team. Those who agreed to participate received a link to the survey for providers and a spreadsheet to sample their patients. The practitioners had a month to sample 70 eligible patients. They were reminded weekly by the research team via email to complete the survey and provide the completed spreadsheet (Figure 1).
Figure 1.
Schematic presentation of the process of recruiting of family physicians and patients.
Symbolic incentives were used to motivate physicians to participate, such as one-time free registration to national congresses of primary care and feedback information for their team.
The patients in the sample were contacted via their preferred method. All patients who provided an email address received the survey link via email. Those who did not provide an email address received the survey link via SMS. Those who did not provide either an email or telephone number, or who opted for a paper survey, received this via the regular mail system. Non-respondents who provided an email address received a reminder via email for two consecutive weeks after receiving the link to the survey, and those who did not give an email address received two reminders via regular mail.
For the quality check of the samples, we identified the characteristics of the provider and patient population and performed chi-square and independent samples t-test. For the providers, we used the data from the aforementioned list of practices. We performed the sample check according to geographic distribution (10 health regions) and status of the providers (public or private with a concession). For the patients, we used the data from the participating practices with regard to the age and gender of their registered patients who were age 45 years or older.
For the evaluation of data collection, we used the following variables:
Mode of completion, participation time for the e-survey, number of restarts with the e-survey, method of access to the e-survey, return code (partial/complete), and type of device used to complete the e-survey. The appropriateness and feasibility of the methodology were assessed using predefined criteria, including achieving at least 80% of the target sample size and a response rate exceeding 30%.
The research team systematically documented reasons for non-participation by directly querying both providers and patients.
On the providers’ side, 21 family medicine practices of the 50 sampled practices participated in the study (42.0%).The family medicine practices managed to recruit 896 patients, out of which 454 responded to the patient questionnaire (50.7%) (Figure 1).
Most providers that completed the questionnaire were family physicians, working in the public sector, located in a city and in a solo practice (Table 1).
Characteristics of the providers.
| Characteristic | N | % |
|---|---|---|
| Family physician | 18 | 85.7 |
| Family medicine trainee | 3 | 14.3 |
| Public (in a primary care centre) | 10 | 47.6 |
| Private with a concession | 3 | 14.3 |
| No answer | 8 | 38.1 |
| City | 9 | 42.9 |
| Town or suburb | 5 | 23.8 |
| Rural area | 7 | 33.3 |
| Solo practice | 16 | 76.2 |
| Group practice with own patients | 4 | 19.0 |
| Group practice with shared patients | 1 | 4.8 |
Most patients were female, 60 years of age and older, with secondary education, retired and living in a rural area (Table 2).
Characteristics of the patients.
| Characteristic | N | % |
|---|---|---|
| Male | 170 | 37.4 |
| Female | 249 | 54.8 |
| I prefer not to say | 2 | 0.4 |
| Not answered | 33 | 7.3 |
| 44 years old or younger | 2 | 0.4 |
| 45-49 years old | 32 | 7.0 |
| 50-54 years old | 45 | 9.9 |
| 55-59 years old | 69 | 15.2 |
| 60-64 years old | 74 | 16.3 |
| 65-69 years old | 95 | 20.9 |
| 70-74 years old | 61 | 13.4 |
| 75-79 years old | 27 | 5.9 |
| 80-84 years old | 13 | 2.9 |
| 85 years or older | 4 | 0.9 |
| Not answered | 32 | 7.0 |
| Primary school not completed | 9 | 2.0 |
| Primary school | 52 | 11.5 |
| Vocational school | 123 | 27.1 |
| Secondary school | 230 | 50.7 |
| University education | 19 | 4.2 |
| Master’s degree or doctorate | 54 | 11.9 |
| Not answered | 35 | 7.7 |
| Self-employed | 14 | 3.1 |
| In paid employment | 150 | 33.0 |
| Looking for work | 4 | 0.9 |
| Looking after the home | 7 | 1.5 |
| Unable to work due to sickness or ill-health | 16 | 3.5 |
| Retired | 227 | 50.0 |
| Other | 2 | 0.4 |
| Not answered | 34 | 7.5 |
| City | 105 | 23.1 |
| Town or suburb | 69 | 15.2 |
| Rural area | 245 | 54.0 |
| Don’t know | 1 | 0.2 |
| Not answered | 34 | 7.5 |
The size of the provider population was 978 and the size of the sample was 50. There was no significant difference between the sample and the population regarding geographical distribution and status of the practice (Table 3).
Provider quality sample check.
| Characteristic | Number (sample vs. population) | Pearson chi-square | Degrees of freedom | p value |
|---|---|---|---|---|
| 0.151 | 9 | |||
| Region 1 | 5 vs. 100 | |||
| Region 2 | 4 vs.74 | |||
| Region 3 | 2 vs. 33 | |||
| Region 4 | 5 vs. 100 | |||
| Region 5 | 15 vs. 303 | |||
| Region 6 | 7 vs. 142 | |||
| Region 7 | 3 vs. 55 | |||
| Region 8 | 3 vs. 53 | |||
| Region 9 | 3 vs. 56 | |||
| Region 10 | 3 vs. 62 | |||
| 0.009 | 1 | |||
| Public | 36 vs. 710 | |||
| Private with a concession | 14 vs. 268 |
The sample and population of patients differed significantly regarding gender and age (Table 4).
Patient quality sample check.
| Characteristic | Number (sample vs. population) | Pearson chi-square | Degrees of freedom | p value |
|---|---|---|---|---|
| 6.954 | 1 | |||
| Male | 170 vs. 84944 | |||
| Female | 249 vs. 95749 | |||
| 98.120 | 8 | |||
| 45-49 years old | 32 vs. 19672 | |||
| 50-54 years old | 45 vs. 11343 | |||
| 55-59 years old | 69 vs. 15712 | |||
| 60-64 years old | 74 vs. 14526 | |||
| 65-69 years old | 95 vs. 14544 | |||
| 70-74 years old | 61 vs. 12801 | |||
| 75-79 years old | 27 vs. 16070 | |||
| 80-84 years old | 13 vs. 8791 | |||
| 85 years or older | 4 vs. 7378 |
All the providers completed the survey online. They mostly used a desktop computer and accessed the survey via a link sent by email (Table 5). The providers completed the e-survey in 20.9±11.1 minutes and had 1.5±1.5 restarts.
Data collection evaluation – providers and patients.
| Characteristic | Providers N (%) | Patients N (%) |
|---|---|---|
| Online | 21 (100) | 340 (74.9) |
| Paper | 0 | 114 (25.1) |
| Link via email | 21 (100) | 199 (58.5) |
| Link via SMS | 0 | 141 (41.5) |
| Desktop | 18 (85.7) | 127 (28.0) |
| Smartphone | 2 (9.5) | 195 (43.0) |
| Tablet | 1 (4.8) | 17 (3.7) |
| Unknown | 0 | 1 (0.2) |
| Complete | 21 (100) | 420 (92.5) |
| Partial (incomplete e-survey) | 0 | 34 (7.5) |
Most patients completed the survey online (e-survey), others used a paper-based questionnaire. They mostly accessed the e-survey via a link sent by email and used a smartphone to do so. Only a small percentage of patients left the e-survey incomplete (Table 5). The completion time was 36.0±22.6 minutes and the mean number of restarts was 1.4±2.2.
Some of the providers refused to participate in the study due to the workload expected from them regarding the sampling and data collection. They stated that they were overburden with their usual work and did not have time to engage in the survey.
Some of the patients refused to participate due to low interest in the survey, the questionnaire being seen as too long for them, and not having enough time to complete it.
The evaluation of the PaRIS survey field trial in Slovenia showed satisfactory results regarding the sampling of providers and data collection among them. However, the sampling of the patients and data collection among them did not yield the desired evaluation outcomes, the main problem being the small sample size.
The response rate of the practices was satisfactory. The literature shows that the response rate can vary significantly (from 20-80%) [19–21], although there are several methods that can be used to improve it [14, 20]. We used personal contact and reminders, which seemed to be effective.
The survey was mostly completed by family physicians. This was expected, as the physician is a team leader in primary care practices in Slovenia, and access to the patient list through the physician was required for further sampling. Regarding the work type of the practice, most providers were in a solo practice. Here, it should be noted that in Slovenia around 70% of the family medicine practices are in community health centres, and 30% work as private practices with a concession [22].
Family physicians in health centres are practicing in their own practice with their own registered patients, working with a team of their own nurses. However, a health centre composed of a family physician, practice nurse and registered nurse is not considered as a group practice in Slovenia. A group practice in Slovenia is described as a practice where two or more physicians are working in the same practice with other healthcare professionals [23]. Hence, what is considered to be a group practice in the international PaRIS survey was recorded as solo practice in Slovenia.
The sample size of the patients was much smaller than anticipated, which is probably a consequence of how the GDPR is interpreted in Slovenia. With this, practices are not allowed to send the contact information of their patients to researchers without the patients’ consent. This forced the researchers to adapt the international methodology, and to ask the practices to both sample the patients and acquire their informed consent. As expected, this placed a significant additional burden on the practices, which then failed to provide a sufficient patient sample size.
All the providers accessed the questionnaire online, and in the literature no significant differences were found regarding the mode of survey [19, 21]. The survey was completed by the providers in 20 minutes, whereas we anticipated 15 minutes would be needed. However, we still consider this feasible.
A quarter of the patients completed a paper questionnaire. Others completed an online survey via link sent by email or SMS. A meta-analysis showed that the response rate to online surveys is approximately 40% [23], and this is consistent with our results. On the other hand, the literature shows that using a paper questionnaire does not yield a lower response rate [24]. This might indicate that paper questionnaires can now be dropped, but this would be a mistake as some respondents still prefer paper versions [24]. Patients on average needed 36 minutes to complete the online survey, while there is no data on the time needed for the paper questionnaire. We expected that the online survey should be completed in 25 minutes, which turned out to be rather optimistic. The literature shows that questionnaires should include 25 to 30 questions, and be possible to complete in 30 minutes [24, 25]. In the case of longer questionnaires, we can expect a larger percentage of non-responders, a larger amount of missing data, and an increase in providing answers automatically with the last items [24, 25]. In our study, only a small number of patients left the questionnaire incomplete, which indicates that although the questionnaire was long, it was still manageable.
One limitation of the study was that we could not evaluate the same indicators on the paper-based survey. It is thus possible that we missed some lessons for the main survey. Another limitation is the missing characteristics of the population and sample, which would have helped with regard to detecting additional significant differences. Moreover, the patient sample did not reflect the characteristics of the population, and there was also a problem with sampling according to inclusion criteria, as two patients were younger than 45 years old, although this could be due to an error when completing the questionnaire. Therefore, we cannot assume that the patient sample was representative, and one solution might be to omit this age category within the questionnaire.
We evaluated the sampling method of the practices as satisfactory, so no adjustments will be made for the main survey. The same is true for data collection of the practices.
For patient sampling, the evaluation indicated the need for revision of the methodology to achieve a better response rate. We are thus planning to provide the practices with printed survey invitations for patients containing a QR code with the link to the online questionnaire. This seems to be a good method of data collection [26]. We also plan to provide the practices with paper questionnaires. The practices will draw up a patient sample as in the field trial, but will not need to obtain the patients’ informed consent for sending the contact information. Instead, the practices will mail either the QR code or a paper questionnaire to the patients in the sample. With such methodology, we expect a lower burden on the practices and higher response rate. On the other hand, such an approach will make it impossible to only send the reminders to the nonrespondents.
Shortening the patient questionnaire might also contribute to better response rates.
This study provides insights into the methodological challenges when conducting a cross-sectional observational study on patient outcomes and experiences in Slovenian primary care. The results suggest a need for pre-testing the methodology of quality assessment studies before the main survey, enabling researchers to refine their approaches for optimal results. Early consideration of legal issues in methodology development is essential. Additionally, awareness of study feasibility is crucial to mitigate low participation rates.