Cultural Adaptation of Patient-Reported Indicator Surveys (PaRIS) Patient and Primary Care Practice Questionnaires to the Slovenian Context

Patient-Reported Outcome Measures (PROMs) and Patient-Reported Experience Measures (PREMs) are crucial in evaluating healthcare quality from the patient’s perspective, particularly in chronic care management within primary care settings. PROMs provide insights into the patient’s health status, treatment outcomes and quality of life, while PREMs focus on patient experiences with healthcare services and healthcare professionals (1). Primary care has a significant place in the management of people with chronic conditions as it offers continuous, coordinated, people-centred and comprehensive care. Yet PROMs and PREMs data in primary care are scarce.

In large-scale international studies, ensuring international comparability is essential, through using the same survey design and instruments. However, adaptation of the survey and tools to the national context is important to address contextual issues (4). Structured and rigorous translation processes are key in international studies where the source instruments are developed in a foreign language, and ensure the validity and reliability of data collection. While using existing instruments has the advantage of established validity, a previously validated instrument may not be valid in a different time, culture, or context (2). Self-report scales can show variations due to culturally determined factors such as social desirability and response styles (3). Therefore, cultural adaptation, in addition to translation, becomes important to ensure content equivalence between source and target versions (3). There is no consensus on how to adapt instruments for different cultural settings, but simple translation is insufficient, as it can result in multiple, inconsistent translations, losing the original intent (4). Rigorous translation and cognitive testing are two important steps in ensuring that instruments such as PROMs and PREMs are appropriately adapted to different cultural contexts, preserving their original intent (7, 8).

The OECD’s Patient-Reported Indicator Survey (PaRIS) (5) aims to measure the performance of health systems in delivering high quality safe primary care by collecting PROMs and PREMs. It assesses the outcomes and experiences of the healthcare of people living with chronic conditions aged 45 and above through a patient questionnaire (PaRIS-PQ) (2) and through a primary care practice questionnaire (PaRIS-PCPQ) (3) which collects information on the characteristics of care such as delivery design and type of practice (e.g., group practice). Based on the PaRIS conceptual framework, PaRIS-PQ and PaRIS-PCPQ were developed in English and French through a rigorous process of review of literature, modified Delphi process and stakeholder consultations from existing validated instruments (10–13).

The aim of this study was to adapt the PaRIS-PQ and PaRIS-PCPQ to the Slovenian setting and language. The study focused on cognitive testing to ensure the questionnaires’ comprehensibility, relevance and effectiveness in the national context.

Cognitive testing of the PaRIS-PQ and PaRIS-PCPQ have significantly improved the clarity and user experience of the PaRIS questionnaires in the Slovenian context. The assessment identified beneficial changes for 30 out of 120 PaRIS-PQ items and for 23 out of 34 PaRIS-PCPQ items, improving comprehension of the questionnaires through changes including question design, translation accuracy, cultural relevance and sequence design. Clearer terminology was introduced, replacing terms that were not well understood, and adjustments were made to simplify wording and reduce complexity in the questionnaire.

Cognitive testing identified “Poor source question design” as a significant issue. Despite using validated questions, cognitive testing highlighted issues with how these questions functioned in the context of the Slovenian survey. These problems suggest that even validated instruments can fail in new cultural or linguistic contexts. This finding is very important since cognitive practice often focused more on checking specific question understanding, rather than how a question plays out in the context of an entire questionnaire. This indicates that some questions were not effectively constructed, leading to confusion among respondents. This was also shown in other participating countries and was the most prevalent type of error in PaRIS-PCPQ international cognitive testing (3). In Slovenia, most issues associated with poor source question design were linked to cultural and contextual differences. Such issues can lead to biased results. The report notes the importance of cultural adaptation beyond mere translation. Poor design was often linked to a lack of cultural portability, underscoring that validated instruments require contextual refinement to maintain their integrity across diverse populations. The issues identified during cognitive interviews (e.g., confusion or misunderstanding) were pivotal in refining the questions. This indicates that validation in prior studies does not guarantee flawless application in all scenarios, particularly in multilingual and multicultural studies. Maintaining international comparability while addressing local context is a complex challenge. Some of the poor design elements may have persisted because questions were developed with an international audience in mind, which can dilute their relevance or clarity in specific national settings. It is essential to adapt questions to fit the cultural context of the target population. This includes considering local terminologies, customs and healthcare practices. Misunderstandings due to cultural discrepancies can significantly impact the validity of the data collected (11).

Both questionnaires encountered translation errors, which can distort the intended meaning of questions. Similar issues were observed in international cognitive testing (3). Ensuring accurate translations that preserve the original meaning is essential, as well as consultation with native speakers (12, 13).

Some questions did not translate well across different cultural contexts, an issue known as cultural portability which was found as a prevalent issue in the international cognitive testing (3). This underscores the utmost importance of culturally sensitive adaptations that respect local contexts and terminologies (12, 13).

The cognitive testing highlighted issues with the design of introductions and the sequence of questions. Long and complex introductions were particularly problematic, especially in languages such as Slovenian that require gender-specific language. On the contrary, at the international level, this was the least prevalent issue (3). This finding underlines the importance of continuous testing and refinement of the terminology used in large-scale surveys as well as adaptation to national contexts while ensuring international comparability of results (12). The findings of this study confirm that cognitive interviewing in addition to translation is a crucial step in the development of questionnaires and the adaptation process, especially for developing PROMs and PREMs (14, 15). In large-scale initiatives, it is even more important to do cognitive testing in local languages, since even if the survey instruments used are validated in other contexts, it does not necessarily mean that they have the same face validity in all participating countries. This is why by design, PaRIS ensured that all countries followed a rigorous translation and cognitive testing process, meaning that the final questionnaires were understood by all participating patients, providing comparable data.

Cognitive testing is the most direct way of finding out whether respondents understand questions consistently, have the information needed to answer the questions and can use the response alternatives provided to describe their experiences or their opinions accurately. It does not directly assess the validity of the answers, but cognitive problems will seriously compromise validity and reliability (16, 17). Therefore, we can conclude that cognitive interviewing ensured that the Slovenian version of both PaRIS questionnaires had much better face validity.

Our study has some limitations that must be addressed. Cognitive testing was performed only on specific questions - those questions that were identified and pre-selected for testing in Slovenia by the PaRIS consortium. However, the cognitive complexity of sentences may be different in different languages (e.g. some wording in English sounds less complex than in Slovenian). Also, the testing was adapted to the conditions of the individual interview (time duration, fatigue etc.), therefore not all questions were tested on all participants.

Additionally, during cognitive testing, some participants had difficulty developing rich metacognitive thinking (awareness of their own thoughts and mental processes), which is an important factor in effective cognitive interviewing. Despite encouragement, some participants could not consistently think aloud and were unable to reflect much despite verbal probing. Furthermore, cognitive interviewing was not performed face-to face, but online (GoToMeeting (video) conference) and by telephone due to COVID-related restrictions, which limits the ability to observe non-verbal communication.

Finally and importantly, cooperating in this international cognitive interviewing study was beneficial since it ensured that the questions which were further tested in the PaRIS survey Field Trial were similar in different countries, which ensures good final data comparability.

This study describes the cognitive testing of the international PaRIS-PQ and PaRIS-PCPQ in the Slovenian language and in a sample of Slovenian primary care providers and patients. To our knowledge, this is the first time PREMs or PROMs for primary care have undergone cognitive testing in Slovenia, marking a critical step toward better aligning healthcare evaluation tools with local cultural and linguistic contexts. Identified errors enabled the questionnaires to be adapted to the Slovenian context, while keeping consistency with international standards for cross-country comparisons. Through a rigorous process of translation and cognitive testing, we obtained culturally adapted PREMs and PROMs measures in the Slovenian language for family medicine practices, ensuring better validity for any future studies that also use them.