Psychometric Validation of an Instrument for Measuring Patient Experiences with Outpatient Healthcare

The questionnaire was developed in several different stages: A literature review, expert evaluations and discussions with Steering Committee members, focus groups with patients and cognitive interviews were conducted to identify key aspects of healthcare from both provider and patient perspectives, and to test patients’ understanding of the items (15). The Anglo-American and Scandinavian literature was searched for aspects of patient experiences of relevance in an outpatient setting, such as admission, privacy, work of doctors, work of nurses, information about patient rights and continuation of treatment and overall score (9,10,11,12, 16). The questionnaire was piloted in three different settings after which further changes were made to improve the content validity of the items.

The study was conducted from June 1 to June 30, 2020, as a cross-sectional survey using a paper or online questionnaire. The patients surveyed received specialist healthcare in 171 specialist clinics in the following medical fields: neurology and neurosurgery, cardiology, ophthalmology, gynaecology and obstetrics, and orthopaedics. At the end of the specialist visit, the nurse handed each patient a letter with a paper questionnaire and a prepaid envelope and invited them to participate in research. The patient also had the option of completing an online survey, for which they received a one-time password. Out of the total 8,406 participants, 7,877 (94%) completed the paper survey, and 529 (6%) completed the online survey. Patients were also given information about the study, including a statement that participation was voluntary and anonymous, and that their responses would never be linked to them as individuals. Participation or nonparticipation in the study did not affect patient’s treatment or healthcare. No reminders were sent.

Of the 8,406 patients who participated in the study, 68.0% were female and 32.0% were male. The two largest age groups were those aged 65 to 74 years (25.0% of all respondents) and 55 to 64 years (21.1%). More than 60.0% of participating patients had completed high school or above. A total of 53.8% of respondents rated their own health as good or very good, and 7.8% rated it as poor or very poor, while the rest of the patients rated their health as average. A total of 45.1% of the patients surveyed had a chronic disease. The response rate was estimated at roughly 31%. A comparison of the sociodemographic characteristics of the respondents and all patients having an eReferral for a specialist clinic for the time the survey was conducted, based on the eReferral database, revealed the predominance of female and relatively young respondents (further details are presented in the online supplement: https://osf.io/4guch/?view_only=1c3474c36e95481b959dd22a2cd03460).

Returned paper questionnaires were manually entered into the database via the input mask. To check the quality of data entry, 10% of all questionnaires were double-entered. In the case of an online survey, the data was recorded in the database automatically. Forty-five records with a completion-rate less than 50% were excluded from the analysis.

Patient experience is not a unitary construct, so we tailored the analyses according to the characteristics and content of the items. The items that did not rate the patient experience were not relevant to the analysis. These included:

questions asking for general information, that did not assess patient’s experience, including age, education, presence of chronic disease, reasons for visiting the doctor etc. (questions Q1, Q2, Q31-Q34), and

The questions related to the doctor’s (Q12–Q17) and nurses’ work (Q21–Q23) were analysed as homogeneous psychometric scales. We used confirmatory factor analysis with the WLSMV estimator with pairwise estimation to assess the dimensionality of the scale and the discrimination power of the items. The “not applicable” (NA) responses were treated as missing values in these analyses. Item Q16 is a group of four separate ratings, so they were analysed as separate indicators. Because they are all related to the doctor’s explanations, we allowed correlated residuals between these four items. For each scale we computed two measures of score accuracy: the coefficient omega, indicating the proportion of the scale score variance, attributable to the common factor, and the coefficient alpha, a standard lower bound to the scale score reliability. After the factor analysis, we performed a Rasch analysis, based on the partial credit model for ordinal responses. In addition, we applied a unidimensional variant of the optimal scaling, called correlational aspects optimization. In this analysis, each response category is assigned a scale value so that the weighted sum of the items is as unidimensional as possible.

To assess the fit of the factor analysis model, we used the criteria proposed by Hu and Bentler (17): RMSEA<.06, CFI>.95, SRMR<.08. For the Rasch analysis, we used the criterion value of 1.3 for the INFIT and OUTFIT statistics (18).

The significance level of α=5% was used for all statistical tests.

For the remaining items, standard psychometric analysis was not meaningful, either because they were standalone measures (for instance, a general evaluation in item Q27), or because an item was presented conditionally on the response to the preceding items (for instance, the waiting time and receiving explanations for a prolonged waiting time in items Q7 and Q8). The frequency distributions of the responses were calculated for these items, and their validity is based on their content. In examining the frequency distributions, we checked that the proportions of missing values and NA responses were reasonable (it should be noted that such an evaluation depends on the item content; for instance, a very low proportion of NA responses should be expected for item Q12, but less so for item Q14), and that no unusual frequency patterns emerged.

The main psychometric analyses (factor analysis, coefficient omega, and Rasch scaling) allow the presence of missing values. Only for the computation of the coefficient alpha and for the optimal scaling did we impute the missing values using the random forest algorithm.

All analyses were performed in R (19). We used the packages lavaan (20) for the confirmatory factor analysis, MBESS (21) for the calculation of coefficient omega, eRm (22) for the Rasch analysis, aspect (23) for optimal scaling, psych (24) for general descriptive statistics, and missForest (25) for missing data imputation.

A major strength of our study is the large sample size, which implies more stable statistical estimates, including more accurate estimates of reliability coefficients, factor loadings and other model parameters. Since the instrument was developed in Slovenia, rather than being a translation of a foreign instrument, it may be assumed to be better adapted to the needs of the patients in the Slovenian healthcare system.

However, some limitations should be kept in mind when interpreting the results. Although all patients treated in the selected period were invited, it was not possible to control whether a nurse handed the invitation to participate in the survey to each patient, as foreseen in methodology. Since the participation was voluntary, different subgroups of patients could participate to a lesser or greater. The patients who were asked to participate the survey but chose not to might have answered differently to our questions. This means that sampling was not probabilistic in the strict sense. In the case that non-participants differed systematically from study participants, the final sample may suffer from selection bias. In the future, legal provision for random individual patient sampling from eReferral and e-SZBO (outpatient specialist visit database) is planned to avoid or at least reduce this limitation.

The relatively low response rate which might have influenced the composition of our sample is another limitation of the study. This is not uncommon in PREM studies with a similar methodology (i.e. inviting patients without additional reminders), where it is very difficult to achieve high response rates (31, 32). For example, in a Dutch study with a consumer quality index as a standardized survey method to measure patient experience of chronic dialysis care had a combined response rate that was higher, at 48%, but the methodology allowed for up to three reminders where necessary (33).

In our study, older participants and especially male participants (see online Appendix A, Table A1) were underrepresented in the sample, compared to the total sampling frame. While the size of the sociodemographic differences between the sampling frame and the actual sample is not dramatic, these deviations, in combination with a possible selection bias, limit the generalizability of the results. In particular, the average scale values and other descriptive statistics should be taken with caution; for instance, an NHS study carried out in the UK (34) reports that the underrepresented groups tend to be the ones with more negative experiences of care. On the other hand, the focus of this study was not the estimation of the (average) experiences in the population of patients, but the psychometric validation, and the psychometric indices are not affected by small or moderate shifts of average values. Response rates are reported to be only weakly associated with non-response bias in surveys that adhere to high standards of survey methodology (35), and surveys with response rates typical of those in public sector surveys (e.g., 35–40%) are often regarded as acceptable for the purpose of routine healthcare monitoring (36).

Similar problems were encountered in other comparable studies. For example, a Norwegian study (37) found that non-respondents were more likely to be younger and male. In the British NHS study (34), participants in online the surveys tended to be younger and better educated than participants who respond by other survey methods. These authors thus recommended the use of different alternative completion methods to mitigate the non-response bias and achieve a representative sample. In our study, we used a combination of online and paper surveys to improve the response rate and the representativeness of the sample. In this regard it should be noted that the use of the different data collection modes may also introduce biases. The phenomenon that the data collection mode affects the responses of participants is known as the measurement bias (or the lack of the measurement invariance) in the psychometric literature (38). Measurement bias can be caused by differences in design, perceived social desirability of responses and other irrelevant factors. This is a serious problem because it can undermine the comparability of responses of persons responding participating in different data collection modes. In our case, we do not see any reasons to expect a notable measurement bias: the responding in both data collection modes was anonymous and performed at home. The graphical designs of the online and paper surveys were also made as similar as possible. The previous PREM studies that were available to us did not report measurement bias in relation to the paper vs. online survey methods. Nevertheless, we compared the distributions of scale scores and the correlational structure in both sub-samples and found that they were practically the same, supporting the assumption of the measurement invariance. For details see the online supplement (https://osf.io/4guch/?view_only=1c3474c36e95481b959dd22a2cd03460).

The “not applicable” responses often cannot be avoided. The composite scores cannot be meaningfully calculated for patients who gave NA responses; in such cases, the responses should be interpreted at the item level. For research and evaluation purposes imputation techniques may be used. In such cases it should be remembered that the imputed values are only approximations used to obtain an estimate of the general assessment. In performing the analyses, we compared the results with respect to various treatments of missing data (pairwise estimation, listwise deletion, and imputation using the random forest technique) and found no notable differences.

There is a large variability in both the number and type of validity and reliability testing undertaken for the PREM instruments (6). We opted for factor analysis and the Rasch model. The results of the factor structure and the reliability of the DW and NW scales, and, in a more indirect manner, the distributions of the responses on the evaluation items not grouped into scales, suggest that the instrument is able to adequately capture patient reported experiences in intended clinical settings and populations. In conclusion, our study is an important step forward in the implementation of a long-term system for monitoring patient experience. Patient sampling will be improved in the future by using eReferral and e-SZBO databases as sampling frames. This would reduce the burden on healthcare providers (e.g., delivering envelopes), while giving us better control over the distribution of the questionnaires, increasing the quality of the data obtained. The response rate may be additionally improved by establishing an online platform, where the data collected through PREM surveys would be published and thus be useful to patients.