Orientation and Mobility Client Evaluation Tool (CET) Adult

There has been considerable anecdotal evidence reporting that people with vision impairment often appear more skillful, mobile, and confident as a result of orientation and mobility (O&M) training (Edwards, 2002; Howe, 1872; Lloyd, La Grow, Stafford, & Budge, 2008; Malamazian, 1970; The Seeing Eye, 2007, as cited in Franck, Haneline, Brooks, & Whitstock, 2010). However, proving this has been somewhat difficult. Deducing the way to measure O&M outcomes that can be applied to many types of mobility programs (e.g., Guide Dog, GPS, neurological) and administered by O&M specialists in a succinct, easy to use, and meaningful way has been challenging. Only in recent years have O&M outcome tools begun to be adequately researched and developed (La Grow, Ebrahim, & Towers, 2013-14; La Grow, Towers, Kim, & Haneline, 2015).

The importance of measuring client outcome cannot be overemphasised. It is central to demonstrating whether or not an O&M program benefits a person with vision impairment (National Resource Center, 2010, p. 10). Outcome measures can be used to report on the quality of the training provided and as such, could be an essential reporting item to government, funding agencies, corporate supporters, major donors, and stakeholders to secure the future of O&M services and the organisations that provide them. In an increasingly competitive O&M service environment (like that which currently exists in Australia) applying O&M outcome tools to programs will assist to identify quality service providers.

To date, only two O&M outcome measures have been developed, piloted, and validated albeit using small sample sizes. First, La Grow, Ebrahim, and Towers (2013-14) developed a 23-item difficulty with mobility questionnaire (DMQ) involving 32 participants. The DMQ is based on the premise that the purpose of O&M is to restore a person’s ability to travel around environments using adaptive aids, and a range of techniques and strategies, thereby enabling reduced difficulty with the 23 items (e.g., crossing busy streets with pedestrian control devices; getting around in the immediate neighbourhood; getting around in supermarkets; negotiating parking lots; travelling in unfamiliar outdoor environments; using public transportation). The DMQ assesses the person’s perception of difficulty performing each of the tasks before and after O&M training. Clients are also asked to rate their ability to get around (AGA). The DMQ is intended to be an indicator of O&M specialist effectiveness although this version is not reflective of a complete mobility program, but rather only aspects of one. The authors stated that they intend to investigate the feasibility of developing a briefer version of the DMQ for actual O&M program use validated on a larger sample of participants.

Second, in a later study, the DMQ was tested as an outcome measure for Dog Guide instruction involving 49 participants (La Grow, Towers, Dae Kim, and Haneline, 2015). The authors did acknowledge the few limitations of the study, for example, their use of convenience sampling and the variation of intervention in the content between first time dog guide users and those training with a replacement dog.

In 2011, a small project team at Guide Dogs NSW/ACT commenced the development of a client evaluation tool (Appendix A). The major motivation for its development was to increase results-based accountability that would feed into the organisation’s key performance indicators, assist strategic direction, and increase service outcomes for clients. The initial project team comprised an external consultant, five clients, four O&M specialists, and an allied health specialist with all specialists having 20 or more years of experience, who were also managers, and researchers. The project team deduced that an effective tool must be administered pre- and post-training, apply to all types of O&M programs, take into account the judgement and feedback of O&M specialists and clients; be quick and easy to use; and consider the complexity of clients who have additional disabilities, and the complexity of the training environment (Deverell, 2011). Information about these components was to be gathered during the client assessment and/or during the initial demonstration of mobility skills by clients prior to a formal O&M training program. This information could be confirmed via client medical reports, a declaration by the client, or through O&M specialist observation.

The major components of the tool follow:

The first three versions of the CET (from Jan to Sept 2012) included the Personal Wellbeing Index – Adult Version (PWI-A) (International Wellbeing Group, 2006). The rationale for its inclusion was that O&M training might have the added advantage of influencing a client’s feeling of wellbeing or satisfaction with life. However, pilot results indicated that this was not the case and the PWI was a cause of distress for a majority of clients. A majority of clients opted not to complete the PWI commenting it was intrusive and irrelevant to their O&M program. The few clients who did attempt to complete the PWI-A objected to or commented on the intrusive nature of the questions, in particular: ‘how satisfied are you with your personal relationships’ and ‘how satisfied are you with your future security?’ Therefore, this measure was deleted from the CET. The majority of clients appeared happy to complete their section of the CET as long as it was quick and easy and did not ask questions unrelated to their O&M training. It might be important to note that the majority of clients had been vision impaired for some time and had not recently experienced sudden vision loss from a stroke or other cause. It appeared clients had adjusted, to varying extents, to their vision impairment and were building on their existing mobility skills. Perhaps those people who have suddenly lost their vision, who have not yet developed strategies to cope, or skills to get around, would most likely report increases in quality of life pre- and post-O&M training. To date, no research has been conducted to identify under what conditions, if any, lifestyle scales measuring quality of life indicate O&M intervention effectiveness.

A total of three CET pilot evaluations occurred between 2012 and 2013 including testing for validity and reliability. Each version of the tool was refined based on pilot data outcome, and client and O&M specialist feedback. The pilot details follow:

The final version of the CET comprises five pages (Appendix A). Pages 1-4 are completed by the O&M specialist. Page 1 includes demographic information about the client, the region in which the O&M specialist works, and details related to the training program. It also includes a brief description of the CET. Page 2 requires information about the client’s level of vision impairment and client complexity. Page 3 requires the rating of the highest environmental complexity in which the training is to be conducted. If the environment changes in complexity from pre to post-measure, then it is noted on this page. Page 4 requires a rating by the specialist about the client’s skill and confidence level as well as a rationale for this rating. Page 5 requires a client rating of their skill and confidence level as well as a rationale for this rating. Detailed instructions for using the tool are also available.

The final version of the tool was rolled out across Guide Dogs NSW/ACT in June 2014. A one-year analysis of data occurred from July 2014 to July 2015.

The test-retest repeatability for the four items in the CET was evaluated using agreement and a weighted Kappa statistic. The Kappa statistic was interpreted according to Viera and Garrett (2005) where 0.61-0.80 represents substantial agreement and 0.81-0.99 almost perfect agreement. Construct validity was investigated by comparing change in the CET items against the global question from the WHOQOL BREF. The responsiveness of the CET was assessed using the Wilcoxon Signed Rank sum test to compare pre/post-program measures for each item separately. The predictors of significant improvement (2 points or more on 5-point scale) for each item were modelled using logistic regression. Any factors significantly associated (p<0.20) were considered in a multivariate model and removed using step-wise backward elimination to determine a final multivariate model predicting improvement.

There were 40 clients in the validity study, 18 men and 22 women, aged 52 years on average (range 18-85). There were 11 clients (28%) with low vision, 23 (58%) that were legally blind and six (15%) with no vision or light perception only. Agreement was 90-100% and the weighted Kappa statistic was >0.80 for all items on the CET, indicating almost perfect agreement, high repeatability, and that specialist and client ratings were consistent.

There were measurable improvements (Wilcoxon Signed Rank sum test p<0.0001) in instructor-rated confidence (1.35±0.92) and skill (2.88±1.11) and client-rated confidence (1.60±1.10) and skill (1.93±1.33). These changes in client confidence and skill levels pre- and post-program were highly significant, suggesting O&M goals were attained resulting in increased levels of client confidence.

There was moderate correlation as shown by the Pearson’s Correlation Coefficient between improvements on the global question ‘how well are you able to get around?’ and improvement on the items on the CET: instructor-rated skill (0.52) and confidence (0.57) and client-rated skill (0.66) and confidence (0.52) indicating construct validity. Importantly, the CET measures two specific constructs (confidence and skill) whereas the ‘ability to get around’ is subjective to interpretation by the client.

All adult clients (18+ years) requiring O&M training were included in the analysis (n=361). Clients were both existing clients requiring an O&M program to learn a new skill, as well as people new to the service requiring formal O&M training. Only those clients whose programs were completed were included in the analysis. Fifty-seven O&M specialists provided the O&M training. The analysis involved clients and specialists from all seven office regions throughout NSW and ACT. The characteristics of the group are summarised in Table 1.

Table 1.

Specialist rating of client skill and confidence:

One post-program evaluation was missing therefore, graphs show data for 361 clients pre-program and 360 post-program. On average, the instructor-rated skill score improved by 1.6 points on the 5-point scale (1.6±1.3, mean ± standard deviation p<0.0001) and the instructor-rated confidence score by 0.8 points (0.8±1.0, p<0.0001). Thus, the improvement in skill and confidence rating was statistically significant for pre- and post comparisons with approximately two-thirds of clients being rated as independent and half ‘very good’ for confidence (Figure 1).

Figure 1.

Client rating of client skill and confidence:

There were 22 clients who did not rate their skill or confidence on at least one of the assessments so data are presented for 344 clients before their program and 345 after their program. The clients rated their skill as on average 1.3 points higher (1.3±1.3, p<0.0001) and confidence 1.0 points higher (1.0±1.3, p<0.0001) after the program. Thus, the improvement in skill and confidence rating was statistically significant for pre and post comparisons and around 50% used the highest rating on the 5-point scale for these items after the program (Figure 2).

Figure 2.

O&M program information:

The majority of programs took place in the larger metro regions. The majority of clients trained in dynamic environments participating in a wide variety of O&M programs (Table 2).

Table 2.

Predictors of the degree of improvement (2 points or more) in skill and confidence:

Client characteristics and environmental factors were considered as candidate predictive factors. The data are odds ratios and their 95% confidence intervals (Table 3) represent the likelihood of improvement in the CET items, where an odds ratio <1 implies that the factor reduces the likelihood of a 2-point improvement and an odds ratio >1 increases the likelihood of an improvement. The odds ratios that are statistically significant have 95% confidence intervals that do not overlap 1 and are highlighted in bold font (Table 3).

Table 3.

Data are odds ratios with 95% confidence intervals. An odds ratio <1 implies that the factor reduces the likelihood of a 2-point improvement. An odds ratio >1 increases the likelihood of an improvement. The odds ratios that are statistically significant have 95% confidence intervals that do not overlap 1 and are highlighted in bold font (Table 3).

Age was independently predictive of a 2-point improvement on the instructor-rated scale for skill whereby older clients were 6% less likely to have this level of improvement per 5 years older (OR 0.94, 95% CI 0.89-0.995). The other factor predicting 2-point improvement was client complexity whereby more complex clients were more likely to make substantial gains in skills. The 2-point improvement in instructor-rated confidence was 30% less likely in male clients (OR 0.70, 95% CI 0.54-0.91) but three times more likely for the small proportion of clients (5%) with communication difficulty (OR 3.02, 95% CI 1.17-7.80).

For the client-rated scores, there were no client specific or environmental factors which influenced the chance of a 2-point improvement on skill but complex environments reduced the likelihood of 2-point improvement in confidence by approximately 85% (OR 0.85, 95% CI 0.74-0.98).

Of the 361 clients there were partially more females than males (male 44%; female 56%) with an average age of 57 years. The majority of clients were either legally blind (45%) or blind (35%), with 22% of clients having one complexity that impacted the training program and 11% who had two or more complexities, for example, a physical impairment (e.g., affected limbs, arthritis, pain), memory or problem solving issues, or another condition (e.g., hearing impairment, musculoskeletal disease, diabetes, Usher syndrome).

Clients participated in 16 types of O&M programs with the majority of clients requiring orientation training (32%) to shops and shopping centres, across roads, new workplaces, universities, and meeting places (e.g., community centres). The majority of clients requiring orientation training were independently using a primary mobility aid (long cane or guide dog). The other major training programs were guide dog (16%), long cane (14%), and support cane (8%). The majority of the programs were taught in complex and dynamic environments (88%) in metropolitan regions (75%).

Pre-program, specialists rated client skill at a lower level than did clients. A majority of specialists indicated that clients would need verbal prompting and observation to acquire the O&M skill. Specialists used their experience to make the rating and were perhaps more objective in their rating than clients. Clients who had previously received O&M training programs over two or more years rated their skill level to achieve their current O&M goal as ‘average’ or ‘good’. These clients had previously experienced successful O&M programs and believed they had O&M skills that would enable them to achieve the program goal. For example, a specialist commented that:

The client commented:

Post-program, specialists rated client skill as slightly higher than clients rated themselves with the majority of clients reporting to have ‘good’ or ‘very good’ skills. Specialists observed that the majority of clients were able to perform the O&M skill independently with only some needing or wanting further observation. Further observation was often provided by partners or friends, who would eventually fade out from the program once clients believed they no longer required observation. A few clients liked being accompanied by their partners when in public despite performing the O&M skill independently (e.g., crossing roads, or travelling on public transport). A client commented:

The confidence level of clients improved significantly from pre- to post-training. The rating by specialists and clients were almost the same. The majority of clients had ‘good’ or ‘very good’ levels of confidence. A client commented:

It is interesting to find that client and specialist pre- and post-program rating of skill and confidence were consistent to one another. That is, pre-training clients rated their skill and confidence levels as similar and post-program similar again. Specialist ratings followed the same pattern. For instance, post-training, about 48% of clients rated their skill as ‘very good’, and 40% as ‘good’. Similarly, post-training about 51% of clients rated their confidence as ‘very good’, and 37% as ‘very good’. Further, post-training, the minority of clients who rated their skill as ‘poor’ (1%), also rated their confidence as ‘poor’ (1%) which replicated the rating of their O&M specialists. This finding seems to infer a strong correlation between skill and confidence as an indicator of O&M performance supporting the claims of Beggs (1992) and Oney and Oksuzoglu-Guven (2015).

A cross analysis of 11 variables was conducted to investigate whether or not these affected client and specialist rating levels of skill and confidence (Table 3). There might be numerous reasons for the results and a brief rationale is offered. The six statistically significant findings follow:

The CET is intended to provide an outcome measure for adult O&M programs which also takes into consideration the complexities of the training environment and additional disabilities of the client that might affect the training program. The CET tool is administered before the commencement of the O&M program and at its completion. The key components are client O&M skill and confidence that appear to relate to O&M performance outcome. The CET is quick and easy to use taking less than 15 minutes to administer. The pre- and post-CET results can be compared immediately without the need for statistical analysis. Guide Dogs NSW/ACT is currently transferring the CET to a research electronic data capture system for use on mobile devices and it is being used to measure ongoing O&M performance.