Development and Initial Validation of the O&M VISSIT for Orientation and Mobility Specialists to Determine Service Intensity

United States federal law mandates that educational goals for students with disabilities include a summary of the child’s present level of academic achievement and functional performance, along with a description of the impact that his or her disability (e.g., vision impairment) has on ability to be involved and make gains in the general education curriculum (US Department of Education, 2004 (section 300.320(a)(1)). O&M services were first named as an important component of individualized programming in the 1997 reauthorisation of the Individuals with Disabilities Education Act (IDEA) (US Department of Education, 2004). According to former Secretary of Education, Richard Riley (2000), it is crucial that children who are blind/vision impaired receive O&M instruction early and are provided necessary services at appropriate times, which

… increases the likelihood that they can participate meaningfully in a variety of aspects of their schooling, including academic, nonacademic, and extracurricular activities. Once these individuals are no longer in school, their use of acquired [O&M] skills should greatly enhance their ability to move around independently in a variety of educational, employment, and community settings. These skills should enhance the ability of blind and visually impaired students to obtain employment, retain their jobs, and participate more fully in family and community life. (36586-36587).

In 2013, the Service Intensity Subcommittee of the Texas Action Committee for the Education of Students with Visual Impairments (a Texas Education Agency-endorsed stakeholder group tasked with ensuring appropriate educational services for students who are blind/vision impaired) carefully examined available service determination tools (Toelle and Blankenship, 2008; Durkel and Miller, 2009; Michigan Department of Education, 2017) and found these tools lacked the guidance needed to determine service type, time, and intensity recommendations. The tools either did not address all skill areas related to individual needs or included aspects unrelated to student need (e.g., practitioner’s workload, material preparation, and travel). The subcommittee determined the analysed models did not effectively devote attention to student needs in the ECC (Hatlen, 1996; Huebner et al. 2004; Lohmeier, Blankenship, and Hatlen, 2009; Sapp and Hatlen, 2010; Wolffe and Kelly, 2011).

Therefore, the subcommittee developed a unique tool, basing programming decisions on evaluation in each area of the ECC, known as the Visual Impairment Scale of Service Intensity of Texas (VISSIT), now published for use by itinerant TVIs to determine service type and intensity for students with vision impairment (VISSIT, 2014; Pogrund et al. 2015). A national validation study on the VISSIT for TVIs was completed in May 2015, revealing that consequential validity, social validity, and content validity were all significantly valid. The internal consistency reliability proved significantly reliable.

Following the success of the VISSIT, the subcommittee received numerous requests for an equivalent tool for O&M specialists. These requests called for a tool that uses complete evaluation of student strengths and needs across the O&M spectrum to provide quantifiable, data-driven support for decision making related to O&M service type and intensity. Having such a tool would assist to establish consistent methods and guidelines to combat variations in service recommendations. A solution for the pervasive service discrepancies throughout the profession of O&M was a motivator for this project. Only a handful of tools are available to assist O&M practitioners in determining service type and intensity, and selecting the right level of service for a particular student’s needs is largely left to professional judgment (Wall Emerson and Anderson, 2014). Orientation and Mobility specialists should prioritize student need to determine adequate service frequency and duration so that each student can develop and practice skills and make substantial, timely progress.

The Service Intensity Subcommittee reconvened in February 2015 and began analysing available tools and methods for O&M service determination to identify limitations of each tool. This undertaking revealed that certain tools, even those widely adopted, lacked the ideals and focus considered essential by subcommittee members. Such ideals included focusing primarily on evaluation data to guide service intensity recommendations and recognising the importance of collaborative consultation as a service delivery model component for O&M. Idol et al. (1995) described collaborative consultation as “an interactive process that enables people with diverse expertize to generate creative solutions to mutually defined problems” (p. 329). Other ideals included documentation of collaborative consultation time and maximal direct service times based on student need rather than factors of “how much service [practitioners] have been able to provide to students during their career” (Wall Emerson and Anderson, 2014, p. 156). For example, in both the O&MSRS and the O&MSRS+ (also known as “the Michigan Scales”), the recommended O&M service times were restricted at a maximum limit of 30 to 90 minutes per week for all but the most severe cases. Even as more students qualify for O&M services, practitioners might compensate for their lack of availability by reducing frequencies of lessons. Most available service recommendation tools allow consideration of certain factors which might increase or decrease time recommendations.

The O&M VISSIT tool discussed in this article is designed to be appropriate for any and all students with vision impairment who require O&M services, including infants, toddlers, and those with multiple impairments who are medically fragile or deafblind. Because the O&M VISSIT is based on student performance and ability (gathered from functional vision evaluation/learning media assessment/ECC and O&M evaluation data, Individualized Education Program [IEP] progress monitoring, and observation/collaboration), the need for separate scales for different populations is eliminated. The members of the subcommittee created this scale with the belief that factors such as the student’s amount of vision (which comprise over one-third of the Michigan scales’ severity considerations) should not be used in consideration of service type/intensity, respecting the individuality of student performance irrespective of visual acuity, field, function, or other measurement by an eye medical professional. Though these criteria may be used to qualify for services, they do not necessarily address student needs or abilities at any given time.

Pavey et al. (2003) emphasized that individual differences among children (e.g., the divergent implications of an assortment of visual diagnoses) demand distinct and diverse emphases of support. Thus, there is no ‘blanket’ service delivery structure that will address the needs of a caseload across the board. As stated by Riley (2000) “the extent to which [O&M] services are necessary for an individual child and, if so, the amount and duration of those services that are necessary for a child to receive FAPE [Free and Appropriate Public Education] are decisions for the child’s IEP team” (p.36590) and should be updated or supplemented as often as required depending on individual factors as the student matures.

The primary protocol for the O&M VISSIT was drafted in June 2015, and initial field testing was completed in September 2015. Revisions occurred the following month, and pilot testing began in January 2016. The initial validation study was designed in September 2016, and the O&M VISSIT tool and surveys were distributed in January 2017. Data collection was completed in May 2017. The Texas Tech University Institutional Review Board approved this study prior to commencement.

This study evaluated validity and reliability aspects of the O&M VISSIT using mixed-methods survey research with purposive sampling. Data were collected using a follow-up electronic questionnaire. The questionnaire included Likert scale questions and open-ended response questions to gather both quantitative and qualitative information.

This study took place during one academic semester. Orientation and Mobility specialists used the O&M VISSIT for at least one student on their caseloads who needed or recently had an O&M evaluation (initial or three-year reevaluation).

Procedure

Researchers emailed recruitment letters in spring, 2017 to the regional ESC vision/O&M consultants in Texas and to TSBVI Outreach specialists, with instructions to forward a separate recruitment letter to O&M specialists in their respective geographic locations who met the criteria for participation. Upon receiving responses from selected practitioners, researchers electronically disseminated a copy of the O&M VISSIT to all who agreed to participate (Fig. 1).

Each participant selected at least one student from his or her caseload and conducted an O&M evaluation and/or reviewed all current evaluation results for the student(s). The O&M specialist then used the O&M VISSIT to determine recommended service time for each student based on the results of the evaluation data. Each participant used evaluations they deemed appropriate and used those assessments to evaluate each student’s present level of performance in every skill area of the scale. After completing the O&M VISSIT on at least one student, each participant mailed their completed protocol (removing all identifying student information) to the researchers. When the researchers received the completed protocols, they sent an email to the corresponding participant, linking an anonymous electronic survey designed to gather information about the O&M VISSIT and its use.

Survey data was collected using Qualtrics, an internet-based survey generator. The surveys featured Likert scale quantitative questions (formulated to garner information about the scale’s reliability and validity) as well as qualitative questions (formulated from free-response opportunities to gather user’s perceptions about the O&M VISSIT format, completion time, and suggestions for revision). Data collection was completed in May 2017, after which data analysis commenced.

Figure 1

Means and standard deviations for each quantitative question in the survey were determined using descriptive statistics. For eight electronic survey questions, participants were instructed to score opinions on a five-point Likert scale. One additional question asked participants to rate the relevance of each of the 36 distinct O&M VISSIT items on a four-point scale. Higher numbers indicate the highest ratings, with the number 1 being the lowest. Seven questions were yes/no, with “yes” represented by a value of 2, and “no” represented by a value of 1. For those questions, means closer to 2 indicated more affirmative responses. Table 1 lists the Likert scale questions, their means, standard deviations, and the number of respondents for each survey question.

Analysis of the survey data indicated that the version of the O&M VISSIT used in this study was moderately valid in consequential validity and significantly valid in social and content validity (both content validity ratio (CVR) and content validity index (CVI)). The internal consistency reliability was found to be moderately reliable.

Messick (1989) described consequential validity as a measure of participants’ opinions of a tool’s intended and unintended consequences in interpretation and use. The O&M VISSIT was intended for use by O&M specialists, with intended consequence of providing guidance in determining appropriate type/amount of O&M services needed by an individual student, based on evaluated need in comprehensive areas. Consequential validity of the O&M VISSIT was moderately supported in the data by 76% of participants stating that results obtained using this tool for recommendations of student service needs matched their professional judgment. Fifty-three percent of participants agreed that results relating to type/amount of service matched what they would recommend. The intended consequence of using the O&M VISSIT is supported by the participants’ responses, which, in turn, support its consequential validity.

As described in “Procedures,” the O&M VISSIT’s design supports its consequential validity. Elements involved in developing this scale included a specified purpose, the population for whom its use was intended, the data that would be collected through its use, guidelines for decision-making, the tool’s boundaries, and reporting requirements (Suen and Rzasa, 2004). Inclusion of these features in the O&M VISSIT helped to safeguard validity by preventing unintended consequences of its use.

Social validity describes the intervention procedures specifically indicating the level of users’ satisfaction with, and acceptability of, those procedures (Luiselli and Reed, 2011). In this study, social validity measured participants’ acceptability and satisfaction regarding the O&M VISSIT’s usefulness and effectiveness when used in decisions of service time. Social validity was not only established by 88% of participants, stating the O&M VISSIT was easy to use for this purpose, but was strongly reinforced when 100% of the participants declared they would use this tool again in the future when making recommendations for type/amount of O&M services.

In the follow-up survey, participants rated the acceptable relevance of each item included in the O&M VISSIT using a 4-point Likert scale (4 = Completely relevant, 3 = Very relevant, 2 = Somewhat relevant, 1 = Not at all relevant). CVR was conducted for each item, calculating the quotient of the number of experts (participants) who rated the item either 3 or 4 in terms of relevance and the number of experts evaluating the relevance of each item. To be considered highly valid, an item’s CVR must be at least 0.48 (for each item, n = 16). All O&M VISSIT items were considered highly relevant, with the highest CVR = 1.0, and the lowest CVR = 0.59. CVI (content validity of the instrument as a whole; the mean of all CVR results for each scale item) was calculated for the entire scale, with a resulting score of 0.80. This indicates that the content of the O&M VISSIT was highly valid in its entirety (CVI at or above 0.80 is considered high content validity) (Hair et al., 2006).

For reliability, Cronbach’s alpha statistic was used on the survey results (this measure indicates the strength of the consistency in the scale items, thus the reliability of these items to measure the targeted concepts, in this case, the construct of student need for O&M services). Internal consistency reliability (meaning this set of items are correlated, measuring the same thing) for the set of all items on the O&M VISSIT was 0.742 using Cronbach’s alpha. Scores at or above 0.7 on Cronbach’s alpha are considered reliable (Hair et al., 2006), supporting the O&M VISSIT’s moderate reliability in that all items included on the scale relate to measuring student need for O&M services.

The O&M VISSIT was well-received by study participants, with 88.23% reporting it was either mostly or completely easy to use, and 94.12% stating that the instructions were either mostly or completely clear and understandable. The average time it took to complete one O&M VISSIT was around 25 min, taking less time once the tool was used multiple times. Similarly, 88.23% of respondents agreed that section descriptions of O&M Skill Areas included in the O&M VISSIT were mostly or completely clear and understandable, with user commentary stating, “I like how there’s a SKILL listed and then the descriptions follow it. That’s very helpful,” and “Very clear. I feel like the descriptions were accurate. Nothing needed to be added. Very concise. Perfectly done.”

Relating to scoring, when asked whether the O&M VISSIT criteria were clear and understandable, 93.75% of participants agreed that the criteria were clear. Feedback indicated that completing the O&M VISSIT electronically made scoring “very easy.” Participants responded positively to the O&M VISSIT’s inclusion of the Additional Areas of Family Support table (which addresses collaboration services not reflected elsewhere in the scale), finding it effective in incorporating family needs into service time decisions. One participant declared this table a “crucial component.”

Participants were asked whether their O&M VISSIT scoring of student need was based on student’s evaluation results. Answers varied, 58.82% claiming mostly, and 29.41% claimed completely based on the evaluation results. Individual participant feedback mentioned using a variety of sources for information, or the timing of the O&M VISSIT did not match up with the student’s most recent evaluations. Other stated reasons included “caseload is too high to provide the amount of services suggested.”

The majority of participants (94.12%) agreed that the Contributing Factors section (i.e., medical issues, transition, and time-intensive instruction leading to increased/decreased service time) was either useful, mostly useful, or completely useful, adding, “because each student is different, it is good to have a specific section to address those specific needs.”

When asked whether O&M VISSIT results matched professional judgment regarding student need and recommended type/amount of service, 11.76% said the results completely did not match, 11.76% said they somewhat matched, and the 76.48% said the results matched either mostly or completely. Overall feedback was positive, with most commenting that the results were surprisingly accurate to what participants were already implementing. However, other responses reported that the “results were 20-30% greater than the current service level for my student. The [O&M] VISSIT helped me to more fully account for all of the ‘intangibles’ that go into services for this student.” Others stated, “It mostly matched what I know the student NEEDS, however, unfortunately does not match what I am able to provide.”

When asked whether O&M VISSIT results directly translated into the IEP service recommendations, responses fell across all five Likert categories with 11.76% saying completely did not, 11.76% said mostly did not, and 70.59% saying results did translate into IEP service recommendations. One participant commented, “It is what I would like to do, but we currently are under-serving O&M due to the logistical side of things and a somewhat resistant administration.”

In the final items of this survey, all participants (100%) reported that they currently engage in collaborative consultation, yet only half (50%) are documenting the amount of time spent engaging in collaborative consultation. When asked whether collaborative consultation should be documented, all (100%) replied affirmatively, and all (100%) replied ‘yes’ when asked if they feel that an O&M service delivery model that includes collaborative consultation is an efficient way of delivering O&M services. Comments included, “As an O&M specialist, you cannot possibly be there to help the student generalise their skills daily. This collaboration will help with generalisation,” and “Yes, it impacts follow through of other professionals working with my students. It also helps me to stay on top of what is occurring when I am not there. Collaborative consultation is necessary to ensure student success.”

The majority of participants (82%) reported that the O&M VISSIT is a better tool for use in determining type and amount of service than other tools or methods currently used. Those who answered ‘no’ were asked to indicate their preference, and two participants preferred the Michigan Scale and years of experience.

All participants (100%) in this study said they would use the O&M VISSIT again in the future for determining type and amount of service for their students. Comments included, “I thought this was the most accurate, concrete, and reliable source of information compared to the Michigan. Yes, I would use it for all of my students,” and “I would like to use it forever and ever as it is highly effective and matches up well with VISSIT for TVI services,” and “I liked using the O&M VISSIT. It was very helpful in determining a student’s service time. I would highly recommend this to others.”

Quantitative data provided moderate support for consequential validity, but significant support for social validity and content validity (CVR and CVI) of the O&M VISSIT. Qualitative data also provided positive feedback regarding the use of the tool, as well as the beneficial impact that the tool can have on determining appropriate service type and time. This tool will assist O&M specialists in making determinations regarding the type and amount of services to address individual student needs. Its developers strongly maintain that all students receiving direct O&M instruction require collaborative consultation services as well to provide information, identify areas of need, monitor reinforcement of skills, facilitate generalization, and support all team members including the TVI and the student’s family.

The O&M VISSIT provides O&M specialists the opportunity to document, plan, and allocate their time for collaborative consultation. Documentation of this service type is overwhelmingly supported and performed by specialists, separating this scale from other available tools. Close examination of the way O&M specialists currently determine service type/intensity and the factors influencing those decisions might be an area for future research along with validation of an O&M evaluation tool for consistent use by O&M specialists.

Eighty-two percent of participants agreed the O&M VISSIT was a better tool than other available methods for determining service type and time. Despite the fact that some participants currently or previously have used other available service intensity tools, all who used the O&M VISSIT (100%) said they would use this tool again. As the profession moves forward, practitioners and specialists face increased demand for accountability and research-based practices (Pogrund et al., 2015), and the design of the O&M VISSIT includes built-in documentation to assist quantify the often-subjective levels of student need for O&M services.

Some limitations of the study include: the limited generalizability of O&M VISSIT from the way in which participants were chosen for the study (purposive sampling), which may not adequately reflect responses of a random sample of the larger population. Also, the number of participants in this initial study was relatively small, influenced by the decision to use the expert sampling method. In addition, qualifications for participants excluded O&M specialists lacking sufficient career experience. To obtain the most accurate feedback, authors of the O&M VISSIT were exacting in their selection of the most qualified individuals to gain maximum validity and reliability from data collected. By relying on experienced O&M specialists’ expert opinions, researchers hoped to ensure every participant was intimately familiar with making determinations (with or without a similar method or tool) for type and amount of service to recommend for students. Their feedback demonstrated the mostly significant validity and moderate reliability of the O&M VISSIT. The authors anticipated that potential limitations due to expert sampling (with few participants) may be assuaged by considering that inexperienced O&M specialists who use the O&M VISSIT for service recommendations will achieve proper intended guidance from these preliminary findings by veteran professionals. A further limitation is that reliability and validity levels might have been impacted by the small participant pool producing less overall data for analysis.

The final limitation is that this study was restricted to O&M specialists in Texas. As resulting data only represent a single state in the U.S., investigation in other regions, states, and countries should be explored as the O&M VISSIT becomes more widely available (pending addition of the tool to the tsbvi.edu website). This is hoped to be mitigated soon, as a large-scale national validation study of the use of the O&M VISSIT is anticipated in spring, 2018.

The O&M VISSIT is shown to be a moderately reliable tool in terms of internal consistency, with moderate consequential validity, significant social validity, and significant content validity (for each item independently and for the instrument as a whole). O&M specialists can use this tool for guidance in determining the proper type (direct and collaborative consultation) and amount of service for all students receiving O&M due to vision impairment.

As this tool becomes available to a wider audience, its expanding reach could potentially increase reliability and validity of the scale, fortifying its viability among the few tools/methods currently available to O&M specialists to determine service intensity for students.

The O&M VISSIT is not a caseload/workload analysis tool. Rather, it is only one piece of workload analysis planning, helping to establish appropriate caseload size. The tool does not account for all issues related to workload (e.g., planning and travel). However, practitioners might wish to use O&M VISSIT results to provide tangible data with which to present to administrators in requesting the need for additional staff positions. Students with vision impairment who receive the appropriate type and amount of O&M services while in the educational system will be better prepared for college, career, and independent living to the greatest degree possible when they transition to adult life.