Content Validity and Cognitive Testing in the Development of a Motivational Interviewing Self-Assessment Questionnaire

The authors of this paper conducted a comprehensive literature review, including the foundational work of the MI authors Miller and Rollnick (1), literature on the main MI practice coding systems (3–8), and a literature review focusing on self-assessment of MI practice. This helped generate the initial pool of items for the first version of the questionnaire.

The literature review was conducted using the PubMed bibliographic database in 2018, 2020, and during the summer of 2023. Keywords used in the title/abstract search included “self-evaluation questionnaire”, “self-assessment scale”, “self-evaluation”, and “selfassessment”. An article was considered relevant if it was an original research paper or a scientific review article that discussed self-assessment questionnaires related to the practice of MI. We excluded articles that focused on self-assessment of health outcomes in patients/clients or that were self-assessment questionnaires not specific to the practice of MI (e.g., attitudes toward practicing MI, its effects, satisfaction with MI training, etc.). We assessed the following data in the articles: the purpose of the tool, including who it was intended for and the time period it assessed, the MI elements selected, number of items, response categories defined, and number of response options on the response scales.

We initially identified 19 articles and subsequently excluded four articles either because they focused on selfassessment in patients/clients (14–16) or did not address the practice of MI (17).

Analysis of the remaining articles revealed the following: All of the current self-assessment tools were developed or published after 2003 (18–32), with almost half from 2020 onwards (10, 27–32).

Most of the self-assessment tools focused on evaluating the implementation of MI in a single conducted session (20, 21, 23–27, 29–31). In some cases, these tools were derived from instruments used to assess MI integrity, such as MISC (27) or MITI (20, 31), or from a supervisory tool according to MIA:STEP (21). In some of the other instances, they took the form of checklist-style inventories (27, 28).

Various tools addressed the use of different elements of MI. Almost all of them encompassed selected aspects of the spirit of MI, with many focusing on skills (21–24, 27, 29) and emphasizing strategies for assessing readiness for change (18, 21, 26, 29, 32). The range of relevant items in these tools varied from one (19) to 20 (23). Some items were ‘double/triple etc.-barreled’ (actually contained two/three etc. different questions within one) (21, 8).

Response scales were often 5-point (18–20, 23, 27, 31) or 3-point (20, 29, 31), but some were 4-point (24, 25), binary (26), 6-point (29), or 7-point (21). The scales measured frequency (18), agreement (23), the extent of behaviour (19–21, 31), or the number of occurrences of behaviour (20, 31), expertise (24, 25), optimality (27), or capability (29). In three cases, the possible response scales were not described (28, 30, 32).

From our literature review, we generated a pool of 58 items addressing five important aspects of MI: partnership, acceptance, evoking, resisting the righting reflex, and strengthening self-efficacy. Some aspects of MI we did not assess include: focussing, planning, compassion, and developing discrepancy. For practical reasons, we reduced the number of items to 30. We introduced a 7-point frequency scale and included instructions for completing the questionnaire. This marked the creation of version one of the MI questionnaire (MIQ 1.0).

The first expert panel round revealed concerns about the clarity of instructions and the understandability of items. Specifically, there were questions about what the period or frequency referred to, whether it was about the total number of times the element was practiced, the number of times in one session, with one or all patients/ clients, or the duration when it occurred. Some experts raised concerns about the questionnaire’s length and abundant response options, and some questioned the often indifferent neutral middle option. Certain sections were questioned regarding their understandability, and these concerns were given special consideration during the subsequent cognitive testing.

In the first round of cognitive testing all five respondents quickly adapted to the instructions and almost instantly discussed all the required aspects in one flow.

For example:

KT1_1_36-39/1/ (in item P3): “Yes - (reads the question:) How often have you checked if you and the patient (skips the words ‘slash client’) are working together towards the same goal? (short pause, thinking) How often? Well... this actually refers to, it refers to one patient, if I understand correctly, I would interpret it this way: it refers to one patient over several sessions or encounters (note: it means meetings), and I would understand it as, do I check with the patient at each encounter if we are on the same path (short pause)... yes, I would answer (short pause) ‘almost always’. Almost every time the patient came for an encounter, I somehow checked, actually, even between the lines, if we were heading toward the same goal. I would answer ‘almost always’.”

Moderator: (waits for a moment) “I see, okay, now you’ve also told me how you came to your thoughts. What do you think of this question - is it difficult/easy, understandable?”

KT1_1_36-39(1): (short pause) “I find this question quite okay. It’s fine with me. Good.”

Moderator: “And what about the appropriateness of the answers, are they fine? The options, are they okay?” KT1_1_36-39(1): “Yes, ‘never’ is out, well, ‘almost every time’, yes, you kind of refresh or check at almost every encounter if we are both working toward the same goal. I could choose ‘frequently’, well, either ‘frequently’ or ‘almost always’ I would choose.”

Moderator: “I see, what would you choose?”

KT1_1_36-39(1): (pause) “Now, if there were only ‘never’, ‘sometimes’, ‘always’, I would choose ‘sometimes’, well, now, because I have two more sub-questions, ‘frequently’ and ‘almost always’. yes, I chose ‘almost always.’”

Some answers were based less on actual experience (or memory of it) and were more principle-based or considered less thoughtfully. This was primarily the case for some items related to the spirit of MI, particularly partnership and acceptance. Items containing the verbs “ask” and “tell” were affected to a lesser extent. At times different respondents or even the same respondent had particular patients/clients in mind. Respondents showed a good general understanding of the optimal practice of MI elements. Some testers liked the multiple response options, some found them unnecessary. Typically, respondents tended to select the middle answer with less consideration, and they did not encounter difficulty in choosing an adjacent option when prompted.

Using the respondents’ answers, we adapted the instructions to be more precise and direct in completing the questionnaire based on actual experience over principle-based answers. We added adverbial or adjectival emphasis to certain words and underlined them (e.g., actively strive). We also removed the middle option for answers. This resulted in the creation of version two of the MI questionnaire (MIQ 2.0).

In the second expert panel round the indices and the modified kappa coefficient indicated that some experts found understandability problematic with regard to the elements of partnership, acceptance and resisting the righting reflex, and relevance in element evoking according to the S-CVI-UA value. However, no item had any of the four categories indices with values lower than 0.50, at which point an item would automatically be removed, as indicated by Halek (11). As suggested in the literature (9, 10), they were instead taken into special consideration for further adaptation and/or testing. Detailed values of the indices and k*s in all four categories are presented in Tables 1, 2 and 3.

In the second round of cognitive testing, the respondents properly understood both items that were considered potentially problematic in terms of understandability by the expert panel. For example, the expression “personal growth”, considered too broad and not understandable by two panel experts in round two, was consistently viewed by respondents in the second round of cognitive testing as one’s general ability to change one’s way of thinking and behaving, to undergo the necessary behaviour change, or to stop drinking (excessively). In this round the respondents also demonstrated appropriate knowledge regarding the optimal practice of different MI elements and remembered different patients/clients and situations. However, they provided fewer principle-based answers (although some instances still occurred, again in the partnership and acceptance subscales) and relied more on their memory of actual situations.

Based on insights and suggestions from the second round of the expert panel and the cognitive testing we made changes to some expressions, and divided some items into two separate questions, made further improvements to the instructions, and created the third version of the MI questionnaire (MIQ 3.0).

This final version of the questionnaire was then approved by the expert panel in the third round, and no further comments were received from respondents during the questionnaire piloting.