The APISS Questionnaire: A New Tool to Assess the Epidemiology of Systemic Allergic Reactions to Bee Venom in Beekeepers

Initially, a systematic literature review (data not published yet) was carried out using the methodology previously reported in a protocol paper (8), aiming to identify the observed outcome (questionnaires) of the relevant surveys to generate a comprehensive new instrument, making it possible, through the targeted environmental history, to address epidemiological public health aspects of the self-reported SAR to bee venom and the risk factors for SAR among beekeepers to provide a more complete information in the process of assessing clinical aspects and diagnosis of HVA. Providing evidence-based data, the following objectives were: a) assessing the prevalence of the self-reported SAR to Hymenoptera venom in beekeepers worldwide, b) exploring the impact of environmental differences worldwide on the prevalence rates, and c) updating the principal risk factors for SAR. In total, of 442 studies identified, seven original articles were included and used for developing a comprehensive questionnaire in the Slovenian language. Preliminary data show the prevalence of the self–reported SAR to bee venom ranging from 2.1 to 37.6%. Several risk factors were identified that either increase, decrease risk or no association was proven. The identification of pertinent content domains and item generation were initially carried out by the first author based on two main sources: descriptions of studies’ questionnaires and a copy of a German questionnaire (Questionnaire for the Assessment of Beekeepers’ Health, QABH) obtained on request (9). The pool of domains and items were reviewed and refined by the consensus group, comprising the three authors of this paper, until complete agreement on the domains/items was achieved, resulting in a first draft of the study questionnaire. An example of the refinement is the question about current smoking status (smoking status domain). The predefined responses were refined according to the World Health Organization Countrywide Integrated Noncommunicable Diseases Intervention (CINDI) programme, more specifically according to the CINDI Health Monitor Questionnaire (10).

A panel of 15 experts from different fields (public health, sanitary engineering, clinical medicine, veterinary medicine, biology, microbiology and beekeeping), was nominated and selected based on the criteria presented by Fernandez-Gomez (11). All of these individuals were knowledgeable about the subject, either because of their professional knowledge/academic background or because of their epidemiological and/or clinical work experience. The recruitment period lasted from 5th to 10th September 2021. Each nominee was individually and confidentially contacted through email and invited to be a part of the expert panel. The experts who indicated their willingness to participate were emailed again and provided with all the documents needed to conduct the validity process (i.e., an informed consent form attached to an explanatory cover letter, a first version of the questionnaire, a brief demographic questionnaire (i.e., on gender, age, statistical region, place of residence), and a content validation form). The cover letter included a short description of the study’s purpose, a description of the content review instrument with instructions, a rating score to help guide the experts with regard to the scoring method and the reasons for their (the experts’) nomination (12).

For qualitative validity, members of the expert panel were asked to critically review each item of the targeted domain prior to scoring its relevance. Following the experts’ assessment, domains and items were revised by the consensus group and, if necessary, modified upon their agreement.

For the quantitative assessment of the content validity, a panel of experts was asked to rate the instrument items in terms of their relevance. A four-point Likert scale based on a criterion propounded by Davis (13) was used as follows: 1 = not relevant to the measured domain; 2 = somewhat relevant to the measured domain; 3 = quite relevant to the measured domain; and 4 = highly relevant to the measured domain. The scale was recoded into dichotomised relevant (rating of 3 and 4) and non-relevant (rating of 1 and 2) values prior to calculating the content validity indices (CVIs) (14).

Quantitative data were analysed by the computing CVIs at the item (item-level content validity index, I-CVI) and scale levels (scale-level content validity index, S-CVI), the latter being calculated using two methods, i.e., the average (Ave, S-CVI/Ave) and the universal agreement (UA, S-CVI/UA) (15). Calculations were made using the following equations (1–3): Equation 1 I-CVI=agreed itemnumber of expert {\boldsymbol{I - CVI}} = {{{\boldsymbol{agreed}}\;{\boldsymbol{item}}} \over {{\boldsymbol{number}}\;{\boldsymbol{of}}\;{\boldsymbol{expert}}}}

I = item

In relation to the number of experts, items and using the questionnaire as a tool, the cut-off scores of I-CVI ≥0.78 (13) and S-CVI/Ave ≥0.90, respectively, were considered to be acceptable (16), while for S-CVI/UA a value of ≥0.80 was accepted as good (17, 18).

Pre-piloting was performed among a sample of the target population, beekeepers (n=14) of any age, registered with the Slovenian Beekeepers’ Association (SBA), according to which a beekeeper is a member of the SBA who undergone through a standardized educational and training system in the framework of a lifelong learning, led by the SBA, regardless beekeeping status (professional or amateur). This includes training under a supervisor (mentor) who is well educated in beekeeping, has long experience of beekeeping, uses the principles of good beekeeping practice, performs beekeeping nearby and in the same hive system, and is willing to share knowledge and experience in order to help the new beekeeper with the initial steps on their beekeeping path. The SBA includes 207 beekeeping societies and 14 regional beekeeping associations, with a total of about 7,500 amateur and professional beekeepers (19). The following exclusion criteria were applied: stings caused by Hymenoptera species other than bees (e.g., wasp, hornet); toxic reaction; acute or chronic conditions that prevent the beekeeper from participating in the study; refusal to participate after obtaining informed consent. The recruitment period took place from 25th to 29th September 2021 using an online approach – a Google form published on the SBA website. Participants willing to take the survey marked their informed consent and expressed their preferred time and date to conduct the survey using a face-to-face method. The physical location for the in-person meetings was determined individually and tailored to the participant’s request.

For the quantitative assessment of the response process validation, the members of the rater panel were requested to rate the instrument items in terms of clarity and comprehensibility. The same scoring and the same recoding procedure were applied as for the CVIs. The item-level face validity index (I-FVI) was computed as the number of agreed items divided by the number of raters (Equation 4). Items with FVI values above 0.83 were considered to be accepted (20). Equation 4 I-FVI=agreed itemsnumber of rater {\boldsymbol{I - FVI}} = {{{\boldsymbol{agreed}}\;{\boldsymbol{items}}} \over {{\boldsymbol{number}}\;{\boldsymbol{of}}\;{\boldsymbol{rater}}}}

I = item

A synthesis of the data resulted in the identification of seven domains, as follows: 1) socio-demographics, 2) beekeeping, 3) self-reported SAR to bee venom, 4) treatment, 5) comorbidities, 6) smoking, and 7) beekeeping among family members. An initial pool of 46 items was reviewed iteratively until the consensus group reached agreement on 39 items. The instrument was named APISS-Q, based on the genus name Apis for “bee”, the letter “S” for “Slovenia” and “Q” for “questionnaire”.

Of the 15 experts selected and invited to participate in the study, 13 responded to the mailed request (response rate: 86.7%). The panel experts were aged between 24 and 66 years (mean age 46.7 years) with a slight female predominance (seven of 13; 53.9%). The vast majority came from the Central Slovenia statistical region (eight of 13; 61.5%), followed by Gorenjska (two of 13; 15.4%), while Goriška, Savinjska, and Podravska statistical regions each had one participating expert (one of 13; 7.7%).

Out of 14 study participants, 11 were male and three were female, and they were aged between 34 and 71 years (mean age 54.7 years). The vast majority came from the Central Slovenia statistical region (six of 14; 42.9%), followed by the Goriška (three of 14; 21.4%) and Primorsko-notranjska (two of 14; 14.3%). Jugovzhodna, Gorenjska and Podravska statistical regions each had one pre-test study participant (one of 14; 7.1%).

The APISS-Q in its final form contained 50 items under the following domains: 1) socio-demography (i.e., gender, age, bodyweight, etc.), 2) beekeeping (i.e., type of beekeeping – amateur, professional; type of protective clothing while beekeeping, etc.), 3) self-reported SAR to bee venom (i.e., SAR grading system with the description of typical symptoms for each grade, the number of bees causing SAR, etc.), 4) treatment modalities of the self-reported SAR to bee venom (i.e., medical services accessed after the first SAR to bee venom, allergy specialist review after the first SAR to bee venom, etc.), 5) comorbidities and regular drug therapy (i.e., symptoms of atopy or other allergic diseases, other comorbidities, etc.), 6) behavioural risk factors (smoking status), and 7) beekeeping among family members/close blood relatives.

The relevance ratings of the 13 experts with regard to the item scale are shown in Table 1. All 50 items met the I-CVI threshold of ≥0.78. An average item quality expressed by the S-CVI/Ave was 0.97, indicating a high content validity of the questionnaire. The S-CVI/UA method, reflecting the proportion of items in the tool that achieved a rating of three or four by all the panel members, was below the value accepted as good (0.72). An item level of the response process validity index of 1.00 indicated clarity and comprehensibility for all items of the APISS-Q. The flow chart of the questionnaire development and validation process is shown in Figure 1.

Figure 1.

All beekeepers who filled out an online consent form responded to the survey (100% response rate). On average, it took a little more than 10 minutes to complete the telephone interview, with all participants finding the length to be acceptable. Out of 50 study participants, 43 were male, and seven were female, aged between 18 and over 61 years (mean age 51.1 years). The vast majority came from the Central Slovenia statistical region (20 of 50; 40.0%), followed by the Goriška (nine of 50; 18.0%); Gorenjska (eight of 50; 16.0%), Savinjska (seven of 50; 14.0%), and Jugovzhodna (two of 50; 4.0%). Zasavska, Pomurska, Obalno-kraška and Primorsko-notranjska statistical region each presented had one study participant (one of 50; 2.0%).