A Statistical Approach for Comparative Assessment of the Effect of Smoke Exposure in In Vivo Experiments: A Case Study of an OECD 90-Day Inhalation Study Including 3R4F and 1R6F Reference Cigarettes

The term “comparative assessment” refers to testing for differences, which in statistical terms translates to testing the null hypothesis that products do not differ versus the alternative hypothesis of product difference. Detection of statistically significant differences leads to rejection of the null hypothesis. This is commonly achieved using probability evidence provided by the so-called (p)-values. For a significance level (α), one declares differences to be statistically significant when the p-value is smaller than α. The level α is also known as the consumer’s risk or the probability of falsely concluding that a difference exists when there is none. As the number of out-comes on which the products are compared increases, so does the consumer’s risk, exceeding the level α, so the p-values must be adjusted (10). The use of p-values is contentious, and the literature includes extensive discussions on their use and misuse (11, 12,13,14). In a contrasting approach, equivalence testing has been used to demonstrate equivalence between two products (15, 16). This approach has mainly been applied in absorption studies during drug development under the name of bioequivalence, in accordance with U.S. Food and Drug Administration (FDA) and European regulatory authority guidelines (17,18, 19,20). The starting point (null hypothesis) in equivalence testing is that the two products are different, so they are declared equivalent with rejection of the null hypothesis. Two one-sided tests (TOST) (21, 22) are used for this purpose, in contrast to the single two-sided test performed in difference testing.

Confidence intervals are used to assess differences and/or equivalence (23, 24), providing more insight into the comparative assessment between two products. They provide information on the direction and the magnitude of the observed differences, as well as the associated uncertainty, which is reflected in their width. The width of the confidence interval depends on the sample size and confidence level 1−α, creating a direct link between confidence intervals and statistical tests. For difference testing, we look for a confidence interval on the (1−α)% confidence level; for testing equivalence with TOST we look for a (1−2α)% confidence interval. Figure 1 illustrates several hypothetical confidence intervals and their link to statistical difference and equivalence testing resulting from a comparison of mean ratios between two products. Row A indicates no difference between the two products with high certainty. The mean ratio between the two products is equal to one, and the confidence interval is narrow. The confidence interval in row H is equally narrow, but the effect of the test product is reduced by almost 50% relative to the reference product. In terms of a difference testing approach, the observed effect in case H is highly significant, while the one in case A is not. Rows C and G illustrate statistically significant changes; a reduction and an increase for rows C and G, respectively. Statistically significant increase is also depicted in row D and row E. For equivalence testing, case H shows no equivalence between the two products, whereas case A shows equivalence within the standard equivalence limits of [0.8, 1.25]. No equivalence is the conclusion for all rows other than A, given all confidence intervals don't entirely lie within the equivalence zone of [0.8, 1.25].These equivalence limits are commonly used in bioequivalence studies and are also recommended by the FDA. Note that they are symmetric in the ratio scale (1/0.8 = 1.25). We propose to extend the equivalence limits beyond [0.8, 1.25] according to: [1] δL,δU=exp±log1.25,tα,df+tβ/2,dfσRn $$\left[ {{\delta _L},{\delta _U}} \right] = \exp \left\{ { \pm \left( {\log \left( {1.25} \right),\left( {{t_{\alpha ,df}} + {t_{\beta /2,df}}} \right){{{\sigma _R}} \over {\sqrt n }}} \right)} \right\}$$ where σ_R represents long-term variability (in standard deviation terms), the constants t_α and t_β/2 represent the student percentiles at consumer’s and producer’s risks α and β, respectively, with df = 2n−2 degrees of freedom, where 2n is the total sample of animals allocated to the two products. For illustration purposes we report in Table 1 and Table 2 variable lower and upper equivalence limits using two different proposals (using equation [1] and an additional approach described in the Supplementary Material) under different statistical design parameters. The Supplementary Material provides a more detailed description, references, and insights into the equation above and the statistical methods used within the proposed framework. In Figure 1, this approach is illustrated by overlaying blue boxes to indicate the new (variable) equivalence range limits. Comparing the confidence intervals and blue boxes in Figure 1, the interpretation of the statistical comparisons’ changes concern rows C, F, and G. In all three cases, the two products would be considered equivalent with respect to the new (variable) equivalence range limits, while using the [0.8,1.25] equivalence limits they would not. Rows F and I illustrate inconclusive cases where neither statistical difference nor statistical equivalence is met, when the standard equivalence limits are used. Yet, given the new (variable) equivalence range limits, row F would classify as equivalent.

Figure 1.

Illustrated cases of the relative effect of a test item versus a reference item as provided by their geometric mean ratio (black circles) and associated 90% confidence intervals (black bars). The blue boxes show the variability of the reference item. Rows A to I illustrate different scenarios (explained further in the main text) for the value of the mean ratio and confidence interval and how they influence conclusions about the differences and equivalence of the two items.

Data from a 90-day OECD rat inhalation study served as a use case for our statistical approach on the comparative assessment of the exposure effect to smoke from the standard reference cigarette 3R4F versus the new reference cigarette 1R6F. The study data and analytical methods used for their generation are described below. In addition to the current study data, historical 3R4F data on the same set of endpoints were extracted from three previous inhalation studies and were gathered to build the historical reference data set. Starting from the current study data, the geometric mean ratio between the new reference cigarette over the standard one was computed for each endpoint (Supplementary Material I). The statistical comparisons were performed on the ratio scale, and 90% confidence intervals for the geometric mean ratio were computed for equivalence testing. Equivalence is met when the entire confidence interval on the geometric mean ratio for an endpoint lies within the equivalence limits. These are defined using the historical 3R4F data and expand the equivalence limits beyond the [0.8, 1.25] range using Equation 1. For illustrative purposes, the long-term variability (σ_R) estimates for all the analyzed endpoints on 3R4F (estimated across the three previous inhalation studies) are plotted in Figure 2.

Figure 2.

Long-term variability estimates (σ_R) depicted in blue bars for all endpoints across the historical 3R4F data. Variability estimates are expressed in percentages.

The analysis presented in this work focuses on the comparison of two products: new (1R6F) versus reference (3R4F) in a 90-day OECD rat inhalation study that followed OECD test guideline 413 (1) and was conducted in compliance with the OECD Principles of Good Laboratory Practice and the test facility’s quality management system. The test facility is National Parks Board/Animal and Veterinary Service (NParks/AVS-licensed) and Association for Assessment and Accreditation of Laboratory Animal Care (AAALAC) accredited, and care and use of the rats was in accordance with the National Advisory Committee for Laboratory Animal Research (NACLAR) Guideline (25) and AAALAC requirements (26). All animal experiments were approved by the Institutional Animal Care and Use Committee.

The cigarette smoke was administered by nose-only exposure inhalation to outbred male and female Sprague Dawley rats [Crl:CD(SD)] bred under specific pathogen-free conditions (10 rats per sex). The exposure was conducted for approximately 13 weeks (6 hours per day, 5 days per week). Week 1 consisted of a time-adaptation phase during which the rats were exposed to increasing exposure durations over 7 days. All exposures were targeted to deliver 23 μg/L nicotine in the test atmosphere.

A set of relevant endpoints was analyzed as part of the comparative assessment between 3R4F and 1R6F. These endpoints were grouped into four main categories: (i)

test atmosphere characteristics in the exposure chambers;

Data from selected endpoints from three historical studies that used 3R4F (27,28,29) were also collected and analyzed as part of the comparative assessment between 3R4F and 1R6F.

A tabulated method description for the analyzed parameters is given in the Supplementary Material II (Supplemental Tables A–C).