Longitudinal Assessment of ROX and HACOR Scores to Predict Non-Invasive Ventilation Failure in Patients with SARS-CoV-2 Pneumonia

SARS-CoV-2 infection has wide range of clinical manifestations. Severe pneumonia leading to acute respiratory distress syndrome (ARDS) is one of the most catastrophic manifestation having high mortality [1]. ARDS presents as quickly escalating hypoxemia with a significant ventilation-perfusion mismatch [2]. Lung compliance is preserved early in ARDS, and as the disease progresses, lung compliance is reduced due to worsening alveolar-oedema and fibrosis [2].. Prior to SARS-CoV-2 pandemic, traditional approach for management of ARDS has been to intubate and perform invasive, lung-protective ventilation with proning. However during SARS-CoV-2 pandemic large numbers, limitation in resources and avoidance of endotracheal intubation prompted a large number of patients to be managed by non-invasive ventilation [3].

Non-invasive ventilation (NIV) is administered either by a high-pressure or a high-flow system. High-pressure systems depend on a specialized NIV-mask and deliver positive end-expiratory pressure (PEEP), which increases oxygenation [4,5]. In-contrast, high-flow systems depend on a specialized nasal canula (HFNC) and delivers oxygen and air at a high flow rate. Both methods reduce work of breathing and assist inspiration. Since assisted ventilation is often required for a prolonged duration and use of either of these techniques depends on availability and patient-comfort, often both these techniques are used alternately in a same patient. Failure of NIV to correct hypoxemia may lead to intubation and invasive mechanical ventilation (IMV) [6,7,8,9].

The incremental duration of non-invasive ventilation (NIV failure) has a direct association with self-inflicted lung injury which further leads to worsening of respiratory mechanics and poor outcome [10–11. HACOR score (heart-rate, acidosis, consciousness, oxygenation, and respiratory rate) was initially proposed for NIV failure [10]. A more simplified ROX index (ratio of respiratory rate and oxygenation) was developed for prediction of high-flow failure [12].

Prior to COVID-19 era, HACOR score was exclusively calculated for patients on NIV failure and ROX index was exclusively calculated for HNFC patients.. During the COVID-19 pneumonia Valencia et al exclusively compared these score in patients requiring HFNC [13]. We in this study explore further comparative performance of these two indices in NIV failure from longitudinal perspective. The secondary objectives of the study were to detect the optimum cut-off values of superior indicator and to check the effect of other co-variables on NIV status from the multivariable perspective after adjusting the effect of superior indicator(ROX/HACOR).

Out of 653 ICU admissions between 1^st April 2020 to 15^th June 2021, 441 (67.5%) received NIV as initial ventilatory strategy for >24 hours. While 92 (14.0%) were intubated and mechanically ventilated within 24 hours of ICU admission and another 102 (15.6%) recovered on oxygenation alone. Some patients left against medical advice. A total of 262 patients had NIV failure and were shifted either to invasive ventilation or died during NIV. The whole study flow is summarized in Figure 1 IMV. Individuals with NIV-failure had a higher age, more severe ARDS, higher SOFA scores, and a greater prevalence of Acute Kidney Injury, hypotension and vasopressor use as compared to those who did not fail NIV. Individuals with no NIV-failure received more steroids and had a lower length of ICU stay (Table 1).

Fig. 1.

Performance of both HACOR score and ROX index was found to be similar. Individuals who had NIV-failure had a progressive rise in HACOR score, and a corresponding decline in ROX-index from day 1 to day 6 (Table 2). The similar or non-inferior discrimination of both HACOR score and ROX index between the NIV-failures and NIV-success subgroups in visually depicted in Figure 2. ROC analysis further corroborates to this finding. The area-under-curve for prediction of failure rises from 0.84 to 0.94 from day 1 to day 3 for ROX index and from 0.79 to 0.92 for HACOR score in the same period while further increment in predictiveness is marginal for both scores from day 4 to 6 (Figure 3). DeLong's test for two correlated ROC curves also had an insignificant difference expect on day-1 (D1: 0.03 to 0.08; p=3.191e-05, D2: −0.002 to 0.02; p = .2671, D3: −0.003 to 0.04; p= 0.1065, D4: −0.01 to 0.02; p= 0.7326, D5: −0.03 to 0.02; p= 0.5062, D6: −0.01 to 0.02; p= 0.9423).

Fig. 2.

Fig. 3.

The optimum cut-off values for the ROX score to predict NIV failure was determined as 4.47 at day-3. The detailed description of the day wise optimum cut off points, corresponding to 4 selected methods and sensitivity, specificity, PPV, NPV and likelihood point estimates with 95% CI are given in Supplementary appendix-S2.

On multivariable analysis, ROX index with a cutoff of greater than 4.47 was found protective for NIV-failure (OR 0.15 (95% CI 0.03–0.23). Age more than 60 years and higher SOFA score were other significant independent predictors of NIV-failure (Table 3).

The visualization of marginal effects of age, SOFA and ROX score using probability distribution plots is shown in Figure 4. Older age independently seems to increase the probability of NIV failure irrespective of SOFA and ROX value. At older ages a relatively moderate SOFA score (8 or more) favours NIV failure even if ROX is on lower side. On the other hand, at younger ages a lower SOFA score with a moderate ROX favours NIV success. This observation may warrant the importance of estimating SOFA at admission amongst the relatively younger COVID-19 patients.

Fig. 4.

The HACOR score was developed by Duan et al where they observed an upward trends of HACOR score in patients with NIV failure [10]. The diagnostic accuracy for HACOR score of greater than 5, after 1 hour of NIV therapy was 81.8% and remained above 80% thereafter [10]. This accuracy was not found to be influenced with NIV duration, diagnosis, age, or disease. HACOR score is a summation based score where each indicative parameter is assigned a numerical value. A higher additive score thus indicates higher probability of NIV failure. On other hand, ROX index is a fraction based score where of SP02 to FiO2 ratio with respiratory rate as denominator [12]. With this context this study to the best of our knowledge is one of the largest single center series from India where 441 COVID-19 patients were initiated on NIV.

The faltering of the ROX is also evident in our study. In NIV failure group, amongst the subgroup having the ROX score less than 4.47 (and subsequently lending into failure) the ROX goes from 3.27± 0.59 (d1) to 3.23±0.66 (d3) to 2.94±0.72(d6). The subgroup having the ROX score greater than 4.47 in NIV failure group, the score trajectory was 5.26±0.70 (d1) to 5.54±1.01 (d3) to 5.48±0.65 (d6). In NIV success group, the subgroups having ROX scores above and below the identified cut-off values showed a score trajectory respectively from 5.83±1.00 (d1) to 6.35±1.39 (d3) to 7.28±1.67 (d6) and from 3.91±0.45 (d1) to 3.86 ±0.43 (d3) to 4.01±4.07 (d6). These descriptive statistics assigns a notion of stalling (faltering) courses of ROX in destined to be a NIV failure group. This stalling phenomenon holds true even for the subgroup erroneously misclassified as ‘normal’ based on identified ROX cut off in NIV failure group as well.

The discriminative capacity of HACOR was 0.79 (AUC-ROC-0.79) in our study; apart from study by Valencia et al (AUC-ROC-0.71); Santus et al (AUCROC-0.74); Guia et al (AUC ROC-0.87) the score has not been validated as per se in COVID-19 disease [13,14,15].

The ROX index greater than or equal to 4.88 measured after 12 hours of HFNC was significantly associated with a lower risk for MV (hazard ratio, 0.273 [95% confidence interval, 0.121–0.618]; p=0.002) [12]. Here in our study we calculated the cut off by three methods i.e. Direct logistic regression, 90% minimum sensitivity, high negative predictive value and by ROC, by all these methods ROX Cut off of 4.47 was >90% sensitive of predicting NIV failure. In recently published meta-analysis by JP et al., which included 1300 COVID-19 patients on HFNC therapy optimal cut off value for ROX index may fall close to 5 within 24 hours of admission with time to assessment was taken as 6 hours [16]. We in our study had a lower cutoff, this could be due to inclusion of patients with only on NIV therapy. These patients are much more hypoxemic than patients on HFNC and above all this cut off is for patients on NIV and not for patients on HFNC therapy. Other reason could be till now ROX scores have been computed in patients for a period of 48 hours but in our study we serially followed up to Day 6 where in patients with NIV failure, the serial ROX (48 vs 72 hours) would have a declining trend. AUC for ROX on D3 was equivalent to D6, hence Day 3 ROX would be better predictor of NIV failure. Logically also D3 seems fine once the patient is admitted to ICU, apart from respiratory support in our case it was NIV, anti-inflammatory therapy is initiated and by D3 it gives a clear picture whether the patient is improving or not and if not it's better to intubate as prolonging him on NIV for long time precipitates patient induced self-inflicted lung injury (SILI) [17].

In study by Valencia et al AUROC for ROX index was 0.72, HACOR was 0.71, these were marginally better than AUROC for respiratory rate 0.69 [13]. In meta-analysis by JP et al ROX index showed a good discriminatory value, sAUC 0.81 with sensitivity of 70% and specificity of 79% for predicting HFNC failure [16]. The reasons for non-inferiority of ROX index in patients with COVID-19 ARDS could be, the factors listed in HACOR score such as HR, GCS and pH if these factors are affected i.e. if sensorium is altered, tachycardia is due to shock/severe hypoxemia, or fall in pH is either due to respiratory acidosis or metabolic acidosis would eventually indicate that patient's needs mechanical ventilation within 24 hours. The criteria of pH was included in this score as this score was initially conceptualized for patients with acute exacerbation of COPD whereas in patients with COVID-19 pneumonia it's basically respiratory alkalosis which is troublesome rather than acidosis. We had eventually 92 patients who were immediately intubated i.e.<24 hours as they fitted into one of these criteria, hence were not included in final analysis. In COVID-19 patients the oxygenation and respiratory rates are usually affected whilst other parameters enlisted in HACOR score such as GCS, HR and pH are usually spared and if either of these three are involved then patient usually lands up in mechanical ventilation. This could be the reason leading to non-inferiority of ROX index in our study, and hence these scores should be interpreted with caution in other causes of respiratory failure besides COVID-19. The advantage of ROX index is it doesn’t require ABG and can be easily calculated bed side.

SOFA score was also one of the most important parameter on multimodal logistic regression. 75% of patients in current study had moderate to severe ARDS with a Respiratory SOFA of 3–4. A further increased SOFA would indicate renal or cardiovascular involvement, as CNS, coagulation SOFA and liver enzymes are relatively less affected. An indirect evidence of CVS and renal involvement may be thought off by higher vasopressor use and higher incidence of AKI in failure group. Age has been the significant factor which was associated with NIV failure, this has been demonstrated in study by Chacko et al [18]. Age has been associated with poor outcomes, various reasons could be poor host defense mechanisms, multiple comorbidities and poor cardiovascular response to hypoxemia, increased risk of AKI. A summary of the studies predicting NIV failure is illustrated in Table 4 [13,17,16,18,19,20,21].

Age, SOFA and ROX D3 were significant factors on multivariate logistic regression model which predicted the failure, this seems logical as SOFA score basically covers each organ system. What additional factor ROX provides us is the RR, SOFA doesn’t provide. Age is important determinant as it decides two most important things host response to infection and above all body's cardiovascular response to hypoxia.

This is the first study to compare the ability to predict NIV failure using HACOR and ROX scores. A serial assessment of scores was done up to 6 days where none of the studies have assessed them for more than a day. Non-inferiority of ROX index, warrants future prospective trials on comparing these scores towards monitoring these patients in non-invasive ways. There is a need to redefine HACOR score for pts with hypoxemic resp failure as more focus should be paid on other parameters rather than traditionally described in HACOR scores such as SOFA, pneumonia, immunosuppression, ARDS and septic shock [11].

The limitation of this study may be thought off in terms of classical limitations associated with tendency of abstraction in a retrospective study, incomplete information of participants and changeability of medical professionals during patient dealing in different phases. However, all the attempts were made to minimize these possible sources of errors by defining research hypothesis a-priori, investigating through multitude of modalities like exploratory visualizations, univariable and multivariable analysis, inclusion of complete cases only in final analysis, running multiple step-up models and using multiple methods to determine convergence.

The ROX score seems to be non-inferior to HACOR score in predicting NIV failure in patients with COVID-19 pneumonia, but this result should be interpreted with caution in other causes of acute respiratory failure. A cut of 4.47 at day-3 for ROX score consists of good discriminatory capacity to predicts NIV failure. From a multivariable perspective, older ages and SOFA score at admission were independent covariables with ROX score to predict NIV failure.