Overview for developing Delphi-based interdisciplinary consensus statements on imaging: pros and cons

Empirical peer-reviewed evidence is the preferred choice when considering diagnostic and management pathways. However, in the absence of such evidence, expert opinion becomes necessary. It is important to acknowledge that opinions can differ among physicians, institutions, countries, and healthcare systems. The Delphi method, when correctly employed, offers a systematic and structured approach to consolidate, assess and summarize limited scientific evidence, enabling a majority of experts to converge toward an optimal solution. Thus, the Delphi method helps to minimize the influence of marginal opinions on the final consensus statements^(1–6).

Based on the literature review and our own experience as leaders and participants in several projects^(1,3–9), in this review paper we provide an overview of the process for conducting Delphi-based interdisciplinary consensus statements on imaging.

It has been shown that several diagnostic and therapeutic pathways vary considerably among international experts and centers, even for frequent and common scenarios in diagnostic imaging including ultrasonography, and related disciplines such as nuclear medicine^(10–11).

For instance, a cartilaginous mass of 5.1 cm in the femur seen on knee radiographs without any associated malignant imaging characteristics such as endosteal scalloping, cortical perforation, periosteal reaction or related soft-tissue tumor may yield various management recommendations of the reporting imaging specialist to the referring physician⁽¹²⁾. The differential diagnosis of five imaging specialists from four continents for this benign cartilaginous tumor included enchondroma, atypical cartilaginous tumor, and lowgrade chondrosarcoma⁽¹²⁾. Management recommendations of the responsible imaging specialist for this short clinical vignette varied between MRI for closer evaluation of the cartilaginous tumor, clinical follow-up without imaging, imaging-based follow-up with radiographs and MRI or referral to an orthopedic oncologist. Finally, clinical management of the benign cartilaginous tumor may include long-term imaging and tumor curettage in rare circumstances such as tumor size enlargement, symptoms linked to enchondroma, or the patient’s concerns about indecision⁽¹²⁾.

Thus, traditional and commonly encountered challenges on imaging, such as scapholunate instability, scaphoid fractures, distal radioulnar joint instability and triangular fibrocartilage complex injuries, clinical indications for musculoskeletal ultrasonography, anorectal fistula, neoplasms and rheumatic disorders, are suitable for interdisciplinary consensus statements on imaging. Moreover, emerging topics with a rapid growing and large amount of literature, such as imaging on femoroacetabular impingement during the past two decades, can be identified as topics for interdisciplinary consensus statements on imaging^(1,3–6,13).

It has been stated that selection of qualified experts is one of the key elements of the Delphi-based process⁽²⁾. Having a substantial number of panelists from diverse countries and continents ensures that consensus statements are rooted in a wide range of expert opinions, enhancing their potential generalizability across various health systems. In contrast, when the expert panel primarily consists of subspecialized imaging specialists with academic backgrounds from a single continent, there is a risk of bias against the viewpoints of non-academic imaging specialists practicing in other regions worldwide^(1,2).

The process of identifying experts may vary. A common and practical approach for the project leader is to invite experts to participate in the project and seek their recommendations for additional experts⁽²⁾. Based on our experience, this method of expert selection can form a dependable group with a swift response time for various tasks, such as survey participation and manuscript editing, while minimizing the occurrence of outliers or dropouts among the experts. One drawback of this expert selection method is that it can introduce selection bias, as experts tend to recommend individuals whose opinions align with their own viewpoints⁽²⁾. An alternative approach to expert selection is to entrust the process to individuals holding key positions within a society, such as the society’s president, chairman of a task group, or an editor or section editor of a relevant journal. The designated representative from the society or journal can then propose suitable experts to serve as panelists responsible for developing the consensus statements⁽¹⁰⁾.

Additional selection criteria for individual panelists may include meeting various qualification benchmarks of clinical or scientific experience, such as fulfilling at least three of the following criteria: a specified number of years of clinical experience, a minimum annual patient caseload related to the Delphi survey topic, proficiency in a range of procedures and techniques relevant to the Delphi survey topic, a minimum quantity of published scientific articles in a particular subspecialty, and experience with receiving or reviewing tertiary referrals⁽¹⁴⁾.

The panelists may be guided by leadership that follows the Primus inter pares principle, which can be embodied by a single individual or a leadership team selected from among the panelists. The leadership individual or team is accountable for overseeing all tasks, which encompass analyzing responses from the remaining panelists, revising consensus statements and related discussions based on panelists’ input and, ultimately, manuscript editing and drafting^(1,3–6). A neutral statistician can provide support for the analyses and ensure adherence to high standards of scientific practice^(1,6).

Relevant bibliographic databases should be consulted to gather the pertinent literature based on specified keywords. Following this step, imaging specialists are welcome to augment the literature database with additional publications related to the consensus statements. Articles that provide scientific evidence for the Delphi process should be stored in a cloud-based directory, ensuring access for all panelists^(1,6). The cloud-based literature folder should be continuously updated during the Delphi process. Importantly, from the outset of the project and throughout the entire Delphi process, all panel members should have access to the relevant study material.

Meetings among panelists facilitate collaboration. They can be conducted on-site, through video-conferences or as hybrid sessions combining both on-site and video-conferencing methods. International conferences, like the European Congress of Radiology (ECR), the annual meetings of the European Society of Musculoskeletal Radiology (ESSR), the International Skeletal Society (ISS), the European Society of Gastrointestinal and Abdominal Radiology (ESGAR) or other similar events, provide an excellent setting for organizing on-site meetings for panelists. It is advisable to brief the panelists about the scientific background, present clinical uncertainties, and imaging challenges related to the topic of the consensus statements. Additionally, the panelists should be informed about their individual tasks, be given an overview of the Delphi process methodology, and be updated on the current results of the Delphi survey^(1,3–6).

After identifying the topic for consensus statements and conducting a thorough literature review, the subsequent step involves formulating relevant questions related to issues requiring expert opinion by consensus. Depending on the topic, experts may formulate such questions or, alternatively, referring physicians, such as surgeons, may be invited to formulate questions related to the imaging aspect of the chosen consensus statement topic. These questions may address the referring physicians’ (such as surgeons’) uncertainties and challenges on imaging, or expectations from imaging studies and imaging specialists in daily clinical decision-making for the selected topic. All questions need to be compiled and structured into specific templates.

Experts, called panelists in the Delphi method, are asked to prepare answers to the relevant questions related to issues requiring expert opinion by consensus statements. Such statements should be based on the published literature, preferentially original articles, and metaanalyses. If a question lacks adequate scientific evidence to support it, the corresponding statements will be based on expert opinion. Nevertheless, it is imperative to distinctly highlight and acknowledge when an expert opinion is being presented. Panelists may be grouped into small task teams based on their expertise, with one designated as the group leader. Subsequently, every question requiring expert opinion by consensus statements should be allocated to a specific small task team of three experts. The dataset created by the experts should comprise three key components: a response, referred to as a ‘statement,’ directly addressing the question; a comprehensive discussion detailing their insights and expertise on the topic, while also providing an explanation for the statement; and a list of references supporting their statement. Additionally, teams are asked to assign an evidence level to each reference, utilizing the five-point scale set by the Oxford Centre for Evidence-Based Medicine^(1,3–6,15). These refined statements will serve as the basis for the interdisciplinary consensus statements on imaging^(1,6).

The strength of scientific evidence supporting each consensus statement can be rated by identifying the article with the most significant impact on the statement, offering the highest level of applicable scientific evidence. According to the five-point scale established by the Oxford Centre for Evidence-Based Medicine Meta-Analyses, Systematic Reviews of blinded cross-sectional studies with consistent reference standards are categorized as evidence level ‘1’^(15,16). Original articles fall within evidence levels ‘2’ to ‘4’, while blinded crosssectional studies with consistent reference standards are placed at evidence level ‘2’^(15,17–19). Editorials, commentaries, case reports, and review articles are assigned evidence level ‘5’, indicating reliance on expert opinion or mechanism-based reasoning^(15,20–22).

The Delphi method is a prominent consensus-building technique used in medical research, designed for highly structured group interactions to achieve consensus among panelists. It proves especially beneficial when empirical evidence is limited, or when scientific knowledge is either ambiguous or contradictory. The Delphi process is a validated method for achieving consensus through repeated presentations of statements or corresponding answers. These are assessed over multiple iterative rounds. Sources of these statements include clinical practice, literature reviews, or findings from prior research. Participants evaluate the same statements across multiple Delphi rounds, with feedback from preceding rounds available, ensuring that each participant reviews the same or revised statement multiple times^{(1–6,23–29)}.

As the group (referred to as a panel) provides feedback, it might influence individual judgments in the consecutive Delphi rounds. Reading the perspectives of their peers, participants might consider revising their responses to obtain a consensus/agreement for a given statement. Concomitantly, the Delphi process maintains anonymity, which helps prevent undue influence or panel interaction effects^{(1–6,23–29)}.

A practical method to quantify the level of agreement of each panelist to a specific statement addressing a referring physician’s question is to use an 11-point Likert scale, where 0 represents complete disagreement, 5 indicates neutrality (neither agreement nor disagreement), values of 8 or above (≥8) reflect agreement, and 10 implicates complete agreement. This approach suggests a minimum agreement threshold of 8 out of 10 on the 11-point Likert scale^(1,6). ‘Delphi-Manager’, a web-based system designed to facilitate the building and management of Delphi surveys, along with ‘Google Sheets’ for conducting Delphi Rounds, and ‘Microsoft Forms’ for the rating exercises, are all useful tools in this process.

In the initial Delphi round, panelists are encouraged to provide feedback on the wording and content of the survey’s questions and statements, especially if their rating indicates anything less than full agreement (a score of ≤7). The feedback can lead to the rephrasing of statements in subsequent Delphi rounds or even to the addition of new questions and statements. Consequently, the questions and statements might undergo cyclical revisions by the panel leaders based on the panelists’ scores, recommendations, and remarks from one round to the next. For example, the second and third Delphi rounds should incorporate both the updated and expanded questions and statements, as well as the original ones from the preceding round. It might be beneficial to include statistics and visual representations, like graphs, to showcase the group’s consensus level for each statement from the previous Delphi round. This information enables participants to adjust their rankings based on the feedback from others. Panelists who do not vote in one round will not be invited for subsequent rounds.

Criteria to stop the Delphi process is an important topic. Thus, it is essential to determine and inform all panelists of the maximum number of Delphi process rounds before the first round begins. For instance, the Delphi process may be restricted to three rounds or until consensus is reached for each individual statement, whichever happens first. The goal is to achieve consensus or, if it is unattainable, to determine the level of agreement for each statement. If consensus cannot be reached, the degree of agreement for each statement will be assessed. It is advisable to use survey administration software in the process^{(1–6,23–29)}.

Although standards for achieving consensus in Delphi surveys remain undefined, the steering committee must agree on consensus definition prior to the Delphi rounds. A systematic review has shown that the median threshold for consensus in Delphi studies is 75%, with a varied range spanning from 50% to 97%. For instance, a Delphi survey may set the group consensus threshold at 80% or higher, with panelists rating their agreement level as ‘8’, ‘9’, or ‘10’. Considering these preconditions, a ‘group consensus’ is not established if fewer than 80% of all panelists rate the agreement level of the statement as ‘8’, ‘9’, or ‘10’. The higher the threshold, the more refined the questions must be in subsequent Delphi rounds to yield acceptance by the majority of panelists. Alternatively, it may be suggested to categorize group consensus into ‘low group consensus’ and ‘high group consensus’, based on varying levels of consensus among the group. Median and interquartile range values can serve as additional measures to reflect polarization among panelists. Importantly, panelists should be encouraged to suggest improvements to the statements between the Delphi rounds, especially if any panelist disagrees with an individual statement^{(1,3–6,23,29)}.

The outcomes of a Delphi survey typically feature a literature research summary for each question, coupled with a consensus statement detailing the level of scientific evidence as per the Oxford Centre for Evidence-Based Medicine. It is essential to highlight the areas where scientific evidence is either lacking or contradictory concerning any question or consensus statement. This approach not only offers clarity but also prompts fellow researchers to delve into uncharted territories.

It is recommended to report the number of statements that achieved group consensus across the first, second, or third Delphi rounds, as well as those that did not reach consensus. Subsequently, a detailed list of all individual questions and consensus statements can be provided, showcasing group consensus levels in terms of absolute numbers, percentages, medians, and interquartile ranges of panelist agreement. Complementing the results with tables, graphs, illustrations, and an in-depth discussion section can offer further insights and clarity to readers^(1,3–6,13).

In general, every scientific project is susceptible to failure at any stage, and possessing prior knowledge of significant challenges and pitfalls is crucial for minimizing or bypassing them. Discipline among all panelists and a strong team spirit are paramount for reaching consensus in Delphi-based interdisciplinary statements on imaging, especially when dealing with a large number of participants. Therefore, a carefully selected group of panelists has the potential to ensure reliable, swift, and successful completion of both individual tasks and the entire project. The ‘Cons’ section of Tab. 1 entitled ‘Fundamentals of the Delphi Methodology’ outlines several potential pitfalls. Nevertheless, it is essential to also consider the following more significant pitfalls listed below.

The formulation of questions holds significant importance. Vague questions pose a high risk of hindering concise statements and consensus among panelists. The absence or inadequacy of explicit guidance and methodological direction, including a lack of defined methods and delays in executing the Delphi survey, greatly increase the likelihood of failure in successfully concluding Delphi-based consensus statements on imaging⁽³⁰⁾.

The allocation of responsibilities and time required from each panelist for various contributions, such as participating in meetings, conducting literature research, suggesting questions and statements, preparing discussions, along with reference lists, and committing to Delphi surveys, should align with the time resources available to the panelists. Failure to do so may result in delays and frustrations. Furthermore, maintaining frequent and prompt communication between project leaders and panelists is crucial for sustaining commitment and preventing panelist resignations⁽³⁰⁾. An unlimited number of Delphi rounds can prolong the consensus-building process and lead to panelist fatigue. Therefore, it is advisable to inform all panelists in advance that the number of Delphi rounds is limited and to establish criteria for halting the Delphi process before the first round commences⁽³⁰⁾.

Finally, an effort should be made to publish the consensus statements promptly to prevent the need to restart or update the entire process in response to new scientific evidence, which could make the statements appear outdated.

This review paper offers a concise guide on executing Delphi-based interdisciplinary consensus statements concerning imaging. Fundamentals of the Delphi methodology, including pros and cons, are presented in Tab. 1. The most important principles are listed below: •

Anonymity among participants: This ensures no single individual or group applies too much influence over others.

These principles underscore the importance of statistical group responses and a structured interaction inherent in the Delphi methodology^{(1–6,23–29)}.

In a broad view, the process of formulating Delphi-based interdisciplinary consensus statements on imaging unfolds in seven stages, as presented in Fig. 1: •

Identifying the research field that requires consensus statements.

Fig. 1.

Flowchart of Delphi-based interdisciplinary consensus statements on imaging

Lastly, summarizing and reporting the finalized Delphi-based interdisciplinary consensus statements on imaging^{(1–6,23–29)}.