Use of the modified Delphi technique for developing clinical practice guidelines

A brief review

Siddhartha Sharma & Naresh Shetty 

Ethics Committee of Indian Orthopaedic Association 2022

As is the case with other specialties of medicine, Orthopedics practice relies on evidence and consensus guidelines to make best decisions for patients. A number of academic organizations, such as the American Academy of Orthopedic Surgeons (AAOS), the British Orthopedics Association (BOA) etc. have proposed clinical practice guidelines for a variety of orthopedics conditions. However, many of the existing guidelines are difficult to implement in the developing countries owing to several factors, that include, but are not limited to economic and infrastructure restraints, as well as cultural and ethnic differences. Hence, there is a need to develop indigenous best practice consensus guidelines. The modified Delphi method is a widely used method to achieve consensus amongst experts, and can be used to develop clinical practice guidelines (1, 2). In this brief paper, we outline the keys steps for performing a modified Delphi study to develop clinical practice guidelines. For the sake of the reader’s understanding, we quote some examples from published studies available in the literature. The aim of this paper is to provide the reader with an overview of the steps involved in the Delphi methodology, rather than to present an exhaustive review. There are several publications on all aspects of this technique, and readers are encouraged to peruse these. 

1.  Formulation of Steering Committee & Protocol

The modified Delphi study involves a number of well-defined steps. The first step is to formulate a ‘Steering Committee’ which takes care of the study, from its inception until publication. Members of the steering committee are chosen according to their expertise on the given subject. It is advisable to have at least one expert from Biostatistics who has experience with the Delphi methodology. The members of the Steering Committee should then formulate the study protocol a priori, which should be preferably made available online. The protocol can be published in a peer-reviewed journal, or made available on a pre-print server.

2. Ethical Considerations

Although this is not mandatory, ethics clearance should be preferably obtained from the institution of the lead author(s) or member(s) of the steering committee.

3.  Number of Experts

There are no set guidelines on the number of experts that should be included in the panel. Generally, the more the number of experts, the better it is considered. A higher number of experts allows the study investigators to capture the diverse nature of experience and knowledge that experts might have, and to also factor in the geographical and ethnic differences. Experts may be chosen by the study investigators, or invited to participate. The international consensus statement (3) on prosthetic joint infections (PJI) was formulated by 432 experts from all over the world. 

4.  Selection of topics and themes to be deliberated

The topics and themes to be deliberated in the modified Delphi study can be chosen by a systematic review of literature, by asking the expert panel, or both. A systematic review of literature helps in not only identifying the existing literature on the topic, but also helps in determining the strength of evidence. The methodology of the systematic review preceding the Delphi method should follow the PRISMA guidelines (4). If the clinical practice guideline intends to look at several aspects of a problem, multiple systematic reviews may be necessary. It is advisable to divide the issues into several themes. For example, the international consensus statement on PJI (3) looked at 15 different thematic issues which covered several aspects of diagnosis and management. After the systematic review has been completed, the Steering Committee prepares a master list of ‘position statements’ that will be put up to the expert panel for deliberation. 

5.  The modified Delphi process

The modified Delphi study can be conducted in several rounds; however, it is common place to have three rounds. The first round is usually semi-qualitative, and allows the experts to provide comments, suggestions and modifications. The second round allows the experts to determine which statements are of importance for the proposed study.  The final round is usually carried out in person, and allows the experts to determine whether each statement under deliberation should be accepted or rejected. The details of each round are presented below:

1. First Round: The ‘master list’ of position statements is presented to the expert panel. This is done by means of an anonymous, online questionnaire (for e.g., Google Forms, https://www.google.com/forms/about/). The questionnaire is made be available for a set period of time (2-6 weeks). At this stage, the panel decides whether or not the statement is relevant in terms of the study context. Responses from the expert can be graded on a binary scale (agree/disagree) or more preferably, a Likert Scale (1 = strongly disagree, 2 = disagree, 3 = neither disagree nor agree, 4 = agree, 5 = strongly agree). At the end of Round 1, all responses are compiled into a spreadsheet. Each statement is classified as either ‘included’, ‘excluded’ or ‘no consensus’. The thresholds for this should be defined a-priori. One way to do this is presented below.

For example, if a statement is rated as 1 or 2 by of 80% or more (n=12) of the experts, it is classified as ‘excluded’. If a statement is rated as 4 or 5 by 80% of more of the experts, it will be classified as ‘included’. Statements which do not meet the criteria for inclusion or exclusion are classified as ‘no consensus’ ones (Table 2). Any suggestions from the expert panel are also taken into account. At the end of Round 1, all statements that have been graded as ‘included’ or ‘no consensus’ are carried forward into Round 2. If no statement is graded as ‘included’ or ‘no consensus’, the cut-off thresholds should be lowered until a minimum of predefined statements (this can be 10, or more) can be carried forward into Round 2. 

Table 1: An example of how a position statement is graded by the expert panel

b)            Round 2: The aim of Round 2 is to start building the consensus. Statements carried forward from Round 1 (including new statements and modifications, if any) are presented to the expert panel again. The experts are asked to grade each statement again. Similar to Round 1, each statement is graded on a Likert Scale from 1 - 5. The cut-off thresholds described for Round 1 are used, yet again to classify each statement as ‘included’, ‘excluded’ or ‘no consensus’; statements classified as ‘included’ and ‘no consensus’ are carried forward into Round 3. 

Table 2: An example of how threshold criteria can be used for classifying statements in the modified Delphi process 

c)  Round 3: an in-person meeting is conducted with the expert panel. An experienced moderator should conduct this session. Statements carried forward from Round 2 are presented in decreasing order of their rankings to the expert panel. The expert panel is given a chance to reword any ambiguous statements at this stage, however, no new statements are admitted as this stage. The aim of the final round is to vote on each position statement, and not to discuss it in detail. The voting can be done in a binary fashion (agree/disagree) or via the Likert Scale. For example, each statement can be graded from 1-10, with a score of 10 representing the highest level of agreement with the position statement, and 1 representing the highest level of disagreement. 

6. Evaluation of final Results

The final results should be presented as guidelines in a tabular form. There are several ways in which consensus statements can be presented. One of the most common ways is to classify each as ‘We Recommend’, ‘We do not recommend’ or ‘We can neither recommend nor reject’. The international consensus statement on PJI used the following classification criteria: a) simple majority: no consensus: 50.01% to 59% b) supermajority: weak consensus: 60% (1.5:1) to 65% (1.9:1) c) supermajority: consensus: 66% (2:1) to 74% (2.9:1) d) supermajority: strong consensus: ≥ 75% (3:1 or better) e) unanimous: 100%. The level of evidence for each statement should be presented alongside each statement for the reader to understand the strength of evidence.

7. Conclusion

The modified Delphi method follows several well-defined steps to reach consensus amongst experts on a given topic. Within the broad framework presented in this paper, a number of modifications are possible, depending on the scenario under consideration. However, meticulous planning, clear research questions and coherence within the steering committee and expert panel are the key prerequisites to conduct such studies. 

8. References

1. Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Management science. 1963 Apr;9(3):458-67.

2. Eubank BH, Mohtadi NG, Lafave MR, Wiley JP, Bois AJ, Boorman RS, Sheps DM. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC medical research methodology. 2016 Dec;16(1):1-5.

3. Cats-Baril, W., Gehrke, T., Huff, K. et al. International Consensus on Periprosthetic Joint Infection: Description of the Consensus Process. Clin Orthop Relat Res 471, 4065–4075 (2013).

4.Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.