What we know about Delphi Techniques: A Bibliometric Analysis for the Health Sciences

Julia Schifano^*, Larissa Karl, Marlen Niederberger

Department of Research Methods in Health Promotion and Prevention, Institute of Health Sciences, University of Education Schwäbisch Gmünd, Germany

*Corresponding Autho: Julia Schifano, Department of Research Methods in Health Promotion and Prevention, Institute of Health Sciences, University of Education Schwäbisch Gmünd, Germany

Received: 08 October 2025; Accepted: 16 October 2025; Published: 25 November 2025

Introduction

The Delphi technique is a structured, multi-stage communication process in which experts assess uncertain and complex issues [1–3]. Its use in health sciences is on the rise, though the methodological design varies to some extent between the different disciplines. These differences could be traced back to discipline-specific thought styles, i.e., collectively shared epistemic routines and frames of reference [4]. According to Hurrelmann et al. [5], the disciplines of health sciences can be divided into those that tend to follow a medical-scientific paradigm (primarily medicine, psychiatry, and neurology) and those that tend to follow a social-behavioral paradigm (primarily health sociology, health management, health economics and health politics). However, there is currently a lack of systematic analyses that reveal possible disciplinary differences in the application of the Delphi technique. Bibliometric analyses can be used to reveal the structure of a research field via citation networks. Novello [6] investigated knowledge production and circulation based on citation networks in the mixed-methods community. Previous bibliometric research on Delphi techniques have mostly focused on dissemination or thematic priorities without examining discipline-specific methodological practices in detail [7–9]. This study examines this research gap through a bibliometric analysis of Delphi primary studies in health sciences. The aim is to use citation networks to identify potential discipline-specific thought styles and examine their importance in the methodological design of Delphi techniques.

Epistemological background to the Delphi technique

The origins of the Delphi technique date back to the 1950s. At that time, the Research and Development (RAND) Corporation in the US conducted a Delphi study to forecast military developments [1–3]. The name of the technique was derived from the analogy between the procedure and the prophecies of the Oracle of Delphi from Greek mythology [1]. Dalkey and Helmer [3] define the Delphi technique as „[…] the most reliable consensus of opinion of a group of experts. It attempts to achieve this by a series of intensive questionnaires interspersed with controlled opinion feedback" [3]. Publications on Delphi techniques in health sciences now also refer to other definitions, e.g. by Hsu and Sandford [10], Jones and Hunter [11], Hasson et al. [12], Diamond et al. [13], Boulkedid et al. [14]. These definitions state the same characteristics of the Delphi technique, namely the repeated questioning of experts in at least two rounds using a questionnaire, the integration of feedback from the second round onwards, as well as the anonymity of the respondents. The Delphi technique was originally used to forecast possible futures and has been continuously reflected and further developed due to new technical possibilities and demands on knowledge-generating research [15]. In the 2010s, computer-based survey procedures became the standard, enabling the implementation of variants such as Real-Time Delphi [1]. In addition to the further development of the technique due to technical advances, the range of disciplines in which Delphi techniques are used has increased and become more differentiated [9]. Especially in health sciences there has been an enormous increase in publications on Delphi techniques since the 1990s [9]. Here, Delphi techniques typically aim to reach consensus [16]. Methodological discussions in health sciences currently focus on issues such as the participation of lifeworld experts, e.g., patients, as part of the expert panel [17,18] or the combination of survey modes, e.g., workshops and written questionnaires [19,20]. This has also led to new variants of the Delphi technique, e.g., the group Delphi [21] or the Café Delphi [19].

Methodological analyses suggest that the application of Delphi techniques varies across different health science disciplines [22]. Epistemologically, discipline-specific differences are captured by the concept of thought styles [4]. Thought collectives are communities of people who share a particular thought style, i.e., who draw on specific bodies of knowledge and certain cultural practices, also with a view to developing them further or rethink them [4]. Accordingly, the production of knowledge varies between disciplines due to different socially and culturally developed structures and ways of thinking, or “thought styles.” Scientific publications can be used to empirically shed light on thought styles [6,23,24]. These represent preliminary and personal knowledge that, over time, through further citations or use, is transformed into collective “handbook science” [4,23]. Through their presentations of their scientific work, authors influence which references readers see, use, and possibly cite in further scientific publications [6,24]. Thus, bibliometric analyses of scientific publications on Delphi techniques can be used to examine thought styles, namely 1. via reported methodological practices and 2. via historical developments and dynamics. We assume that the thought styles explain the differences in the methodological designs of Delphi techniques.

State of research: Bibliometric analyses of publications on Delphi techniques 

Bibliometric research usually analyzes publications quantitatively and descriptively, e.g., to find out who has published how many publications in a given subject area, which authors and references are cited, to map networks between authors or institutions, or to determine which knowledge base is being drawn upon [6,25]. According to Öztürk et al. [26], bibliometric research is conducted in four steps (Figure 1).

Figure 1: Process of bibliometric research according to Öztürk et al. [26] (own illustration).

• 1) Defining the aim of the research: This requires knowledge of the current state of research on the topic. This can be achieved, for example, through a systematic review.
• 2) Collecting data on the relevant literature: The publications for the bibliometric analysis are obtained from a digital database (e.g., Scopus, Web of Science). Suitable databases are those that are recognized in the respective professional community, offer a broad base of publications that are as freely accessible as possible, and in which cited references can be stored as metadata and exported [27–29]. PubMed (as of July 2025) is therefore not suitable for network analysis because cited references cannot be exported. The publications are screened according to defined inclusion criteria, and the included publications, together with their cited references, form the dataset for the analysis.
• 3) Analysis and visualization: Before analysis, the data is cleaned up to correct incomplete information and duplicates. Due to the large amount of data in the cited references, it is not always possible to check each piece of information individually. Therefore, cited references that are rarely cited in the dataset (n≤10) are often not taken into account. This is because they do not appear in later bibliometric analyses and are unlikely to influence the central clusters, such as the most frequently cited references [30]. The most common methods of bibliometric evaluation include performance analysis, i.e., descriptive analysis, e.g., by number of publications per year or per journal, and science mapping methods. These include co-citation analysis, which examines the frequency and distribution of citations [6], or bibliographic coupling, which identifies similarities between publications based on shared references [28]. Special software, such as the free tool VOSviewer [30,31], may be necessary for analyzing and visualizing bibliometric data.
• 4) Interpreting the findings and results: Finally, the quantitative analyses must be interpreted. This usually requires a certain amount of expertise and follow-up research in the relevant field [26]. Abstracts of the publications or cited references can also be used in the analysis in order to interpret the clusters from co-citation analysis or bibliographic coupling.

To date, only a few bibliometric analyses on Delphi techniques have been published (Table 1). The prevalence of the Delphi technique is explored by subject area, as is the frequency with which certain topics appear in publications. For example, the number of publications with a methodological focus on the Delphi technique is analyzed (Table 1).

Table 1: Bibliometric research on Delphi techniques.

¹For better readability, the figures have been rounded without decimal places. This may result in rounding inaccuracies.

The findings show that primary studies using Delphi techniques are published more frequently than methodological studies on Delphi techniques [7–9]. The analysis by Calleo and Pilla [32] identifies key topics from Delphi studies on scenario building in the field of climate change research, e.g., the combination of Delphi techniques with other methods. Bibliometric analyses show that Delphi techniques are increasingly being used in health sciences. However, they do not allow conclusions to be drawn about possible discipline-specific thought styles, as they did not perform co-citation analyses. Based on data from scientific publications on primary studies using Delphi techniques, this bibliometric analysis therefore examines the following overarching research question:

Can discipline-specific thought styles in Delphi techniques in the health sciences be identified by analyzing relevant publications on Delphi techniques?

In order to answer the overarching research question, the following sub-questions will be examined:

1. Which authors and references are cited in health science publications reporting on Delphi studies, and can clusters or networks be identified among the cited references and authors?
2. Can the clusters or networks be assigned to different disciplines (medical-scientific or social-behavioral sciences)?

Methods

For a descriptive overview of the dataset, we evaluate how many scientific publications on Delphi techniques in health sciences are published per year, per journal, and per author. In order to answer the research questions, we analyze the cited references, i.e., the bibliography of the publications. It should be noted that the content focus of the cited references is not restricted. We evaluate the bibliographies using a co-citation analysis of the cited references and authors (Figure 2). We follow the recommendations for conducting and reporting bibliometric research according to Öztürk et al. [26].

Figure 2: Methodological approach (own illustration).

Data collection

The data basis for the bibliometric analysis consists of scientific publications on Delphi primary studies in the health sciences (Table 2). The literature search is conducted in the Web of Science database (https://www.webofscience.com), as it contains Delphi studies from the health sciences that can be assigned to different disciplines.

Table 2: Inclusion and exclusion criteria for the screening process of the Delphi primary studies from the health sciences.

The search is limited to scientific publications (filter: Article) published between 2017 and 2023 that contain the keywords “Delphi” in the title and “health*” in the title or abstract ((TI=(delphi)) AND TS=(health*)). The keywords have already been used in previous studies to identify publications on Delphi studies in health sciences [33,34].  Subsequently, a title-abstract screening of the scientific publications is performed using Rayyan software [35]. Original research in English on Delphi primary studies in the health sciences are included, regardless of the Delphi variant (e.g., classic Delphi, modified Delphi, e-Delphi, real-time Delphi) and the specific research question (Table 2). The title-abstract screening is carried out by four trained scientists (LK, RA, LJ, JS) and supervised by a leader (JS). The search query on April 23, 2024, yielded 1,843 publications in the Web of Science database. N=223 publications were excluded since they did not meet the inclusion criteria or no information on the cited references were available (Figure 3). In n=58 cases, the decision on whether to include or exclude the publication was discussed among the authors. N=1,618 scientific publications on Delphi primary studies were included and cleaned. They formed the dataset for the analysis.

Figure 3: Flowchart for searching for literature in Web of Science (own illustration).

Data cleaning

The dataset is cleaned using the Software VOSviewer Version 1.6.20 (https://www.vosviewer.com/) prior to analysis. Cited references from the included scientific publications on Delphi primary studies that are listed at least ten times across all publications (n=169) are exported. Incomplete information was deleted and inconsistent information (e.g., inconsistent citation of a reference with regard to the spelling of the author's name with and without the middle name) was corrected. For one cited reference (Hsu, C. & Sandford, B. A., (2007) “Minimizing Non-Response in The Delphi Process: How to Respond to Non-Response”, Practical Assessment, Research, and Evaluation 12(1): 17. doi: https://doi.org/10.7275/by88-4025) different DOI numbers are referenced (e.g. 10.1016/S0169-2070(99)00018-7) or no DOIs are specified which is why we cannot take this reference into account in the analysis. The cleaned dataset can be provided by the authors upon request. It contains a total of n=55,137 cited references and n=42,906 cited authors.

Bibliometric analysis

The cleaned dataset is analyzed descriptively and presented graphically using VOSviewer (Version 1.6.20). For visualization purposes, a minimum number of citations is specified, which determines which cited authors or cited references are included in the graph. For large datasets, a setting of at least 30 citations is recommended [31].

Identification of the clusters

According to Steinhardt et al. [36], the six most frequently cited publications may be sufficient to interpret the calculated clusters in a meaningful way. Therefore, we select the settings for creating the graphs in VOSviewer so that each cluster contains at least six references or authors and remains relatively stable even if the citation threshold is changed by ±5. If the clusters prove to be stable, the citation threshold is set as high as possible to enable a clearer presentation. This reduces the number of overlapping data points and makes the most important cited references and authors more visible in the graphics.

Additional internet research

In order to interpret the clusters from the bibliometric analyses, an internet research is conducted to supplement the information from the VOSviewer (e.g., citations, links) with information on the cited references (e.g., abstracts) and cited authors (e.g., institutional affiliation, country, expertise, number of publications on Delphi techniques (scientific publications, book contributions, gray literature)). The research on April 14, 2025 is conducted and documented for all cited references and authors from the clusters formed (Supplementary File 1). Information about the authors is researched via ResearchGate, the author´s Open Researcher and Contributer ID (ORCID) entry, or institutional websites with Curriculum Vitaes and publication lists. The source of the information is documented (Supplementary File 1).

Interpretation of the clusters

The clusters from the co-citation analyses are interpreted by the team of authors. This requires a certain amount of experiential knowledge. We have conducted Delphi studies in various contexts ourselves, conduct methodological research on Delphi techniques, and have already published several articles on the subject [33,37,38]. We are also part of a scientific network on Delphi techniques in health and social sciences (for more information on the authors and current Delphi projects, see: https://www.ph-gmuend.de/hochschule/fakultaeten/fakultaet-i/institut-fuer-gesundheitswissenschaften/forschungsmethoden-in-der-gesundheitsfoerderung-und-praevention/delphi).

In the co-citation analysis of the cited references, we use the abstracts to identify thematic similarities within the clusters and differences between the clusters. In addition, we make a discipline-specific allocation according to the differentiation proposed by Hurrelmann et al. [5] (medical-scientific or social-behavioral sciences). In the co-citation analysis of the cited authors, we assign the authors to the two discipline categories based on information from the internet research. In all analyses, we compare the number of citations per cluster and describe differences in content and structure.

Results

We included a total of 1,618 health science publications with Delphi studies in the bibliometric analysis. The number of publications tripled between 2017 and 2023 and declined slightly in 2022. Ten authors published six or more scientific publications on studies using Delphi technique during the seven-year study period (Table 3). The order of authors is not taken into account here. The author with most publications (Reavley) has published a total of 20 publications. The two journals with the most publications on Delphi studies are the British Medical Journal (BMJ) Open and the Journal Public Library of Science (Plos) One.

Table 3: Health science publications with Delphi studies (n=1,618) per year, per journal und per author.

*n=41 publications were not published until 2024, but were nevertheless included in the search criteria (years 2017-2023), presumably because they had already been published online in advance in 2023.

Co-citation analysis of the cited references

Below, we answer which references are cited in health science publications using the Delphi technique and which clusters or networks can be identified in the cited references.

Description of the cited references

The co-citation analysis of the cited references shows that n=41 of the references in the dataset are cited at least 35 times (Figure 4). Frequently cited references and strongly linked references are represented by a larger dot, such as Hasson et al. [12], Boulkedid et al. [14], Hsu and Sandford [10] and Diamond et al. [13]. Of the references cited, n=33 explicitly refer to Delphi techniques (Supplementary File 1). The other references deal with various methods of consensus building (n=5), e.g., RAND/UCLA Appropriateness Method, thematic analysis (n=1), a software (REDCap) (n=1), or the development of core outcome sets (n=1). Methodological publications, e.g., discussion papers, systematic reviews, from the health sciences are cited more frequently than classical methodological key literature on Delphi techniques [3,39]. The references cited have been published as book contributions, gray literature, or in journals. In addition to journals that frequently publish articles with a methodological focus on Delphi techniques (e.g., Technological Forecasting and Social Change, BMC Medical Research Methodology, Journal of Clinical Epidemiology), journals from specific disciplines or with a thematic focus are also represented (e.g., Journal of Advanced Nursing, PLoS Medicine, Palliative Medicine).

Cluster-analysis of the cited references

VOSviewer divides the cited references into four clusters (Figure 4): Reference cluster A (gray, n=18), reference cluster B (yellow, n=9), reference cluster C (blue, n=7), and reference cluster D (pink, n=7). Interpreting the content of the reference clusters is challenging, as the references are diverse and it is difficult to identify similarities and differences.

Figure 4: Co-citation analysis of the cited references (n = 41) with a minimum of n = 35 citations.

Explanation: Links are displayed when different publications use the same references. Frequently cited references and strongly linked references are represented by a larger dot. Technical details: normalization = association strength, layout = 5 attraction/1 repulsion, clustering resolution = 1.00; visualization = total link strength; min cluster size = 1.

We therefore used the abstracts to identify the core topics of the reference clusters inductively. Based on our expertise in Delphi techniques, we then contrasted the thematic similarities and differences between the reference clusters (Table 4). We classified the references according to subject area based on their thematic focus and assigned the reference clusters to either the medical-scientific or social-behavioral sciences.

• • Reference cluster A „epistemology“ (shown in gray in Figure 4): The references cited here are about methodically classifying Delphi techniques. It contains methodological key literature on Delphi techniques, which is rooted in social and behavioral sciences. Among other things, methodological challenges of Delphi (e.g., inconsistent use of terminology, selection of experts, quality criteria) and conceptual further development (e.g., e-Delphi) are discussed, partly in specific contexts of health sciences and nursing sciences. The focus is on a conceptual/theoretical level, and the content focus is based on epistemology and reflection. Hasson et al. [12] is the most frequently cited reference (n=398) in reference cluster A and across all reference clusters (Table 4). Keeney et al. [40] (n=148) and Okoli and Pawlowski [41] (n=147) are also among the most frequently cited references in reference cluster A (Table 4). The three references address, among other things, epistemologically relevant questions concerning knowledge production in Delphi techniques, e.g., the collection of expert knowledge, consensus building, and the role of researchers in Delphi studies. Compared to the other reference clusters, the literature tends to be somewhat older, ranging from publications from 1963 to 2012.
• • Reference cluster B „Delphi application“ (shown in yellow in Figure 4): Reference cluster B contains references that describe the application of the Delphi technique in the health sector, with a medical-scientific focus. There are, for example, recommendations on conducting Delphi studies based on systematic reviews [14,42,43], manuals/user guides on consensus methods based on empirical knowledge [44–46], and an introduction to the REDCap software [47]. One focus is on the goal of reaching consensus. We assign reference cluster B to a practical-operational level and consider it to be significantly more application-oriented than reference cluster A, “epistemology.” The most frequently cited references here include Boulkedid et al. [14] (n=197), Jones and Hunter [11] (n=145), and Fitch et al. [44] (n=84) (Table 2). The references address the objectives of the Delphi technique in healthcare, e.g., the development of quality indicators for healthcare, consensus building in areas of uncertain knowledge, and the optimization of healthcare services.
• • Reference cluster C „quality assurance“ (shown in blue in Figure 4): Reference cluster C is dedicated to the methodological quality assurance of Delphi techniques, with a focus on the topics of consensus measurement and stability measurement, errors in application, and the further development of statistical procedures [13,48,49]. Some of the references reflect the use of Delphi techniques in health sciences as a whole and provide guidance on implementation, similar to reference cluster B “Delphi application,” but with a stronger focus on quality assurance of Delphi techniques [13,50–52]. Reference cluster C operates on a methodological-evaluative level in terms of content and cannot be assigned exclusively to the social-behavioral sciences or the medical-scientific. The most frequently cited references here include Diamond et al. [13] (n=261), von der Gracht [48] (n=128) and Trevelyan and Robinson [50] (n=88) (Table 4). All three publications address consensus determination and which aspects are rele-vant for the quality of Delphi studies.
• • Reference cluster D „methodological reflection“ (shown in pink in Figure 4): Reference cluster D focuses on the application of the Delphi technique to current and complex topics in health research (e.g., mental health), combined with methodological reflection [38,53–56]. As with reference cluster C „quality assurance“, no disciplinary classification is possible here either. What is striking is the frequent citation of the reference by Braun and Clarke [57] (n=67) on the application of thematic analysis in psychological studies, without any direct connection to Delphi techniques. We see reference cluster D as essentially methodical and application-oriented: it is application-oriented like reference cluster B “Delphi application,” but with a stronger focus on methodological discussion. The most frequently cited references include Hsu and Sandford [10] (n=186), Jünger et al. [53] (n=181) and Jorm [54] (n=103) (Table 4). The references are, on average, slightly newer than those in the other reference clusters in terms of publication date, dating from 2006 to 2021. None of the other reference clusters contain references from 2020 or later, reference cluster D contains two, Niederberger and Spranger [38] (n=63) and Nasa et al. [55] (n=40).

Table 4: Overview of topics per reference cluster.

The connecting lines show that more frequently cited references are more strongly linked between the four clusters (e.g., 10,12,48). Some references are linked within a reference cluster, particularly in reference cluster B “Delphi application” (e.g., 42, 44, 46). Reference cluster A “epistemology“ and reference cluster B “Delphi application” are further apart than reference cluster A “epistemology” and reference cluster D “methodological reflection” as well as reference cluster C “quality assurance” and reference cluster D “methodological reflection”, which are comparatively close to each other (Figure 4). Reference cluster D “methodological reflection” overlaps with all clusters.

Co-citation analysis of the cited authors

Below, we answer which authors are cited in health science publications using the Delphi technique and which clusters or networks can be identified among the cited authors.

Description of the cited authors       

The co-citation analysis of the cited authors shows that n=52 authors are cited at least 35 times in the dataset, regardless of the publication (Figure 5). Authors who are cited particularly frequently and are linked to other authors multiple times are represented by a larger dot, e.g., Keeney, Hasson, World Health Organization. The additional internet research into the publications on Delphi techniques by the authors cited (n=52) revealed that n=23 authors had published a maximum of two publications on Delphi techniques, while n=11 authors had published ten or more publications. For n=3 authors, we were unable to find any further information (Supplementary File 1). Through our internet research on the expertise of the n=52 authors, we were able to determine that the authors belong to different disciplines. It is striking that the majority of authors have expertise in the field of statistics (e.g., epidemiology, biostatistics, mathematics) (Supplementary File). In terms of geography, most of the cited authors are affiliated with institutions in the UK (n=20), followed by the US (n=10), EU countries (n=7), and Canada (n=5). Countries in Asia (n=4) and Africa (1) are represented sporadically.

Cluster analysis of the cited authors

The cited authors were divided into four clusters using VOSviewer: author cluster 1 (gray, n=15), author cluster 2 (yellow, n=14), author cluster 3 (blue, n=14) und author cluster 4 (pink, n=9) (see Figure 5).

Figure 5: Co-citation analysis of the cited authors (n = 52) with a minimum of n = 35 citations.

Explanation: Links are shown when authors are cited together. Frequently cited authors and authors with strong links to each other are represented by a larger dot. Technical details: normalization = association strength, layout = 4 attraction/1 repulsion, clustering resolution = 1.00; visualisation = total link, strength; minimum cluster size = 6.

Based on the internet research on the authors, we described the publication output of the cited authors, contrasted differences in the number of citations of the authors between the author clusters, i.e., how often they were cited, and determined the disciplinary classification based on expertise and research focus (Table 5).

• • Author cluster 1 “decision management/knowledge management“ (shown in gray in Figure 5): Author cluster 1 has the greatest overall reach, measured by the total number of citations across all authors in the author cluster. The median number of publications on Delphi techniques is highest, at four publications. Hasson is the most frequently cited author (n=441) and, according to our internet research, has published n=16 Delphi publications. Von der Gracht has published the highest number of Delphi publications in author cluster 1 (n=29) and ranks sixth among authors in terms of citation frequency (n=134). The majority of the authors cited can be assigned to social-behavioral science disciplines (Supplementary File 1). Their typical research focuses on interdisciplinary decision support, futurology, health management, and strategic management.
• • Author cluster 2 “evidence synthesis“ (shown in yellow in Figure 5): Author cluster 2 has the lowest number of publications based on the arithmetic mean of the authors' Delphi publications (mean = 8.5) and a similar reach as author cluster 3 “subject specifics,” measured by the total number of citations (n = 1,096). The median number of Delphi publications by the authors is 3.0. Here, Boulkedid is the most frequently cited author (n=204), with n=9 publications on Delphi techniques. Jones (n=148) and Fitch (n=95) are in second and third place. No publications with Delphi in the title could be attributed to either author. However, Jones has published on consensus-building methods in general, and Fitch specifically on the RAND/UCLA Appropriateness Method. Williamson ranks seventh (n=75) in terms of citations in author cluster 2, with the most Delphi publications (n=45). The majority of cited authors from author cluster 2 can be assigned to medical-scientific disciplines (Supplementary File 1). Typical research focuses include clinical studies, evidence-based practice, epidemiology, biostatistics, and public health.
• • Author cluster 3 “subject specifics“ (shown in blue in Figure 5): Author cluster 3 has the lowest overall reach, measured by the number of citations across all authors in the author cluster (1,038), and also the lowest median number of Delphi publications (median=1). Diamond has published one paper on Delphi techniques and is the most frequently cited author (n=263). With n=41 publications, Niederberger is the author with the most Delphi publications in author cluster 3 and ranks fourth in terms of citations (n=73) (Supplementary File 1). The discipline-specific focus is less clear in author cluster 3, which is why we cannot clearly assign this author cluster to either of the two discipline categories (Supplementary File 1). The authors' typical research foci are qualitative research, educational innovation and health literacy, sociology, and public policy.
• • Author cluster 4 “social mandate/transformation“ (shown in pink in Figure 5): Author cluster 4 has the highest publication output based on the arithmetic mean of Delphi publications by the cited authors, but a lower median number of Delphi publications than author clusters 1 “decision management/knowledge management” and 2 “evidence synthesis.” The reach of the authors in author cluster 4 is slightly higher than that of author clusters 2 “evidence synthesis” and 3 “subject-specifics”, with n=1,190 citations. We were able to assign n=2 publications on Delphi techniques to the World Health Organization, and it is the most cited author (n=497) in author cluster 4. Jorm is the author with most Delphi publications (n=45) and, with n=149 citations, ranks second among the most frequently cited authors in author cluster 4. A disciplinary classification is not possible for author cluster 4, either. The authors typically do research on psychosocial health, inclusion, health inequality, vulnerable groups, digitalization and rehabilitation.

Table 5: Overview of topics per author cluster.

¹In order to make it easier to read and assign the author clusters, we have named the clusters according to research focuses or the core expertise of the authors in each cluster. However, the authors per cluster are very mixed and cannot be clearly assigned to one focus.

Compared to the reference clusters (Figure 4), it is noticeable that the author clusters (Figure 5) also frequently overlap. However, here too, author cluster 1 “decision manage-ment/knowledge management” with social-behavioral science grounding of the disciplines and author cluster 2 “evidence synthesis” with medical-scientific grounding of the disci-plines are further apart than author cluster 1 “decision management/knowledge management” and author cluster 3 “subject specifics” or author cluster 2 “evidence synthesis” and author cluster 3 “subject specifics”. Author cluster 4 “social mandate/transformation” has overlaps with all author clusters. Figure 6 provides a summary overview of the reference clusters and author clusters.

Figure 6: Summary of the reference clusters and author clusters (own illustration).

Discussion

The analysis of the cited references and authors provides information on who is frequently cited in primary health science studies in which Delphi techniques were used. The results show that these are mainly methodological publications on Delphi techniques. They are published in various journals that match the topic of the publication on Delphi techniques. It is notable that some authors, despite having published only sporadically on Delphi techniques, are cited more frequently than those with a greater number of relevant publications. One possible explanation for this lies in the reception of certain publications as reference works, which are perceived as particularly influential, easily accessible or connectable and thus also shape the thought style of certain disciplines. Citations are also based on these factors and not exclusively on the discipline-specific or methodological fit or the publication frequency of the authors [6]. The so-called Matthew effect [58] can also play a role: Authors or publications that have already been frequently cited are more likely to be cited again, which further increases their visibility.

In addition, it becomes clear that methodological publications without direct reference to Delphi techniques are also cited. These include the thematic analysis according to Braun and Clarke [57], which is used in Delphi studies as an analysis strategy for open responses [59], or publications on software tools such as REDCap [47], which can generally be used for survey procedures. We can identify a number of potential reasons why Delphi-specific literature on analysis strategies or software tools (e.g., [59], [60]) is cited less frequently. First, it could be because Delphi users are not familiar with these specific Delphi-related procedures and tools or the publications are too new so that they have not yet been sufficiently disseminated or recognized as a standard, such as the Argument-based QUalitative Analysis strategy (AQUA) for analyzing free-text responses in Delphi studies, published in 2023 [59]. It could also be due to the fact that certain methodological approaches and software tools are already well-established within the discipline and are considered reliable. It is possible that these approaches and tools have been institutionalized as a thought collective through many years of use in the professional community and are taught in relevant method books and study courses [4]. This leads to alternative, possibly more specific methods or software solutions receiving less attention, even if they are potentially more suitable.

Methodological publications on Delphi techniques from other disciplines, such as futurology or the social sciences, are also rarely cited. However, bibliometric analyses on Delphi techniques show that methodological research on Delphi techniques is taking place in these areas in particular [9]. These include, for example, methodological tests and discussions, new Delphi variants or innovations to Delphi techniques [60–62]. It is possible that these sources are perceived as less relevant in the health sciences because the thematic focus of Delphi studies is often more on practical issues and not on abstract future developments. At the same time, this poses the risk that new findings, e.g., regarding the length and wording of the questions in Delphi studies [61,63], or critical procedures in the implementation of Delphi techniques, e.g., an unreflected choice of the cut-off value to determine consensus or the lack of analysis of dissent [64,65], are not taken into account and existing methodological practices are continued without reflection and are not further developed.

Clusters of cited references and authors in Delphi primary studies in the health sciences

We identify specific reference clusters and author clusters and assign them to the social-behavioral science or medical-scientific disciplines. They are characterized by different thematic focuses:

• • As has been shown in other bibliometric analyses [9], Delphi techniques are reflected in medical and health science publications as an instrument for finding a consensus for specific questions, i.e. it is a question of application. The social-behavioral science publications focus on epistemology and the further development of the Delphi technique.
• • Indications of different methodological practices between the medical-scientific and social-behavioral disciplines, e.g. that certain Delphi variants are more likely to be cited in one discipline than in the other, were not made clear by the reference clusters. In a contribution to a discussion paper (see Homberg et al. [34]), Cuhls assumes in a comparison of the Delphi technique in the health sciences and in future research that subject-specific differences or different questions are more decisive for the design of the Delphi study than the discipline [34]. A co-citation analysis of the cited references and authors of Delphi studies in the health sciences and futurology could investigate this further.

The results of this bibliometric analysis of publications with Delphi techniques from the health sciences indicate that methodological literature from other disciplines and also Delphi-specific methodological analyses are rarely cited. We hope that this disclosure will promote greater awareness of interdisciplinary and multidisciplinary exchange and encourage Delphi users to consider the relevant methodological literature from different disciplines more comprehensively. A broader discussion of this literature could help to reduce the methodological shortcomings and points of criticism in the implementation and reporting of Delphi techniques documented in numerous reviews (see [16,34,48,55]).

Limitations and critical reflection of the findings 

We were guided by recommendations for conducting bibliometric analyses and by previously published bibliometric analyses on Delphi techniques and other publications with similar questions, such as mixed methods [6,24,26]. We would like to critically reflect the findings and limitations of this bibliometric analysis.

• • Clustering: The number and composition of the reference clusters and author clusters depend largely on the settings in VOSviewer and the underlying statistical models. We chose an exploratory approach and examined changes in different settings to meaningfully define the cluster number and size [26,36]. Other settings could have led to different results and interpretations.
• • Interpretation of the clusters: The interpretation of the clusters is based on the most frequently cited publications – other, less frequently cited articles are not included. Older, frequently cited publications dominate, while more recent publications are (still) underrepresented [28]. Our interpretation is also subject to potential bias, as it is shaped by our own expertise and perspective.
• • Challenges in the author research: Internet research on authors was sometimes difficult, as not all information was up-to-date or the expertise/discipline could not be clearly assigned. Duplicate publications on platforms such as ResearchGate also made it difficult to count the number of publications per author. A supplementary survey of authors could help to record the professional positioning and expertise more precisely and to validate the clusters [66].
• • Limitations of the data basis: We only considered English-language studies. The order of the authors was not taken into account, and actual interdisciplinary collaboration may not be sufficiently visible in the publications if, for example, it is not represented by mutual citation.
• • Conclusion on research practice: The co-citation analysis does not allow any direct conclusions to be drawn about the actual implementation of Delphi studies. Even if publications on Delphi techniques cite the same references, the procedures may differ. In addition, the documentation of the methodology also depends on the requirements of the publishers and the application of certain reporting guidelines.

Conclusions

This research article presents a bibliometric analysis of 1,618 primary health science publications using Delphi techniques between 2017 and 2023. We examined citation practices and intellectual structures to explore differences between medical-scientific and social-behavioral disciplines. Through the co-citation analysis of the cited references and authors, we were able to identify different topics, but no different methodological practices. The methodological practices may have become institutionalized through long-term use within the professional community, resulting in a collective thought style that makes it more difficult to establish innovations. Considering methodological literature from other disciplines and Delphi-specific analysis could help to reduce methodological shortcomings and points of criticism in the application and reporting of Delphi techniques.

Funding

This study is part of the project “Delphi techniques to find consensus in the health sciences. A methodological reflection” (acronym: DEFLEX, project number: 516628412), funded by the German Research Foundation (DFG).

Acknowledgments

We would like to thank Rebecca Abele (RA) and Lotta Jahl (LJ), research associates in the Department of Research Methods in Health Promotion and Prevention, for their support in the screening process.

Author Contribution

JS and MN were involved in designing the study. All authors contributed to the analysis and interpretation of the data, drafting of the manuscript, and reviewing, editing and approval of the final paper. The datasets analyzed during this study are available from the corresponding author on reasonable request.

Competing interests

The authors declare no competing interests. The free version of ChatGPT (GPT-4, OpenAI) was used to support the creation of this manuscript. ChatGPT was used for linguistic precision and to assist in the interpretation and discussion of the findings. All responses from ChatGPT were critically reviewed by the team of authors. The authors bear full responsibility for the entire content of the manuscript.

Consent for publication

The authors have read the manuscript and consented for publication.

Reference

1. Cuhls K. The Delphi Method: An Introduction. In: Niederberger M, Renn O, editors. Delphi Methods In The Social and Health Sciences. Wiesbaden: Springer Fachmedien Wiesbaden (2023): 3–27.
2. Turoff M, Linstone HA, editors. The Delphi Method: Techniques and Applications. Boston: Addison-Wesley (2002).
3. Dalkey N, Helmer O. An Experimental Application of the DELPHI Method to the Use of Experts. Management Science 9 (1963): 458-467.
4. Fleck L, editor. Genesis and development of a scientific fact. Chicago: University of Chicago Press (1979).
5. Hurrelmann K, Ulrich L, Razum O. Entwicklung und Perspektiven der Gesundheitswissenschaften in Deutschland. In: Hurrelmann K, Razum O, editors. Handbuch Gesundheitswissenschaften. 6th ed. Weinheim: Beltz Juventa (2016).
6. Novello N. Communities of Scholars and Mixed Methods Research: Relationships Among Fields and Researchers. Forum Qualitative Sozialforschung 24 (2023).
7. Flostrand A, Pitt L, Bridson S. The Delphi technique in forecasting– A 42-year bibliographic analysis (1975–2017). Technological Forecasting and Social Change 150 (2020).
8. Gupta UG, Clarke RE. Theory and applications of the Delphi technique: A bibliography (1975–1994). Technological Forecasting and Social Change 53 (1996):185-211.
9. Khodyakov D, Grant S, Kroger J, et al. Disciplinary trends in the use of the Delphi method: A bibliometric analysis. PLoS ONE 18 (2023): e0289009.
10. Hsu C-C, Sandford BA. The Delphi Technique: Making Sense of Consensus. Practical Assessment, Research, and Evaluation Practical Assessment, Research, and Evaluation 12 (2007).
11. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 311 (1995): 376-380.
12. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. Journal of Advanced Nursing 32 (2000):1008-1015.
13. Diamond IR, Grant RC, Feldman BM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology 67 (2014): 401-409.
14. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS ONE 6 (2011).
15. Dayé C. How to train your oracle: The Delphi method and its turbulent youth in operations research and the policy sciences. Social Studies of Science 48 (2018): 846-868.
16. Shang Z. Use of Delphi in health sciences research: A narrative review. Medicine (Baltimore) 102 (2023):1-7.
17. Barrington H, Bridget Y, Williamson PR. Patient participation in Delphi surveys to develop core outcome sets: systematic review. BMJ Open 11 (2021).
18. Niederberger M, Sonnberger M. The participation of lifeworld experts in Delphi processes: A reflection on method and practice. MethodsX (2025):103274.
19. Jolly A, Caulfield LS, Sojka B, Iafrati S, Rees J, Massie R. Café Delphi: Hybridising ‘World Café’ and ‘Delphi Techniques’ for successful remote academic collaboration. Social Sciences & Humanities Open 3 (2021):100095.
20. Vikram Koundinya, Yulia Lamoureaux, Julia Van Soelen Kim. Using Focus Groups in Delphi Method to Conduct Participatory Research: Implications for Extension. The Journal of Extension 62 (2024).
21. Niederberger M, Renn O. The Group Delphi Process in the Social and Health Sciences. In: Niederberger M, Renn O, editors. Delphi Methods In The Social and Health Sciences. Wiesbaden: Springer Fachmedien Wiesbaden (2023): 75–91.
22. Niederberger M, Deckert S. Das Delphi-Verfahren: Methodik, Varianten und Anwendungsbeispiele. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen 174 (2022):11-19.
23. Sabisch K. Die Denkstilanalyse nach Ludwik Fleck als Methode der qualitativen Sozialforschung – Theorie und Anwendung [Thought Style Analysis According to Ludwik Fleck as a Method of Qualitative Social Research: Theory and Application]. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research 18 (2017).
24. Dayé C. A Movement Matured: Results of a Co-citation Analysis, and Some Reflections on the Relations Between Social Structure and Ideas in Futures Studies. World Futures Review 16 (2024): 181-198.
25. Münch C, Gracht HA von der. A bibliometric review of scientific theory in futures and foresight: A commentary on Fergnani and Chermack 2021. Futures & Foresight Science 3 (2021).
26. Öztürk O, Kocaman R, Kanbach DK. How to design bibliometric research: an overview and a framework proposal. Review of Managerial Science 18 (2024): 3333-3361.
27. Block JH, Fisch C. Eight tips and questions for your bibliographic study in business and management research. Management Review Quarterly 70 (2020): 307-312.
28. Donthu N, Kumar S, Mukherjee D, Pandey N, Lim WM. How to conduct a bibliometric analysis: An overview and guidelines. Journal of Business Research 133 (2021): 285-296.
29. Birkle C, Pendlebury DA, Schnell J, Adams J. Web of Science as a data source for research on scientific and scholarly activity. Quantitative Science Studies 1 (2020):363-376.
30. van Eck NJ, Waltman L. VOSviewer Manual. 6.20th ed (2023).
31. Schneijderberg C, Steinhardt I. Bibliometric Review – eine Anleitung zur systematischen Erschließung von Forschungsthemen. Kassel (2020). Sozialwissenschaftliche Methodenberatung.
32. Calleo Y, Pilla F. Delphi-based future scenarios: A bibliometric analysis of climate change case studies. Futures 149 (2023):103143.
33. Schifano J, Niederberger M. How Delphi studies in the health sciences find consensus: a scoping review. Systematic Reviews 14 (2025):14.
34. Homberg A, Hirt J, Niederberger M, Häder M, Vollmar HC, Cuhls K, Reiber K, Mohr J. Kann Delphi nur Zukunftsforschung? – Ein Blick in die Gesundheitswissenschaften. Zeitschrift für Zukunftsforschung (2024):1-38.
35. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Systematic Reviews 5 (2016):210.
36. Steinhardt I, Schneijderberg C, Götze N, Baumann J, Krücken G. Mapping the quality assurance of teaching and learning in higher education: the emergence of a specialty? Higher Education 74 (2017):221-237.
37. Niederberger M, Schifano J, Deckert S, Hirt J, Homberg A, Köberich S, Kuhn R, Rommel A, Sonnberger M. Delphi studies in social and health sciences-Recommendations for an interdisciplinary standardized reporting (DELPHISTAR). Results of a Delphi study. PLoS ONE 19 (2024): e0304651.
38. Niederberger M, Spranger J. Delphi technique in health sciences: A Map. Frontiers in Public Health 8 (2020).
39. Dalkey NC. The Delphi Method: An Experimental Study of Group Opinion. Santa Monica, CA: RAND Corporation (1969).
40. Keeney S, Hasson F, McKenna H. The Delphi technique in nursing and health research. Chichester, West Sussex: Wiley-Blackwell (2011).
41. Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Information & Management 42 (2004):15-29.
42. Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med 8 (2011):1-5.
43. Murphy MK, Black NA, Lamping DL, et al. Consensus development methods, and their use in clinical guideline development. Health technology assessment 2 (1998): i–88.
44. Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica (2001).
45. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. American Journal of Public Health 74 (1984): 979-983.
46. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, Tugwell P. Developing core outcome sets for clinical trials: issues to consider. Trials 13 (2012):132.
47. Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics 42 (2009):377-381.
48. von der Gracht HA. Consensus measurement in Delphi studies. Technological Forecasting and Social Change 79 (2012): 1525-1536.
49. Holey EA, Feeley JL, Dixon J, et al. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Medical Research Methodology 7 (2007): 52.
50. Trevelyan EG, Robinson N. Delphi methodology in health research: how to do it? European Journal of Integrative Medicine 7 (2015): 423-428.
51. Villiers MR de, Villiers PJT de, Kent AP. The Delphi technique in health sciences education research. Medical Teacher 27 (2005):639-643.
52. Humphrey-Murto S, Varpio L, Wood TJ, et al. The use of the Delphi and other consensus group methods in medical education research: a review. Academic Medicine 92 (2017): 1491-1498.
53. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliative Medicine 31 (2017):684-706.
54. Jorm AF. Using the Delphi expert consensus method in mental health research. The Australian and New Zealand Journal of Psychiatry 49 (2015):887-897.
55. Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology 11 (2021):116-129.
56. Eubank BH, Mohtadi NG, Lafave MR, et al. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Medical Research Methodology 16 (2016):56.
57. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 3 (2006): 77-101.
58. Merton RK. The Matthew Effect in Science. Science 159 (1968):56-63.
59. Niederberger M, Homberg A. Argument-based QUalitative Analysis strategy (AQUA) for analyzing free-text responses in health sciences Delphi studies. MethodsX 10 (2023).
60. Winkler J, Moser R. Biases in future-oriented Delphi studies: A cognitive perspective. Technological Forecasting and Social Change 105 (2016): 63-76.
61. Andersen PD. Constructing Delphi statements for technology foresight. Futures & Foresight Science 5 (2022).
62. Quirke FA, Battin MR, Bernard C, et al. Multi-Round versus Real-Time Delphi survey approach for achieving consensus in the COHESION core outcome set: a randomised trial. Trials 24 (2023):1-15. PMID:37468987
63. Gargon E, Crew R, Burnside G, Williamson PR. Higher number of items associated with significantly lower response rates in COS Delphi surveys. Journal of Clinical Epidemiology 108 (2019):110-120. PMID:30557677
64. Webbe J, Allin B, Knight M, Modi N, Gale C. How to reach agreement: the impact of different analytical approaches to Delphi process results in core outcomes set development. Trials 24 (2023):345. PMID:37217933
65. Beiderbeck D, Frevel N, Gracht HA von der, Schmidt SL, Schweitzer VM. Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements. MethodsX 8 (2021).
66. Kvale S. The Social Construction of Validity. Qualitative Inquiry 1 (1995): 19-40.