The Delphi method is a structured forecasting/decision-making tool that has been successfully applied to health research in many disciplines ( Balabanova et al., 2011; Booth et al., 2011). The Delphi method creates conditions that are favorable to a convergence of opinions, while at the same time allowing moderators to clearly discern points of dissent. It usually takes the form of a written questionnaire and allows for anonymous and independent consultation and argumentation, thereby avoiding some of the drawbacks of face-to-face confrontations on both the social level (e.g. power relations within a group) and the practical level (time consuming, especially with geographically dispersed individuals, which is also applicable to our case) ( Duin, 2016). Responses are only visible to the moderator(s) and not to the participants in order to avoid self-moderation bias. The iterative nature of the consultation builds on the feedback of the respondents. This allows for the correction of potential bias in the initial questions which is the main fragility of classical (non-iterative and non-interactive) queries. A potential problem with the Delphi method is the creation of a bottleneck towards convergence of opinions during the process ( Linstone & Turoff, 2002; van der Gracht, 2012). It is therefore important to use both open and closed questions, and to take into consideration the whole spectrum of opinions. The online platform used in our study provides additional advantages such as low respondent drop-out rates (reducing effects of self-selection) and the possibility for respondents to revisit, complement or modify their answers during both rounds (generating more data) ( Francois et al., 2011).

Expert users of Philips ISP - CT TAVI and Spectral CT were identified via referral by Philips clinical workflow specialists and selected to achieve a balanced representation of these expert users from different regions within the US. Experts were recruited via email. All experts declared all conflicts of interest before participating in the panel. Throughout the Delphi, respondents were encouraged to explain their choice for closed-ended and multiple-choice questions. All questions are available as part of the Underlying data ( Wiegand, 2020a).

The Delphi process was conducted online using the MESYDEL platform ( Mesydel, 2019) developed by the Spiral Research Center at the University of Liège (Belgium); however, it could readily be performed manually. Two rounds of Delphi queries were launched and analyzed between July 22 and October 14, 2019. The project organizers designed questions on the survey. A duration of six weeks for Round 1 and five weeks for Round 2 were given to panelists to respond to the surveys. Reminder emails were circulated for each iteration.

The second round mainly focused on ambiguities or disagreement in the first round, clarifying areas of disagreement, and fostering more consensus on approaching the issue differently according to the participants’ input. This was achieved by providing the survey responses from Round 1 to the panelist during Round 2 in which clarification or consensus was sought.

Quantitative data for closed questions was tabulated using MESYDEL, an online Delphi panel platform, and Microsoft Excel 365. The responses to the first round of questioning informed the design of the second round of questions, thus allowing for questions to be modified as needed. Particular attention was paid to areas of disagreement. Additionally, questions were reviewed with the survey group based on key observations from round 1 of the survey. All data was collected prospectively and analyzed using Microsoft Excel to obtain descriptive and summary statistics.

The Delphi panel consisted solely of U.S.-based experts. A response rate of 73% was achieved with 11 radiologists responding to the Spectral CT portion of the survey and 80% for the CT TAVI portion of the survey with 8 physicians responding. Of the Spectral CT survey participants, 91% had over 10 years of experience and 55% had over 20 years of experience ( Table 1). Of the participants in the TAVI survey, 87% had over 10 years of experience, and 50% had over 20 years of experience ( Table 2).

The results described here were derived using the Delphi Panel approach which is based on expert opinion. No patients have been used for this study and no patient data is collected and no patient samples or results have been used. No individual can be identified from the results presented. No animals have been used in this study. All participants provided written informed consent to participate.

The sentiment toward the Spectral CT tool within the ISP was very positive, with 90% of the respondents agreeing (hereafter, “agreeing” refers to “somewhat agree” or “completely agree” responses to the Likert-style questions) that it clinically benefits their everyday work ( Figure 1). The respondents qualified their answers with comments and three respondents noted the virtual non-contrast (VNC) test feature as the most relevant feature. In addition, three respondents commented that Spectral CT improves their diagnostic confidence ( Underlying data, Question 4; Wiegand, 2020a). In a question to assess whether the Spectral CT tool has led to more confident decision-making, 100% of respondents agreed with this statement. One respondent stated that the confidence in decision-making is, “ …a result of additional precious information in [the] form of Spectral Detector CT reconstruction, [which is] useful in many situations like evaluating indeterminate cysts and incindentalomas that are routinely encountered.” Other respondents indicated that they valued the VNC feature for use in stroke patients and that they use the additional image types to confirm diagnoses ( Underlying data, Question 12; Wiegand, 2020a).

All participants agreed that the Spectral CT application has led to improved diagnostic capabilities, leading to better treatment paths for patients ( Underlying data, Question 10; Wiegand, 2020a). They state that it helps them with certain types of patients such as stroke patients and those with pulmonary emboli and that the additional imaging lends to more confidence in diagnosing. This confidence is highlighted in an open-ended question, “ What elements of Spectral CT do you feel lead to a better patient experience?” ( Underlying data, Question 36; Wiegand, 2020a), where doctors elaborated on how confidence in diagnosis impacts patients. One doctor wrote: “ Patients may not directly experience this but when a radiologist hedges it significantly changes the arc of their care. On the other hand, when a radiologist is more confident, then the clinician is also more confident putting the patient on anticoagulation.”

In a question to gauge the effect of the Spectral CT application on the work experience ( Underlying data, Question 9; Wiegand, 2020a), 91% of doctors agreed that the application improved their work experience. Two doctors further qualified their statements by reiterating that diagnostic confidence is improved. Another doctor elaborated further and stated that, “ with [Spectral CT] and its reconstructions we are now telling the complete story of the disease rather than just providing the snapshot with conventional CT. For me this leads to a pleasant and better work experience.” In Round 2 in a follow up question to delve deeper into the technologies impact on the work experience, we explored further for those participants that agreed in Round 1. Responses ranged from specific tools provided in the Spectral CT application such as, “ better visualization of enhancement by low keV images and iodine mapping”, and the option to always have a retrospective picture archiving and communication system (PACS), as well as the virtual desktop link to easily collaboratively review and analyze data. They also note the benefit of being able to bookmark and share certain data amongst the clinical team. Another doctor gave a similar comment, but also noted that they believe that the tool may prevent additional follow up exams by allowing them to make a confident diagnosis. Panelists also mention that an improved work experience may help them counter provider burnout ( Underlying data, Question 26; Wiegand, 2020a).

We also asked the panelists questions on the potential of Spectral CT to reduce the number of scans needed for diagnosis, “In Round 1 of the Delphi Panel a participating expert highlighted the following, please state to what extent you agree with the following statement: Spectral CT has helped improve on accuracy and confidence of diagnosis (compared to not having Spectral CT & ISP), leading to less frequent follow-ups and repeat exams.” ( Underlying data, Question 33; Wiegand, 2020a). In total, 27% of respondents somewhat agreed, and 56% completely agreed, while 18% of them completely disagreed with this statement. The experts who did agree feel it resulted in time-savings gave estimates ranging from 15 minutes per scan, 30–60 minutes per day, and 40–50% time reduction. While the rest of the respondents to the open question asking for comment replied that they did not have an accurate sense of how much time was saved.

The CT TAVI application of the ISP was thought to yield time-savings in ‘device sizing’ related tests. A question based on this was asked to the experts and 85% agreed that it did reduce time-savings, while 15% responded “neutral” ( Underlying data, Question 66; Wiegand, 2020a). There was a lack of responses to the open-answer portion of this question, but one piece of critical feedback was that “ while the software may reduce work, the time for analysis is roughly the same”. Another respondent answered that “ CT TAVI produces multiple measurements quickly with a small amount of user input.” In a follow-up question ( Underlying data, Question 67; Wiegand, 2020a) under this topic, we asked for numerical estimates of time-savings. Only five of the respondents responded to this question, but of the four responses which were given in time units “per case”, they estimated that approximately 13.1 minutes are saved per case. One respondent estimated that they saved “ 1 to 2 hours per week”.

We also asked whether CT TAVI decreases one’s cognitive load due to automation in location selection within the tool ( Underlying data, Question 69; Wiegand, 2020a) whereby 100% of the respondents agreed that it did reduce cognitive load. However, in the comments one expert qualified that, “ the software reduces some of the workload, [but] the user still needs to check for errors, which are uncommon with the annular sizing tool, but common with the vascular analysis tool.” A further two experts noted that when the quality of the CT scan is good, annulus points and automated landmarks are accurate.

Questions pertaining to quadruple aim goals were presented to all participants in Round 1 of the survey. Some participants also received follow-up questions where clarification was needed in Round 2. Responses to these questions are summarized in Figure 2 for the Spectral CT portion of the survey and in Figure 3 for the CT TAVI portion of the survey.

We achieved consensus to the question on Spectral CT leading to improved diagnosis capabilities leading to better treatment paths for patients ( Underlying data, Question 36; Wiegand, 2020a). A radiologist commented that confidence on their part leads to confidence on the clinician’s part on choosing therapies for the patient. This is done without hedging and thus removes the need for additional tests to give a diagnosis. Another doctor also noted that the ability to retrospectively perform spectral analysis on every patient is crucial because it removes the need to prospectively determine if a study requires spectral or dual energy analysis. They note, “ the ability to perform the spectral analysis without a significant cost of additional radiation to the patient is also highly beneficial.” A key benefit of Spectral CT is this ability to retrospectively use data to create virtual unenhanced images and enhanced material density maps to glean more from a single scan ( Massat, 2018). Because Spectral CT is a dose-neutral exam compared to conventional CT, the non-contrast phase of some protocols is eliminated and the need for additional exams is reduced ( Massat, 2018). Spectral CT simultaneously uses two different energy settings, which allows differentiation of materials based on material density. Radiologists can use this feature of Spectral CT to acquire virtual unenhanced images as well as iodinated contrast material-enhanced material density images from a single scan ( Silva et al., 2011). These features of Spectral CT improve patient care by minimizing their exposure to radiation dose from scans while also giving radiologists access to additional images to inform a diagnosis.

Diagnostic imaging costs and use have increased dramatically over time. It was reported that CT use increased from 52 per 1000 enrollees in 1996 to 149 per 1000 in 2010 ( Smith-Bindman et al., 2012). Additionally, approximately 20% to 50% of tests fail to provide information that improves clinical care ( Neeman et al., 2012). Amidst burgeoning healthcare spending, it is imperative to investigate new technologies and interventions for reducing excessive tests. A key finding of our survey was that panelists felt that the use of Spectral CT improved their confidence in making a diagnosis. Additionally, they felt that they required fewer tests to make a diagnosis. We found that most doctors did not have an accurate sense of exactly how much time is saved from the implementation of image portals and the use of additional scans from Spectral CT ( Underlying data, Question 33; Wiegand, 2020a). Availability of spectral information could eliminate the need for follow-up scanning in patients with sub-optimal exams and incidental findings, increasing the diagnostic confidence of radiologists and resulting in a potential annual savings of $55,000 or higher depending on the practice and number of exams ( Prabhakar et al., 2018).

Parts of our survey were designed to measure the perceived amount of time-savings from the use of the ISP tool and Spectral CT. However, we found that most of the respondents did not have an accurate sense for how much time is saved. Some radiologists estimated time-savings of 30–60 minutes a day while others estimated 15 minutes per scan was saved. Other respondents noted that elimination of follow up procedures does not shorten their workflow and that more time is needed to process spectral data. These results show that the benefits of Spectral CT would be highly dependent on the exact practice and type of caseload of a radiologist.

Of the eight physicians responding to the TAVI survey, five stated that they believed the tool lead to time savings when used for ‘device sizing’ results. Four of them gave their response in “per case” savings, giving a mean time savings of 13 minutes per case. The last doctor estimated a time savings of one to two hours per week from using the tool. Reducing time needed to conduct tests is crucial in improving radiologist throughput. Throughput of patients is complex and relies on many factors including patient availability, acuity, triage, and other activity ( Levenson et al., 2012). Time-savings when reading any type of test can prove to be beneficial in improving throughput and volume of a practice.

The fourth aim of the IHI pertaining to staff satisfaction or “joy in work” was added because studies have shown that parameters such as physician burnout lead to lower patient satisfaction and even poorer patient outcomes ( Halbesleben & Rathert, 2008). In a systematic review, researchers identified 21 of 30 studies that measured burnout found a significant association between burnout and patient safety, whilst a further four studies also found a connection to patient safety ( Hall et al., 2016). Thus, an important result of our survey was that 91% of respondents agreed that the spectral CT application makes their work experience better. Doctors who gave comment also unanimously agreed that the remote work capabilities of ISP are very convenient and benefits them in sharing data and in continuity of work ( Underlying data, Question 2; Wiegand, 2020a). They also noted that the technology enabled them to assist colleagues from home while providing second opinions for emergent cases through the virtual portal. Even a marginal increase in satisfaction amongst radiologists is significant as it would play some role in preventing burnout and dissatisfaction with work, which could potentially result in improving patient satisfaction, outcomes, and safety.

Figshare: Extended data - Delphi Questions v1.0. https://doi.org/10.6084/m9.figshare.11734353.v1 ( Wiegand, 2020a).

This project contains all questionnaires with all answers given by Delphi panel participants.

Figshare: Extended Data - Delphi Panel ISP. https://doi.org/10.6084/m9.figshare.11778486.v1 ( Wiegand, 2020b).

This project contains the multiple choice questions data used to generate Figure 1– Figure 3.

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).