How the COVID-19 pandemic changed stakeholder engagement processes in sustainability research in the long-term

Interviews

From April to October 2022, we conducted 15 interviews with 13 researchers (“R”) and 2 stakeholders (“S”) to understand how they have experienced the impact of the pandemic on the research process and outcomes, and how they would decide about future online and offline engagement. We interviewed different sustainability research projects: three on agriculture and food, two on energy, two on climate, two on marine management, one on water management, and one on material risk management. Nine projects were funded by the EU Horizon 2020 research programme, whereas the remaining two projects had other national funders, such as ministries and foundations. The authors of this article have not been involved in any of these projects. We selected the projects from which we recruited our interviewees based on three criteria: (i) a topical diversity of projects covering different sustainability topics; (ii) stakeholder engagement was an integral part of the projects; (iii) projects starting no later than January 2020 and running at least until the end of 2020. Projects that fulfil these criteria have been selected from the CORDIS data base and been contacted with an interview request. In addition, we screened for projects among the participants of two thematically related surveys in 2020 (Köpsel et al. 2021; Süsser et al. 2021)where participants indicated a high importance of stakeholder engagement and provided us with the contact details; researchers and/or coordinators from four projects agreed to be interviewed.

Survey

In autumn 2022, we conducted an online survey to explore the longer-term impacts of the pandemic on research projects and perspectives for future projects. We contacted the same projects from energy research and marine sciences that participated in our surveys in 2020 and invited other sustainability researchers to complete the survey via existing networks (incl. social media) and other project collaborations. We received responses from 26 different projects with responses from up to four researchers from a single project. This is a much lower response in comparison to the survey we conducted in 2020. Similarly, more than twice as many respondents as in 2020 started the survey but did not finish it; this can have different reasons such as that it was too long, or respondents faced technical issues. In our analysis we included only 30 completed responses (“survey_ID#”) for qualitative quotes (Figure 1). The responses came from 26 projects representing different fields; half of them from energy research, sixty-eight percent of the respondents were Horizon 2020 projects. Most projects (14) started in 2019, and end(ed) in 2023 (16 projects). The survey’s respondents had a diverse work experience in their field and various levels of experience regarding stakeholder engagement; many (about 40%) had 1-5 years of experience. Two-thirds of the respondents were men, a bit less than a third women and some preferred not to indicate their sex.

Figure 1. Research fields of participating researchers in the survey, n = 30.

Stakeholder engagement in sustainability research

The surveyed researchers involved a range of stakeholders in their projects, as shown in Figure 2. Three-fourth of the researchers involved policymakers and governmental authorities from local to European level, and two thirds engaged business, trade, and industry, as well as civil society organisations. In the survey, researchers also named “other” stakeholders, including small- and medium-size enterprises (SMEs), institutions in the financial and insurance sections, self-employed fisheries, as well as “Friday for Future” activists.

Figure 2. Number of researchers that engaged specific stakeholder groups: Which stakeholder groups are/were engaged in your project?; multiple choices possible, n = 30.

Almost half of the respondents (45%) assessed stakeholder engagement to be crucial for the success of the project; another third saw it as quite important. The importance slightly declined from 2020 to 2022. Researchers had different motives for involving stakeholders in the research, ranging from a pure interest in disseminating research results to even assisting stakeholders in realising their own projects (Figure 3). One third of the researchers took a co-production of knowledge, or capacity building approach, aiming to, either develop the research process together, what is important in pursuing transdisciplinarity in sustainability science (cf. Mielke et al. 2016) or to directly create a change ‘on the ground’. Researchers also indicated that they involve stakeholders with the specific purpose of generating policy impact, e.g., to co-design policies, sharing learnings for policymaking, or supporting decisions. Compared to 2020, there is no notable difference in answers, except that access/collection of data was more important in the 2022 survey (50% of the projects in 2020, 70% in 2022).

Figure 3. Motives for stakeholder engagement: What is/was the purpose of engaging stakeholders at certain points in the research process?; multiple choices possible, n = 30.

The importance of stakeholder engagement for data collection purposes was elaborated by interviewees that valued the insights and perspectives from stakeholders that are often not reflected in quantitative data. One interviewee said that the “system cannot be described properly only with statistical data - also in the fishing industry” (R#5). Furthermore, a recurring theme in the interviews was to involve citizens and communities to increase their “buy-in”, or to find takers for outputs generated in the projects. The two interviewed stakeholders were motivated to participate in the affiliated research projects, because they thematically identified themselves with the topic and they wanted to share their expertise.

Impacts of COVID-19 on sustainability research

Overall, the pandemic had a negative impact on stakeholder engagement: more than half of the surveyed researchers saw impacts as mainly negative, in both periods but with considerably more negative responses in 2020 than in 2022 (Figure 4). Main reasons were that stakeholders could not be reached during the pandemic and that engagement was not as deep as with in-person interactions. For example, one survey participant responded (free text):

Figure 4. Impact of COVID-19 pandemic on stakeholder engagement: How did the COVID-19 pandemic influence your stakeholder engagement in the project overall?; n = 30 in 2022; n = 84 in 2020.

However, some projects also experienced positive impacts, or were neutrally affected from the pandemic (Figure 5). These were mainly projects that had already planned for online activities anyway, or that just started to plan their engagement activities during the pandemic:

Figure 5. Planned stakeholder engagement activities: According to the original project plan/proposal, which kind of stakeholder engagement activities did you plan to perform in 2020/2021?, n = 30, multiple choices possible.

More than one third of the surveyed researchers had planned for online workshops and almost the same numbers for online interviews and surveys. However, most survey respondents and interviewees had planned to engage with stakeholders physically in 2020/2021 ̶ when COVID-19 lockdowns were most stringent (Figure 5).

One third of the projects experienced both positive and negative impacts (Figure 6). One survey respondent stated:

Figure 6. Impact of COVID-19 pandemic on the relationship with stakeholders since the start of the COVID-19 pandemic; n = 30 in 2022; n = 83 in 2020.

This quote highlights that positive impacts were related to the scope of the stakeholders that could be reached, whereas it was not possible to deepen the engagement level.

Process

The pandemic negatively impacted the engagement of stakeholders in research processes: More than half of the survey respondents agreed that it was harder to familiarise stakeholders with the project goals/objectives and one third agreed that stakeholders had less time, or their priority had shifted away (Table 1). There was more agreement that exchange with stakeholders had become less frequent, although the received responses about online meetings varied considerably.

Based on our qualitative interview analysis, we found that three dimensions were mainly affected: relationship, coping and preparation.

Relationship

Researchers reported that it was a challenge to establish relationships with stakeholders. Identifying stakeholders and gaining their trust was made more difficult in online engagement:

Some projects wanted to form working or core groups of stakeholders, but many researchers struggled to achieve that.

Our research confirms the first impressions after the COVID-19 pandemic started (Süsser et al. 2021) that many researchers feel a strongly negative, or somewhat negative, change in their relationship with the stakeholders since the start of the pandemic (Figure 6). The opinion about the change diverged even more, and less researchers experience no change.

Some researchers reported that it was easier to keep contacts online, also in between in-person meetings, and to exchange materials. After difficulties in the first meetings, stakeholders got used to online events:

For almost two thirds of respondents it was harder to engage stakeholders online, and for them, stakeholders were less committed to online engagement formats (Figure 7). Often, participation in online events was reported as less active, i.e., many people had their cameras off, were in the listening-only mode, or were doing other things in the background. A survey participant confirmed:

Figure 7. Respondents’ agreement with the following statements about online engagement; n = 30.

A researcher also admitted that “[it is] too easy to open another window, to open emails. I do it myself” (R#1). Thus, it was also easier for people to “hide” online, while in in-person meetings the “facilitator can rope them in better” (R#11).

At the same time, anonymity made it easier for people to leave during online events: breaks or shifts to breakout sessions were moments when researchers lost many attendees (R#15). One interviewee confirmed this:

We also find that the participation in online events declined over time (Figure 7). While people “were more eager to meet online in the beginning, as the pandemic continued, it became more difficult” (R#9). Interviewees reported that attendance fluctuated, and it was hard to keep contact: “In general, in Corona there is a greater risk of contact being lost somewhere” (S#13).

Finally, the relationships to stakeholders were negatively affected because online engagement does not provide the networking activities, informal interactions, and the potential for establishing more personal ties in equal measure. One interviewee underlined that “in-person gives connection and sympathy and understanding which is hard to create online. Humanness is critical” (R#2). However, it was not only harder to build relationships with stakeholders but also within newly formed research consortia. One researcher explained that they received often late or no responses from project partners and that “[i]f we would have been in the room together once or twice, four times, five times during the project, this just comes way more naturally” (R#1).

Coping

In 2020, we found that one in six engagement activities was implemented as planned (18%) (Süsser et al. 2021) ̶ in 2022, it was one in four (Table 2). After the pandemic started, a dominating strategy was to delay the activities (45% in 2020), hoping that the situation would improve quickly. One interviewee stated that task leaders asked them to wait and delay meetings during the crisis and to meet in-person later in the year; but they did not do any in-person meeting as the pandemic continued (R#12). Similarly, one respondent emphasised the importance of meeting stakeholders at congresses; however, most congresses were cancelled in 2020 and most were even moved to 2022 (R#8). In 2022, substantially less researchers delayed activities in comparison to 2020. As one stakeholder explained: “because it was not physically possible, and we were not yet technically ready. […] In 2020, a lot of things were cancelled and in 2021, everything was more routine. We always had plan B“ (S#13).

Likewise, 2020 and 2022, a minority of some (10) projects implemented the engagement activities with social distancing measures (Table 2). For example, researchers moved from larger to smaller stakeholder event sizes, or even met stakeholders outside:

Also in 2022, many researchers moved activities online (Table 3). Among the coping engagement formats, online workshops and online interviews were the most common-confirming our findings from the 2020 survey. In-person engagement formats were usually directly replaced by a corresponding online format. For example, planned in person workshops were held online on various video conference platforms, often complemented by interactive tools, such as live polling or whiteboards. Additionally, interviewees reported about online capacity-building events and regional demonstration events:

Other projects changed the engagement activity (Table 3). Webinars were less prominently chosen than in the 2020 survey, but were the main choice to replace information events, or congress visits. One interviewee explained that they had to adjust their plan:

Overall, video conferencing tools became much more prominent in engaging stakeholders (Table 4), but their applications, especially at the beginning, made some stakeholders feel digitally isolated:

Finally, researchers also had to adapt the event length. Interviewees commonly agreed that long events do not work. One researcher explained that they tried a two-day online event, or long sessions with a break in between – as would have worked in person – but „doing an event for three or four hours wasn’t possible anymore” (R#15).

Preparation

Most respondents agreed that online engagement saves time and resources (Figure 6), also by avoiding travelling:

Simultaneously, preparing for online engagement may require more effort:

One stakeholder highlighted the capacities needed to implement online event successfully:

In addition, conducting individual interviews was reported to be more time-consuming than the implementation of one workshop, also as “you need to repeat several times to generate the same impacts as if you would meet the people in person” (R#8). Further, it takes more time to explain complex topics online. One interviewee shared the experience that it was difficult to present complex approaches behind mathematical models online; it did not work out because they were not able to explain it, stakeholders could not engage in breakout groups (R#15).

Outcomes

The pandemic also had impacts on the outcomes of the research as well as on the outcomes of the stakeholder engagement specifically.

Outcomes of research

Around half of the researchers reported that the outcomes are (somewhat) less good (Figure 8), while less than one third of the respondents stated that the project was carried out with the envisioned results (Figure 9). The reasons given were mainly related to the depth of the generated results:

Figure 8. Impact of the pandemic on the outcomes of the project and stakeholder engagement, n = 25.

Figure 9. Impact of changes in stakeholder engagement on the overall proceedings and results of projects; multiple choices possible; n = 30 in 2022; n = 83 in 2020.

One interviewee reported that they were able to “tick the boxes” of the Grant Agreement, but not in effective ways.

The impacts of the pandemic caused projects’ extensions that became even more evident in the 2022 survey (Figure 9), with two thirds of the projects extended.

No negative impacts and even better outcomes were reported by minority of respondents, one of them mentioning about different opportunities:

Outcomes of stakeholder engagement

Most respondents agreed that online activities are less satisfying (Figure 7). Online engagement often misses ‘lively’ interactions, as one interviewee explained:

As a result, some researchers emphasised the lower quality of data derived from online engagement:

In contrast, other researchers reported better outcomes from the stakeholder engagement, which were mainly related to an increased participation and the collection of more diverse stakeholder insights. Thus, researchers could gain important insights, albeit “less representative” results (survey_ID35).

Furthermore, deliverable submissions were delayed – in 2022, 70% of the survey participants reported pandemic-related delays in comparison to 30% in 2020 (Figure 8). For example, due to delays in results from stakeholder input, the results could not or only partially be considered in the project context:

When to stay offline, when to go online?

In the interviews, we identified three key influence factors deciding whether to stay offline or go online: (i) intention/ purpose of the stakeholder engagement, (ii) the engaged stakeholder group, and (iii) previous relationships. As one interviewee put it: “a different target group, different topics, and different intention of the project/co-creation would need a different format, but it is hard to know” (R#2).

Online vs. offline depends on the objective of the stakeholder engagement

First, the online vs. offline choice depends on the objective of engagement: “What is the aim? What do you want to get out of stakeholder engagement?” (R#9). In case researchers need to gain expertise fast, purely want to report results, or get feedback, online engagement was often sufficient. In contrast, deeper exchange needs in-person interaction:

Offline experiences are of specific relevance if you want to get impressions “from the ground”:

In addition, one researcher shared that you may miss “off-the-record information”, shared in a more informal setting, which would otherwise help in interpreting the interviews:

Thus, offline engagement seems to be essential if a deeper understanding of people’s contexts is crucial.

Online vs. offline depends on the engaged stakeholder group(s)

Second, the online vs. offline choice depends on the stakeholder group to be involved. Around 40% of the survey’s respondents perceive differences in the responsiveness between stakeholder groups (Figure 6). Researchers noted that some stakeholders could be better reached online, especially for engagement with policymakers or with industry actors. Interviewees explained that it is “sometimes […] easier to reach people that are very busy, because they can just take a short time slot in their agenda” (R#8), or “[i]ndustry appreciated online because they didn’t have to travel” (R#9). One of the stakeholders illustrated a difference between different actors on a different example:

In contrast, other researchers reported less engagement and interest. There was a common agreement that “citizens are hard to engage only online” (survey_ID37). Nevertheless, one survey respondent shared an insight that incentives and the mediator can motivate citizens to participate in online survey:

One fourth of the survey respondents stated that they could not reach some stakeholders online, which can lead to a selection bias. These stakeholders included mainly actors at a local level, such as “local communities with limited internet access and technological skills” (survey_ID40), or “residents who do not have internet or barely use internet” (survey_ID61). One survey participant mentioned that online engagement can exclude certain people:

Differences in possibility of reaching out to different stakeholders have also a geographical or cultural dimension, as one interviewee said:

We found that technical abilities were a barrier, especially in the beginning of the pandemic, as everyone had to learn the new tools. Thus, interviewees also raised the concern that the online community is a subset consisting of people who like using online tools (R#11).

For certain stakeholders it seems to be crucial to stay offline, as they expect to meet researchers in their “environment”, what can bring benefits to the researchers, too:

Online vs. offline depends on the previous relationship to stakeholders

Third, the online vs. offline choice depends on the strength of ties already established with the stakeholders and project partners. One interviewee underlined:

Our findings indicate that an in-person kicking off event where participants get to know each other and then having online interactions boosts the interpersonal collaboration and trust. The same applied to researcher-stakeholder interactions and projects’ consortia:

We also found that online engagement is often much more disciplined and formal. One interviewee stated that “protected spaces” are missing:

Thus, we find the need to stay offline, if an open exchange is wanted, potentially also in smaller groups.

Future perspectives for stakeholder engagement

The experiences with online engagement formats are mixed, but by far most researchers will continue online stakeholder engagement: 92% survey respondents state they will continue doing online stakeholder engagement. In most cases, the main purpose of such future online engagement is to collect data, disseminate research results, verify results and identify research needs (Figure 10). More intensive stakeholder engagement, in comparison, is much less prominent to be performed online in the future, again indicating that deeper stakeholder participation requires in-person engagement.

Figure 10. Motives to engage stakeholders online: For what purpose will you continue involving stakeholders online?; multiple choices possible; n = 24.

No survey respondent plans to strike for fully online, or fully in-person events in the future (Figure 11). Instead, the majority plans a mix of online and in-person formats:

Figure 11. What balance between online and in-person engagement do you plan to strike in the future?; n = 24.

How researchers want to engage stakeholders

We find that most researchers want to continue with online engagement alongside physical engagement. As one interviewee put it: “Only online is no good, lots of travel is also not good“ (R#9). Researchers value personal contacts and interactions.

For the internal project management, researchers want a reduced number of in-person meetings ̶ two in person, otherwise online. One researcher reported that, while planning the final project event in person, stakeholders, and even consortium members wanted to remain online and were not interested in travelling for the final project event (R#1).

Researchers made different experiences with online engagement formats, with interviews, surveys and workshops scoring highest as the most suitable online formats and the ones most likely to remain in the future (Table 5).

Hybrid events were for many a completely new format and researchers made mixed experiences. A survey respondent explained:

Interviewees recognised the time effort behind hybrid events, but also confirmed the advantages for inclusivity:

How stakeholders want to be engaged

Stakeholders want to participate in online formats as complementation but not as full replacement. Stakeholders prefer a balance of online and physical meetings and believe this format will continue in the future:

If a physical engagement is to be organised, stakeholders prefer shorter and more time-efficient events. One researcher shared their experience on their first in-person event after pandemic:

Stakeholders do not see the need to always meet in-person. Interviewees shared that some associations and authorities asked if meetings could be done online (R#5), or that it was hard to get policymakers attend physical events even if the event was designed to fit transport connections and minimising travel time (R#15).

When it comes to specific online tools, stakeholders seem to generally prefer Zoom (R#7). A plethora of online tools were scrutinised during the pandemic, such as virtual whiteboards and online polling, and some stakeholders very much enjoyed new online tools:

Interviewees generally assessed miro and mural boards to be difficult tools for stakeholder interactions and instead, a google document might be a better way to capture feedback from people.

When it comes to the frequency of interactions, stakeholders generally want to stay informed about the timelines and “get a feeling of what further steps are planned” (S#13). Furthermore, they want to be informed about the final results and, specifically about how their inputs added value to the project. One researcher said:

Physical events with hybrid online options are generally fine for stakeholders but mean an additional technological and coordination challenge. Another stakeholder raised issues like:

Ethical considerations

The research was conducted in accordance with the ethical requirements and guidelines of the SENTINEL project (Deliverables 11.1 and 11.2), which follows the guidelines of the European Commission. We used an ethically sound methodology for data collection and processing in this study, under the guidance of the Institute for Advanced Sustainability Studies (IASS), now the Research Institute for Sustainability – Helmholtz Centre Potsdam (RIFS), Data Protection Service. This was supported by bilateral data protection agreements.

Consent to participate

Interviewees agreed to our data protection standards through a signed consent form, and survey respondents agreed to our data protection standards through a GDPR disclaimer and by participating in the survey.

Consent to publish

The results from the survey and the interviews have been de-identified in accordance with the Safe Harbor methods, ensuring that individuals cannot be identified.