Shaping the Future of Research: A perspective from early career researchers in Vancouver, Canada

This paper describes the outcomes of a workshop conducted with n=136 participants, mostly postdoctoral fellows and graduate students, in Vancouver, Canada to discuss issues facing trainees. This event was modeled on a previous meeting held in Boston. The authors present a summary of participants’ views regarding challenges faced by trainees and some potential solutions to address these. This manuscript offers a well-written account of issues facing trainees in Canada, and has potential policy implications. However, I have quite a number of concerns about this manuscript, which I detail below. I hope these are helpful to the authors to improve their manuscript and achieve the impact that they hope to have in Canadian science and research policy.

Reason for Not Approved

Note: I moved this point up to the top of my review after reflecting on the likely reaction of the authors and what they might first want to know from this review.

Research with human participants (a.k.a. "human subjects research") requires oversight by an research ethics board to be publishable in a reputable peer-reviewed journal. Was the survey approved by the relevant research ethics board (in this case, SFU, since that’s where the data collection occurred)? Did all survey respondents provide informed consent to have their responses analysed and published? If the answer to either of these questions is no, this is problematic. This lack of information about ethical approval gave me great pause when reviewing this manuscript. Current practice in many other journals for a manuscript describing data from human participants without a statement about ethical approval is to send it back before it even gets to the editor. This is the reason I selected Not Approved. I would prefer not to have to select this, especially as this is a trainee-led manuscript, but this is a required element for human subjects research to merit publication in a peer-reviewed journal. I can't approve this manuscript without this. I'm sorry.
 
Overall
 
1. Having been extensively involved in policy discussions about Canadian health research funding and funding policy, I began this review by reading the manuscript twice—once with my existing base of knowledge, and a second time, attempting to put myself in the place of someone who is less familiar with the past few years of changes and events in Canadian health research policy. My experience as a reader was significantly worse the second time. To enable this manuscript to contribute to the global literature on science/research policy, it would be preferable if the authors were to more carefully explain terminology and particularities of the Canadian research funding landscape. For example, a reader from outside Canada may not be familiar with #SupportTheReport or the Fundamental Science Review. Merely providing a citation puts the onus on the reader to understand things that the authors should make clear and understandable.
 
2. There is often confusion about the definition of an early career researcher (ECR), particularly among policymakers. As it stands, this manuscript adds to the confusion. In the opening sentence, the authors refer to ECRs as a group that includes graduate students, postdoctoral fellows, and faculty within their first five years of appointment. This muddles the issues facing these groups. The issues faced by scientists who are still completing their training are considerably different from the issues faced by new faculty. Even within trainees, there are key differences between problems faced by graduate students versus postdoctoral fellows in Canada (for example, as the authors note: access to extended health benefits, employment insurance.) The workshop was organized by postdoctoral fellows and seems to have focused almost entirely on issues facing postdoctoral fellows and graduate students. The parent organization was postdoc-organized. ECR participants at the workshop seem to have been largely, or perhaps entirely (this is not clear due to missing data in Table 1), postdoctoral fellows or graduate students. Thus, aside from some introductory paragraphs about new faculty, which read as distinctly out of place within the rest of the manuscript, there is no apparent representation of new faculty’s views about any of these issues. To improve the coherence and clarity of the manuscript, I recommend that the authors (a) clearly define ECRs as graduate students and trainees, (b) put the definition in both the abstract and in the main text, and (c) remove all sections focusing on new faculty.
 
3. What fields are covered in this manuscript and/or by the participants? The authors appear to be largely in biomedicine, with perhaps some representation in natural sciences? Many of the references and terminology hint to the majority of people involved in this work being in biomedicine, or, as termed at the Canadian Institutes of Health Research, “pillar 1” or “theme 1”. The parent organization, Future of Research, also focuses on biomedicine. The other pillars of health research (2: clinical research, 3: health services research, 4: social, cultural, environmental, and population health research) do not seem to have been represented. This is suggested, for example, by the focus on industry as the logical non-academic path for employment, which is not the case in these other areas. In addition, were researchers from other fields (e.g., engineering, social sciences, humanities) represented? The disciplinary backgrounds of the people involved should be explicitly stated and it should be clarified for the reader whether the findings and views presented in the paper reflect the entirety of the Canadian research landscape or are focused on one or more particular segments.
 
4. Is this a policy brief with an appended report, a research paper, or an opinion/commentary paper (the category noted on the manuscript)? It presents aspects of all three of these. It is not possible for one manuscript to be all of these things. I would encourage the authors to pick one and do it well rather than trying to make this manuscript try to be all three simultaneously. The guidance offered by the F1000 website states quite clearly that, “Opinion articles must focus on previously published literature and not include new research and data.”
 
Abstract
 
5. The authors state, “however, Canadian policies are trailing behind the progress being made in other countries.” As far as I can tell, this is never again discussed nor referred to, and no citations are provided to support this statement.
 
Executive Summary
 
6. The first major recommendation by the authors is to, “1. Improve ECR-targeted funding, including grants which provide operating costs for ECRs and support the transition from a PDF to junior faculty position, and recognising ECRs contributions to grants awarded to their supervising professors.” I have three concerns about this. First, experiences in other countries suggest that transitional grants may simply prolong one’s period of career uncertainty. In other words, rather than years of uncertainty ending at the point at which a scientist who wished to remain in academic science either secures a position or doesn’t, such transitional grants move the “up or out” point to several years into one’s faculty position. Second, are the authors concerned that such an option would impact negatively on start-up funds committed by institutions? Faculty start-up funds are already lower in Canada than, for example, in the US. Such a strategy risks further reducing institutions’ commitment to new faculty by expecting them to use their transitional grants in the place of providing start-up funds. Third and finally, can the authors please clarify what they mean by “recognising ECRs contributions to grants awarded to their supervising professors”? I think they are referring to the situation in which a trainee can either be listed on a grant as a co-investigator or be paid off the grant, but not both. However, this is not clear and should be stated explicitly.
 
Introduction
 
7. The opening sentence refers to, “science, engineering, and social research,” in Canada. This is an unusual collection of terms. Where is health research in this list? What happened to the humanities? More importantly, this broad opening leads me as a reader to expect that this paper will address all research areas, when in fact, as noted above in comment 3, this does not appear to be the case. It would help readers orient themselves if the authors were to avoid inviting this confusion in the opening line of the manuscript.
 
8. The authors state in their first paragraph, “However, there has been growing concern that current funding and training structures almost entirely ignore the best interests of ECRs.” I am not aware of any longitudinal analyses of such concerns. Can the authors please provide a citation to support the assertion that such concerns are growing? It would also be helpful to clarify what they define as “the best interests of ECRs.”
 
9. In my view, the Introduction would be stronger overall with the first paragraph completely removed. The second paragraph opens in a more compelling way and more swiftly leads the reader to an understanding of the problems addressed in the manuscript. If the authors were to do this, the concerns I raise above in comments 7 and 8 would no longer exist.
 
Workshop Overview
 
10. Given that the rest of the manuscript focuses on results from the second workshop session, it may be useful to have a summary of the first session beyond the list of people who presented. What did they present? Such a summary may or may not be possible at this date, depending on whether or not the session was recorded, but if it is possible, it would be a good addition to the paper.
 
11. To preface this next set of comments, I will note that my own training and current research combine social science methods with other fields. Also, I will note that this is the third manuscript I have reviewed this year from biomedical/natural scientists or clinicians demonstrating a lack of awareness of best practices in social sciences, including workshop and survey methods. It is too late now, but I hope that in future, the authors will involve colleagues in social sciences to help improve their workshop/focus group methods, survey methods, and analyses of data from these. Considering the data they already have and the things they already did and can’t undo, this manuscript might be improved in the following ways:
a. More detail is needed about the event. Who was invited to the event? How were invitations distributed? Were the organizers hoping to attract specific groups of people? How was this phrased in promotion materials?
b. More detail is also needed about the survey process. The authors state, “Before leaving the Sessions, participants were asked to complete an Exit Survey, which included questions regarding demographic data, opinions about the outcomes of the Breakout Session, and feedback towards future events.” Does that mean that participants were asked to complete the survey on paper right there, or was this a reminder issued to participants to please complete the online survey that would be sent to them later? Was the survey pre-tested in any way? Was it paper or online? The supplementary materials suggest it was a paper survey but it should be explicitly stated.
c. Can the authors please better explain how they grouped ideas? They state, “Note that the solutions supplied were not tied to specific problems that had been raised, and the grouping of this information into general themes has been performed ex post.” How was this done? By whom? Did the person/people doing this grouping have training and experience in qualitative research? Were two independent analysts involved (if so: what was their kappa) or did one person make these decisions? What qualitative methods were used?

Results
 
12. Like the Methods, the Results section is missing key data we would expect from this type of work. The authors should consult the Equator Network (equator-network.org) and consider the available reporting guidelines. Although I don’t believe there exists a reporting guideline for a policy workshop such as this, the authors might consult and consider which elements of STROBE (cohort study) and COREQ (focus group) apply to their work. If their survey was conducted online, I would suggest they also consult CHERRIES.
 
13. What was the survey response rate (number of completes/136)? It looks like the response rate was, at most, about 50%. That’s lower than we might typically expect for an event-linked evaluation survey. How did this happen? Did people leave without completing the survey? Did they not respond to requests to complete it online?
 
14. Several comments about Table 1:
a. In the questionnaire provided in the supplementary file, the categories of positions are: Postdoc, Graduate/PhD student, Faculty, Other (please specify). These same categories (and subcategories as appropriate) should be used in this table. If no participants belonged to a given category (e.g., Faculty) then this should be shown in Table 1 with zeroes.
b. Particularly with n<100, it is inappropriate to report mean age to one decimal point. Even if n>=100, it isn’t useful precision. There isn’t a meaningful difference between 32 years old and 32.2 years old. Please report mean age as integers.
c. Again re: reporting of age, for these kinds of descriptive statistics, one would typically report a measure of central tendency (usually either sample mean or median, depending on distribution) as well as a measure of dispersion (sample standard deviation if mean is used, interquartile range if median is used, sometimes full range.) If the authors are unsure which is most appropriate for their data, they may wish to consult with the statistical consulting service at their institution. These services often provide free or inexpensive consulting to students and trainees.
 
15. Several comments about Tables 2-5:
a. Tables provide grouped listings of perceived problems and suggested solutions. I would suggest naming them this way rather than problems and solutions, to be clearer.
b. Many of these points are opaque. For example, what do the following points mean? “Getting the science to the public” (What science? Which public? How would one define having gotten science to the public?) “Short term pilot project grants with no expectation of returns” (Surely this is not suggesting funding with no anticipated returns whatsoever?) “Quality of the peer-review system” (This is so vague it’s meaningless. Can this be better explained?) “Professional Development (PD) topics too centralized” (What does this mean?) “Lack of extracurricular training” (What is this?) “Pressure for sycophancy to assist career” (What does this mean?) This is not an exhaustive list. There are many other points whose meanings are unclear.
c. Some of the points display a lack of awareness about the research landscape. For example, one such comment is, “Engage reviewers from outside academia for relevant expertise on knowledge translation/knowledge mobilization.” While, in my view, it’s good to have knowledge users and knowledge brokers from outside academia reviewing relevant grants, the authors might consider consulting the Knowledge Translation panel from the last cycle of CIHR Project grants (KTR in the list available here: cihr.ca/e/50845.html), the reports from KT Canada, or the list of people responsible for KT at each province’s SPOR SUPPORT unit and perhaps reflect on whether they are aware of academic expertise in this area. Other such examples include, “Mandated staff composition of labs and groups (by funding agencies).” This is outside the purview of funding agencies.
d. Some of the points contradict each other. For example, directly one after the other are the suggested solutions, “Increased funding for translational and applied research,” and, “Funding devoted to basic research.”
e. Some of the points appear to be poorly classified. For example, “Convince public that research has value even without immediate application,” is listed under Problems while, “Convey to the public that all science matters, not just ‘hot topics’,” is listed under Solutions.
f. A number of the points repeat themselves across tables and some categories lack meaning (e.g., “Assorted.”) See comment 18 below for a potential suggestion of how to address this.
g. While it’s lovely to have the raw data supplied (thank you), the data themselves are even more difficult to interpret than the points in the tables.
 
16. The workshop included trainees in Vancouver from the University of British Columbia and Simon Fraser University. If I am understanding correctly, there were no participants from other academic institutions. Is that correct? This is completely understandable given the nature of the event and the geography of Canada. However, it means that some of the ideas come across as under-informed. For example, Canadian institutions already offer mentorship training to PIs. Université Laval offers such training and, while it is not mandatory, it is strongly recommended. (To illustrate: I have taken it, as has every professor in my unit.)
 
17. Related to the above, some aspects of the report read as national; for example, statistics about Canadian funding, while others read as local; for example, references to, “Vancouver’s high cost of living.” The authors might wish to reflect on whether they wish to add more national context (e.g., “high cost of living in cities like Vancouver and Toronto”) or focus even more on issues that are specific to trainees at institutions in Vancouver.
 
18. Similar to the above, and related to point 15c above, one of the suggestions was, “Training for KT in grant proposals.” Such training exists; for example, trainees may wish to explore the training programs available through the Li Ka Shing Knowledge Institute at St. Michael’s Hospital, Toronto (knowledgetranslation.net), KT Canada, or other organizations. Given they are in BC, they should contact their provincial Strategy for Patient-Oriented Research (SPOR) Support for People and Patient-Oriented Research and Trials (SUPPORT) unit (bcsupportunit.ca). BC’s KT component is led by Dr. Linda Li at UBC. They may also wish to consult the list of resources here cihr.ca/e/49443.html. This is but one example of many in which the authors present perceived problems and suggested solutions with no acknowledgement that such solutions already exist and, indeed, have existed for quite some time. (To illustrate: I attended a KT Canada Summer Institute as a PhD student in 2008.) Others include things like reviewer training, which is already underway at CIHR, or networking opportunities, which raises the question of whether the participants have made use of the existing networking opportunities available to them? It took me less than 60 seconds to search for ‘biomedicine networking event Vancouver’ and find three recent such events. Of course, it is not reasonable to expect trainees (nor anyone) to know every opportunity that’s out there for them, but this manuscript would be improved by better mapping the perceived problems and suggested solutions onto current opportunities to identify true gaps. Right now, unfortunately, the results come across as a very long, semi-organized laundry list of perceived problems and suggestion solutions that range from very insightful to woefully under-informed. This is a shame, as the authors and their colleagues clearly put a great deal of time into understanding the context of research funding, organizing the workshop, and preparing this report. To help this manuscript better capitalize on that effort and the expertise that the authors and workshop participants bring re: their experiences as trainees, the authors might consider identifying a few key issues and solutions and contextualizing them in the broader literature. For example, the structure of postdoctoral training programs may be present as a key theme. This could be put in the context of previous literature such as: PMIDs 27543634, 25673353, 25771193, etc. This would also help reduce the large number of repetitive tables. The authors might consider consulting the chart in the paper published by Future of Research as an example of how taking the time to synthesize can help to better present data from a workshop such as this. Participants’ ratings in Tables 6-9 might help identify the perceived problems and suggested solutions that are most important and worth highlighting. Grouping and identifying key issues and solutions is particularly defensible given the authors’ own statement in the Discussion that, across breakout sessions, “solutions showed significant overlap.”
 
19. There is quite an emphasis on mentorship in the results and discussion. I wonder if the trainees who participated in the workshop have realistic expectations regarding what mentorship is and should be. Even the best-trained, most well-meaning PI may have trouble mentoring trainees seeking positions in industry, simply out of lack of experience in industry and a weak or nonexistent industry network. If the authors choose to continue discussing mentorship as one of their key issues, the manuscript might be improved by drawing more on the relevant literature. I recommend the authors consult work by Dr. Sharon Straus, who has done a great deal of work studying mentorship in academic medicine. Her book written with Dr. David Sackett is comprehensive, and the authors may wish to consult papers such as this systematic review of practices of good mentors and good mentees (Sanbunjack et al¹. Of note, the authors of the systematic review found that good mentees should be in the ‘driver’s seat’, and should be respectful, organized, and committed. The point about mentees being in the driver’s seat is particularly salient here, because if trainees wish to be better equipped to obtain a job in industry, that is a goal they should be driving towards. Reading some of the comments about mentorship, it is not evident that this is well-understood by workshop participants. It is reasonable to expect one's supervisor to support one's career progression. All supervisors should do this. However, in the same way that you wouldn't reasonably expect your hockey coach to help you prepare to play water polo, aiming for a career in industry will require a broader mentorship team than just your PI.
 
Discussion
 
20. The authors seemed to choose certain issues to highlight in the Discussion, for reasons that are not entirely clear. If they choose to follow my suggestion in comment 18 above, their selection of which issues to highlight will be driven by the data from exit surveys. Alternatively, if they choose to keep the current format, they should give the reader more of an indication about why they are focusing on these specific issues.

Minor comments
 
21. The authors open a paragraph in the Introduction, “Despite these important concerns, there is hope for the future.” Although this is categorized as an opinion article, this level of editorializing is probably unnecessary. However, I leave it to the authors to determine what kind of article they wish to write.
 
22. In paragraph 8 of the Introduction, the authors state, “Furthermore, it proposes $210 million over 5 years for Canada Research Chair (CRC) appointments specifically for ECRs.” This did not align with my interpretation of the budget, so I went back and re-read page 89 of the full budget document. As I recalled, the budget signaled a desire for these CRCs to be allocated to more junior researchers, however, this is not the same as the authors’ definitions of ECRs. The Government of Canada may choose to use overly broad and vague terms like “early career researcher” to refer to the recipients of Tier 2 Canada Research Chairs, meaning people with faculty appointments and who are up to 10 years past the receipt of their PhD plus extensions for leaves. However, the authors should strive to be clearer. Furthermore, the Government of Canada “expects” the granting councils to target new funding to early career researchers, which means this will likely occur, but this is a little more nuanced than the authors’ statement that these funds are “specifically for” ECRs.
 
23. At the beginning of the Results, the authors note, “From the 65 responses, the average age of attendees was 33 years old, with only Session 3 showing a notable digression from this average.” Unless the authors believe that the higher mean age is meaningful for some reason and may explain differences between this session and others, I’m not sure this adds all that much to the paper. Also, it looks to me that this is likely because of the preponderance of non-trainee participants in this session (e.g., all the university staff at the workshop attended this session, constituting a full quarter of the people in the session.)
 
24. Were the authors among the n=136 participants? Were authors’ views included in the data presented? If the answer to either of these is yes, this should be explicitly stated.

25. In the Discussion, the authors state that, “Recent campaigns for increased research funding in Canada have led to the creation or increased the activity of ECR advocacy groups such as the Association of Canadian Early Career Health Researchers (ACECHR) …” ACECHR was not formed to campaign for increased research funding, though the organization has certainly worked on this as one of its more recent activities. The organization was formed in reaction to changes to the grant program structure and grant review mechanisms at the Canadian Institutes of Health Research (CIHR) in 2014. Modeling by Dr. Hendricks suggested that new investigators in their first five years as faculty were poised to lose approximately $30M in grant funding in the new system, and indeed, the first new cycles suggested that without the stopgap measures put in place at the CIHR in response to ACECHR advocacy, that size of gap would have occurred. As the person who wrote the first draft of many of ACECHR's documents and now knows all the train numbers and schedules between Quebec City and Ottawa by heart, I can assure you that ACECHR's participation in advocating for sustainable federal research funding was neither the reason for ACECHR's creation nor was it an increase in ACECHR's activity. Joining in with a number of other organizations, with practically the whole Canadian research community on side, was far less work than the activities ACECHR did alone. Please correct this.