Use and effectiveness of the Individual Development Plan among postdoctoral researchers: findings from a cross-sectional study

As noted by Review 1, the manuscript is well written and easy to follow. However, I would agree with both of the other reviews that problems arise from both the survey questions and decisions on method of analysis. The first 3 questions about completing an IDP don’t really lead to a clean number of how many people do and do not complete one. A person could answer no to being required to complete an IDP but could be doing it voluntarily.  And the issue of doing it at all vs. doing it annually makes the numbers even more ambiguous and difficult to sort out.  An additional challenge comes from the 5 choices for questions 1-3 when they are really only yes/no situations. It is hard to imagine what could lead a person to be neutral on these 3 questions.

I would also agree with the other reviewers that there is no rational for combing neutral responses with negative responses. This will skew the interpretation to a negative side without any evidence the respondent meant it to.  This is one reason surveys often don’t provide the neutral point as the data are very difficult to interpret. At least, as reviewer 2 points out, including neutral as a distinct option would allow readers to reach their own conclusions.

I also agree that % responses to questions 4 and 5 should be based only on those who actually complete the IDP as others really don’t have any basis for judgement. But as noted above and by other reviewers, this number of those who completed it is illusive from the question designs.

RE: Figure 2, as pointed out by the other reviewers, it really does not reveal effectiveness of the IDP.  At most it displays associations between some of the questions. I also would raise a concern with the 2 questions about mentors: “My PI/advisor provides real mentorship” and “My PI/advisor provides ample support”. Both of these are very ambiguous – e.g. is the intent to separate ‘real’ mentorship from some other form of mentorship? And what kind of ‘support’ – financial, psychosocial, professional? This level of ambiguity adds to concerns for including neutral responses with the disagree categories because neutral could easily reflect not knowing what the questions are asking for.

This article summarizes data from a subgroup of individuals surveyed about their use of an Individual Development Plan and other factors. Data on the effectiveness and usefulness of the IDP is important to justify policies that require postdoctoral scholars and PhD students to complete an IDP. The findings are interesting and concisely presented. The authors appropriately acknowledge several limitations to this survey. Overall, this manuscript adds important data to a field that is very difficult to quantify, given the variability in the IDP across institutions and training programs. I do have some questions and suggestions to strengthen this manuscript:

1.  As noted by other reviewers, the separation of the Likert scale data into “agree” and “disagree” categories, with “neutral” included in the “disagree” category, has the potential to skew results towards the “disagree” category.  It would be helpful to analyze the data with neutral responses listed as a separate category. 
2. The survey population demographics as reported in Supplementary File 2 show that the population of respondents was 80.7% white. How do the demographics of the survey population reflect the national postdoc population demographics? The numbers in some categories are likely too small to analyze in a statistically meaningful way, but it would be interesting to determine whether different demographic groups (race, gender, etc) had similar responses regarding the usefulness of the IDP and/or mentoring relationships. 
3.  I agree with other reviewers that this paper has relatively minimal introduction and discussion to place it in the context of other work. The issues faced by postdocs are not identical to those faced by PhD students, and there are multiple recent publications discussing the problems facing postdocs (for example, The Postdoc Experience Revisited, National Academies Press 2014¹; Alberts et al, PNAS 2014²). The data in this manuscript that show that only 22.4% of survey respondents felt that the IDP process was helpful to career development suggest that this process may not be an effective strategy for improving the postdoctoral experience and outcomes.

General comments:
This publication addresses the very important topic of use and effectiveness of the individual development plan (IDP) for trainees. This is not a trivial topic to address and I commend the authors for this analysis in the context of current literature. I also appreciated the valuable insights on how the effectiveness of the tool is associated with positive mentoring, pointing out the need for strong relationships with advisors that can affect career trajectories for graduate students and postdocs.

General considerations:
On the background side, not being terribly familiar with the already existing literature on this topic, I would recommend a bit more description of prior studies that could frame this work, and its novelty in the context of existing literature. However, I recognize that it’s also possible literature on the effectiveness of the IDP may be limited, in particular given the abstract nature of this concept. Without having read more background, but judging from the information presented in the introduction, my immediate impression is that prior studies looked at PhD students whereas this work examines postdoctoral researchers. Moreover, the authors mentioned that this was a multi-institutional analysis, which leaves me wondering whether the novelty of this work is looking at a different population (postdocs vs. graduate students) or whether it is related to the number or type of institutions that prior studies hadn’t examined in this context.

On the technical side, the abstract mentions that the authors looked at data from 183 postdocs, although from the methods section it appears that both graduate students and postdocs were examined for a period of March to June 2016. Given that a study of 663 PhD students was previously performed as referenced in the introduction, it would be helpful to see an articulation of the novelty that this current study brings over previous work. It would also be beneficial to clarify whether both populations were included in all the analyses in this publication (in particular figure 2, which appears to refer to PhD students, whereas figure 1 refers to postdocs), and to expand more on the similarities and differences between the effectiveness of the IDP on these two populations. While this is mentioned in the discussion section, if both graduate students and postdocs were analyzed more in-depth in these studies (also taking into account other aspects of the data in this publication), it would make for an interesting comparison as to whether or not the IDP has more of an effect on the career trajectories of one group or another. And while positive mentoring is required for both populations, postdocs may already be well on their way towards a more obvious career path than PhD students are. While the authors state similar trends in these topics, it would be helpful to take this analysis a step further and determine how IDP effectiveness affects career choices for these groups.

In terms of data analysis, the authors state that they grouped together the agree/strongly agree, and neutral/disagree/strongly disagree responses. I wondered why this is the case, perhaps it could be a low sample size that may not enable meaningful conclusions. However, in order to gain a fully comprehensive picture of the issue at hand, I would recommend displaying and analyzing data from each of these categories separately. I believe that for such a topic that is difficult to quantify, it will be important to dissect the prevalence of each of these responses. Examining the various categories (availability of programs, attendance and usefulness) in Supplementary Data 2 could be utilized for a more thorough analysis of this topic. While I wonder how usefulness can be assessed in a practical sense, it was also disheartening to see the percentage of respondents who indicated that they do not attend or did not find available programs helpful. Perhaps this is an area that should be further explored in terms of which programs would likely be helpful for trainees to have. It would also be interesting to dissect further the correlation between the usefulness of the IDP and subsequent career paths chosen. For example, did individuals who found the IDP helpful end up in the top career path predicted by the IDP, and are they currently satisfied in their position? If so, this might indicate that the IDP was useful in helping them achieve desired career goals. The IDP could also open them up to career paths they hadn’t considered before, which would demonstrate the added value of this tool for training and career development.

I appreciated the data transparency in this publication (for example in Dataset 1, Supplementary Data 4). Given the wealth of information and number of questions asks, further analyses of these existing data looking at the effect of other variables on IDP effectiveness would provide a thorough analysis of how we might improve the IDP process based on barriers faced by particular groups. I would also suggest detailing the data analysis and quantification procedure used in this publication (as opposed to referencing prior publications with the information), in order to clarify how percentages in the results were calculated.

I also wonder whether it’s possible to examine other variables together to make predictions that would enrich this publication in the future. For example, how do factors such as race, ethnicity, U.S. citizenship, and others, affect the responses to the IDP process (Supplementary Data 3). While these may not have been the primary objectives of the authors, this type of analysis would add another layer of complexity to whether the IDP is useful to various groups, whose career decisions may also be affected by additional factors. This publication does contain a large amount of raw data that I think could be utilized for a more thorough analysis of how various factors contribute to the effectiveness of the IDP. However, given the data is self-reported and there may be a limited number of responses in each of these categories, it may be difficult to assess the effect of such variables with the current dataset.

Feedback on results:
It was somewhat disheartening to see the percentage of postdocs who had completed the IDP without discussing it with their advisor (Figure 1), and the percentage of those that had honest conversations with their advisors was also not terribly high. These factors point to barriers towards positive mentoring relationships in academe, as well as obstacles to career development for trainees. They could affect the ability of trainees to follow desired career paths, or having to prepare for transitioning into these careers without their advisors knowing, especially if the advisor does not approve of their non-academic career choice. This fundamentally points to systemic flaws in academe and how the enterprise needs to change in order to better support trainees who are using the IDP as a guide to explore career options. Importantly, this also requires advisors to point trainees in the right direction, and be a sounding board during career transitions. I also wonder whether there is a connection between the lack of discussions with the advisor and their ability to have honest conversations (Figure 1), as it appears that this could be a layered response (i.e. they were either having or not having these conversations, and if they were, how honest did the trainees feel they could be with their advisors in terms of desired career options?). I think drawing a connection between these two variables could be valuable to investigate in terms of the barriers affecting the ability of trainees to pursue various career paths, and assess the usefulness of the IDP process for these particular careers.

Figure 2 somewhat addresses this concern, and it was valuable to see that positive mentoring and having honest conversations with advisors can influence the responses of trainees on IDP effectiveness. There is a lot of really valuable information in this figure in terms of how we can improve faculty training to be more supportive of the career choices of their trainees, so that they feel valued and prepared for taking on other careers besides academia. Given the importance of these factors, it would also be interesting in the future to look at how positive relationships with advisors affect other aspects of training and career preparation for trainees. While factors such as the advisor being an asset to their career, providing ample support and positively impacting their emotional and mental well-being, among others variables, are likely very difficult to assess, I believe these are critical investigations that should be pursued further and more in-depth to better understand how to train the next generation of researchers. Along the lines of these ideas, putting these findings into a larger context would be really helpful in discussing how to better equip faculty to help trainees be successful in their desired careers.

Additional recommendations:
It was interesting to learn about the comparison between IDP use and effectiveness for PhD students and postdocs, as detailed in the discussion section. I was surprised to see that postdocs weren’t required to complete the IDP to the same extent that PhD students were, did not discuss the results of completed IDPs with their advisor as much, and found the IDP less helpful for their career development. This observation that merits further investigation, as to whether the lack of usefulness of the IDP for postdocs was due to their inability to discuss it with their advisors, or whether other factors were also involved. I would also be curious to know more about why there is a lesser requirement for postdocs to complete the IDP, and whether reversing this trend would result in a greater percentage of postdocs actually pursuing desired career paths as opposed to traditional academic routes.

In terms of comparing data from PhD students and postdocs, I wonder whether these surveys and subsequent analyses were performed on both populations at the same time (during March to June 2016 as described in the methods) or whether the data discussed here on PhD students came from a previous publication. This analysis could also provide insights into whether we should target certain populations more in terms of IDP assessments, and which populations within academe the IDP is likely to be more useful for in terms of career exploration. For a more extensive analysis, it would also be interesting to compare all of the aspects in Figure 2 between PhDs and postdocs, in order to determine the effect of mentoring relationships on career trajectories of trainees at various stages in their careers.

I appreciated that the authors pointed out limitations of the study in the discussion section, including as it relates to institutional variability. Indeed, Supplementary File 1 indicates that there are very few individuals at the institutions shown in the dataset, and many are at missing institutions. I imagine there is also quite a variability between these institutions in terms of size, number of postdocs, and the type of career development opportunities available that could supplement the IDP findings for trainees. These variables could influence how trainees rate the usefulness of the IDP, in terms of whether additional resources exist for them to further explore careers that were indicated as a good fit from the IDP. For example, it is possible that a limited knowledge on available career options, either due to the lack of resources or the inability of their advisor to help (in addition to not being able to find another suitable mentor to assist with career exploration), trainees may rate the usefulness of the IDP as lower than those with more external information available.

Increasing the sample size of respondents from each institution would also provide a clearer picture of how institutional environments affect career trajectories for trainees. In addition, incorporating other variables into the evaluation of institutions would enable various types of comparisons to be made about IDP effectiveness by trainees from diverse backgrounds, or those in institutions of a certain size or geographical area. These are also factors that could affect their career development - for example a larger city might offer opportunities to interact with other postdocs and take advantage of multi-institutional career development opportunities, which trainees in other geographical areas may not have access to.

Broader picture comments:
I agree with the authors that faculty should receive mentorship training and it would be helpful to see further elaboration by the authors on how this could be achieved. For example, mentorship training for faculty could include manuals with both internal and external resources and contacts from various career paths that trainees might want to pursue, thus enabling them to better train their postdocs for appropriate careers. There is also currently the barrier of trainees not being able to have honest conversations with advisors about their career options, therefore faculty attitudes need to change in order to allow postdocs to pursue non-academic careers.

I also agree that a better career development infrastructure is needed, and that this would be a massive undertaking. Incorporating the findings from this publication, however, into current literature on these topics (and efforts made by others to reform career training in universities) would be a helpful beginning to understanding how such an infrastructure could be developed. Implementing the IDP as a mandatory training for postdocs at the bench as part of their annual assessment may already be happening at some universities, however we should also keep in mind that IDPs are really only the beginning of the career development process - while it is the responsibility of trainees to utilize their IDP results for further career exploration, an infrastructure that supports this process is imperative to their success. This infrastructure I envision could be internal to the university, or there could be an external entity developing resources for several universities to utilize for helping trainees explore career options. As part of this work, the authors could also consider developing a rubric to measure IDP effectiveness as it relates to their ability to achieve career goals outlined in the IDP. I would also be curious to see a rubric for assessing other factors that can influence this effectiveness (such as those in Figure 2) and trying to understand more about particular elements that go into each of these factors.   

Overall, this is a well written manuscript tackling an issue that is difficult to quantify but very important to study from the context of training the next generation of scientists. I believe that more of an in-depth literature overview, further analysis of the existing data and collection of additional data, and a more extensive discussion of the recommendations for change around faculty training to support postdocs, would greatly strengthen the manuscript in the future. I believe these findings are a valuable foundational start to these questions, conducting further investigations on this topic can provide more in-depth understanding of the potential that the IDP could have for training graduate students and postdocs for being successful in their chosen careers.

The authors report on the usage of IDPs by postdoctoral scholars, which is both a timely and fundamental topic within the broader graduate and postdoctoral professional development community. This work extends beyond the authors’ recently published article on IDP usage among doctoral students¹ to showcase how the instrument is currently being used by postdocs. The aforementioned manuscript on doctoral IDP usage extensively discusses policy and other issues surrounding IDPs, while this manuscript is lean on discussion. It would therefore benefit from including prior literature on correlates of success associated with postdocs who have written plans—which would have the added benefit of placing this work into a broader context (for example, see: Davis 2009²).
 
There are a number of points that should be addressed, and they are outlined as follows:

1. The survey instrument asks questions on a 5-point Likert scale of strongly agree, agree, neutral (neither agree nor disagree), disagree, & strongly disagree. However, when analyzing the data, the authors report percentages either as ‘does not agree’ (also reported as ‘disagree’) or ‘agree’ - with ‘strongly disagree, disagree & neutral’ all grouped together as a ‘does not agree’ response. It seems that lumping “neutral (neither agree nor disagree)” into the ‘does not agree’ category would skew results (both in this manuscript and the manuscript on doctoral IDP usage) towards ‘does not agree.’ It is recommended that the authors reanalyze the data and report the ‘neutral’ responses as a third category in order to more accurately reflect intended answers/percentages.
    
2. I agree with reviewer 1 that it is unclear how the authors calculated that 54% completed an IDP; like reviewer 1, I also calculated that 61% completed an IDP either with or without their PI. Similarly, I also agree with reviewer 1 that it is unclear where the 38.4% value came from (percent of respondents that completed an IDP who found the tool helpful). Furthermore, in figure 1, a value of 22.4% of all respondents (whether they complete an IDP or not) is listed as saying the IDP process is helpful to their career development. Since “22.4%” also includes those who never completed an IDP, reporting it in this manner seems to bias perceptions of the tool as unhelpful.
    
3. Aside from the unclear derivation of percentages discussed in point #2, the questions in the survey instrument do not seem to allow confident discernment of who actually completed an IDP. For example, respondents who disagree with the question “I complete an IDP annually with my PI/advisor” could have actually completed an IDP with their PI at the beginning of their postdoc and/or possibly in year 3 of their postdoc—how much weight were respondents giving to the word ‘annually’? Another question reads: “I complete an IDP but I do not discuss it with my PI/advisor” - could respondents complete two different IDPs - one with and one without their PI (and thus agree to both the former and latter questions)? 
    
4. The authors assert that the usefulness of an IDP has decreased since 2014, specifically comparing their results to that in (Hobin et al. 2014³). However, in the (Hobin et al. 2014) paper, an IDP’s overall value is reported either as ‘not helpful’ ‘neutral’ or ‘helpful’. It seems that the results in this manuscript could be compared more accurately with the (Hobin et al. 2014) data by showing the ‘neutral’ responses, rather than lumping them with ‘does not agree’ (see point 1 above). Additionally, it would be helpful to point out the limitations of comparing these two studies (ex: address key differences between the two survey instruments regarding how the IDP questions were asked (and how this might bias responses), address potential respondent audience differences, etc.). 
    
5. Like reviewer 1, I also feel that there are limited questions that address what one might consider IDP ‘effectiveness.’ The manuscript would thus be strengthened by discussing IDP indicators that have been previously reported in the literature (such as measures outlined by (Hobin et al. 2014) - ex: the value of an IDP in helping with self-assessment, helping identify career paths, helping identify skills to strengthen, etc.). Furthermore, (Davis 2009²) reports an in-depth analysis of results from a Sigma Xi Postdoc Survey - identifying many positive correlates of success associated with postdocs who develop a written plan at the outset of their careers (ex: higher publication rate, grant submission rate, better supervisor relationships, etc.). Since the main point of this manuscript is to discuss the use and effectiveness of IDPs among postdocs, it would benefit from elaborating upon such postdoc-specific contextual literature - as well as other literature that documents the general benefits of goal-setting, which is a primary function of IDPs (Locke et al. 2002⁴).
    
6. (Minor) The authors indicate that “additional research is needed,” and it would thus be beneficial to clarify the research questions that should be addressed. For example, it seems that it would be useful to determine the effects of various parameters on IDP effectiveness such as: 1)  when in training an IDP is completed; 2) inclusion/ exclusion of IDP components (such as self-assessment, career exploration, skill-building, goal-setting, etc.); 3) prior experience with/completion of an IDP as a PhD student; 4) completing an IDP of their own accord versus doing so because it is required; 5) receiving training on how to create/implement an IDP; 6)  using a specific IDP instrument (ex: myIDP) versus an institutionally-developed IDP, etc..
    
7. (Minor) As an additional point - since ‘IDP effectiveness’ is subjective as the authors point out, perhaps future studies should address better-defining these parameters so that common IDP evaluation methods can be adopted within the broader community. It would also be especially helpful to ascertain what IDP tools are being used, and how they are being implemented so that standard ‘correlates of effectiveness’ could be tied to specific IDP instruments (or components) and the manner in which they are employed.