How to incorporate patient and public perspectives into the design and conduct of research

International government guidance recommends patient and public involvement (PPI) to improve the relevance and quality of research. PPI is defined as research being carried out ‘with’ or ‘by’ patients and members of the public rather than ‘to’, ‘about’ or ‘for’ them ( http://www.invo.org.uk/). Patient involvement is different from collecting data from patients as participants. Ethical considerations also differ. PPI is about patients actively contributing through discussion to decisions about research design, acceptability, relevance, conduct and governance from study conception to dissemination. Occasionally patients lead or do research. The research methods of PPI range from informal discussions to partnership research approaches such as action research, co-production and co-learning. This article discusses how researchers can involve patients when they are applying for research funding and considers some opportunities and pitfalls. It reviews research funder requirements, draws on the literature and our collective experiences as clinicians, patients, academics and members of UK funding panels.


Introduction
Patient and public involvement (PPI) is recommended from the earliest research stages through to dissemination of the findings [1][2][3][4][5][6] . In the UK, INVOLVE 3 states that research should be done with and by patients, but what does this mean for researchers and patient partners when starting a study? International resources are available (Box 1) and six UK PPI standards are being tested to see if they work in practice 7 . Table 1 summarises on-line guidance for research applications to international government funding programmes that endorse involving patients and the public. Language varies internationally and is evolving as patients take a more central role in deciding what research is done and how. Box 2 provides some definitions which are derived from the INVOLVE jargon buster 8 and international resources ( Table 1). PPI includes patients, potential patients, families, carers, patient groups and members of the public who use or have access to health and social care services 3 . We refer to this broad group as 'patients' to distinguish them from clinicians and academics. This is consistent with Canadian guidance, which defines 'patients' as 'an overarching term inclusive of individuals with personal experience of a health issue and informal caregivers, including family and friends' 9 . However, 'patients' may include people who do not describe themselves in this way. People may selfcare for their condition and general public contributions can add value to research questions. Other relevant international terms, for example stakeholder involvement, consumer involvement, knowledge user engagement and patient orientated research are described in Supplementary File 1, Section A.
PPI is put into practice through patients discussing, helping to make decisions and occasionally doing research in order to enhance study relevance, design, conduct and governance. There is no 'one-size-fits-all' approach. Flexibility is required to tailor patient involvement to the topic, research question, methods and resources available. This article describes steps that researchers and patient partners can follow when preparing a research funding application (Box 3). We refer throughout the article to an illustrative example of a researcher who wants to do a study to improve outcomes for patients with migraine, and we provide examples from the literature and authors' experiences.

Steps for how to involve patients and the public when applying for research funding
Understand what patient and public involvement is At the outset, it is important to understand the theory underpinning PPI. In depth reviews and discussion of theory are available 1,4,10-13 and suggest that, depending on the circumstances, PPI will: -ensure that the research questions and outcomes really matter to patients -provide perspectives that complement or challenge those of researchers and clinicians -make research more relevant to the people whom it is designed to benefit -ensure that proposed research will be acceptable to patients so that they will be willing to participate -improve the quality of research -offer lay knowledge that is either independent for the purpose of governance, or specific to the focus of study to enhance its design or conduct -make research more equitable and ethical, particularly when publicly funded -improve dissemination to reach wider lay audiences -increase the likelihood that research will be implemented into everyday practice and impact on patient care -enable patients to feel that their voice matters.
All of the above could reduce research waste 14-16 if PPI is put into practice in ways that ensure that research is meaningful, acceptable, ethical and useful.
How does patient involvement differ from patient participation in research? Patient perspectives can be sought through patient involvement and through patients participating in surveys, interviews or focus groups to provide data for others to analyse, interpret and act on. The authors have observed that in grant applications and study protocols, PPI is often conflated with qualitative research or patient opinion surveys. Collecting data from patients can be important to gain diverse or representative views, but it is different from PPI and both are often needed (Table 2). Discussion with patients at a workshop can seem similar to collecting data in a focus group, because both involve listening to patients' perspectives, but the context and outcomes from listening differ. PPI means that researchers are in a continuing and reciprocal relationship with patients and make decisions with them about the research. In qualitative research, researchers listen to patients in order to au/book/nhmrc-advice-and-instructions-applicants-2017/nhmrc-adviceand-instructions-2017 Indigenous populations: https://www.nhmrc.gov.au/book/nhmrc-funding-rules/section-b-projectgrants/b-attachment-d-guidance-applicants-address-criteri Mandatory Plain English Summary.
Consumer and community engagement sections are not always mandatory and applicants can decide 'if applicable'.
There is either a form section to complete which asks applicants to enter each community engagement activity separately or instructions to upload either a one or two page document.
Fellowship applications require information about community engagement and participation for the previous 5 years or 10 years.
Some forms have 'if applicable' sections which ask applicants to describe: a) how participants will have access to their own results and how the researchers will be accountable to participants for the overall results of the research: b) how they will ensure consumers are involved in the research and how they will communicate results to participants and the community. The European Union (EU) website refers to 'citizen Involvement' which includes upstream priority setting, influencing funding decisions to a more direct downstream involvement of citizens and patients in the use and application of medical knowledge and information. It covers both active citizens who engage from a position of agency as well as those unaware of their contribution 25, 'Citizen Science' is used as an EU umbrella term which is envisioned as various forms of public engagement with science as a way to promote responsible research and innovation.
Partnership is when people who get actively involved in research have a relationship that involves mutual respect and have an equal voice. This contrasts to someone who is consulted occasionally. PCORI consider that the principle is demonstrated when time and contributions of patients and stakeholder partners are valued and demonstrated in fair financial compensation, as well as in reasonable and thoughtful requests for time commitment. When PCORI studies include priority populations, the research team is committed to diversity across all project activities and demonstrates cultural competency, including people with disabilities, when appropriate.
Reciprocal Relationships is one of six PCORI engagement principles. They are demonstrated when the roles and decision-making authority of all research partners, including patients, are defined collaboratively and clearly stated.
Collaborating is active, on-going involvement in the research process; however, responsibilities are not equally shared like they are in partnerships. Patients may be co-applicants on a grant application, take part in an advisory group or work with researchers to design, undertake and/or disseminate the results of a research project.
Engaging is a term used in the USA by PCORI and the Canadian Institutes of Health Research. PCORI define engagement as meaningful involvement of patients, carers, citizens, clinicians and other healthcare stakeholders in the topic selection, design, conduct and dissemination of research findings. There are six PCORI patient engagement principles: reciprocal relationships; co-learning, partnerships, transparency, honesty and trust. The UK National Co-ordinating Centre for Public Engagement in Research (https://www.publicengagement. ac.uk/do-it) defines public engagement as: 'the myriad of ways in which the activity and benefits of higher education and research can be shared with the public. Engagement is by definition a two-way process, involving interaction and listening, with the goal of generating mutual benefit'.
Devolving is to place decision making in the hands of patients or communities, for example, a community development approach 26 .
Consulting is gaining feedback from patients and communities through e.g. meetings, on-line fora, workshops. The role is considered to be relatively passive when compared to 'engagement' Action Research brings about improvement or practical change. A group of people who know about a problem work together in a 'partnership' to develop an idea about how it might be resolved. They then go and test this idea. The people who take part in the testing provide feedback on their experiences. It has key tenets 20 : -Flexible planning -the detailed content and the direction of the research are not determined at the outset -Iterative cycles with all involved to i) decide what the problem is, ii) decide an action iii) take action iv) learn the lessons from the action v) reconsider the problem and repeat the cycle -Subjective meanings of those involved determine the content, direction and measures of success of the research -The research simultaneously improves the situation -The unique and ever changing social context is taken into account Co-production means people who use services, members of the public and professionals working together in a 'partnership' to produce research or service improvement. It is an umbrella term for a concept that means coming together to find a shared solution. 'Co-' can be put before specific research tasks like 'co-design', 'co-build' and 'co-construct'. Co-production 27 covers the whole research process from idea to dissemination of findings in order to change practice.
Co-learning is a term used by PCORI, where the goal is to help patients or other partners to understand research processes. The goal is not to turn patient partners into researchers. PCORI use the term in the context of 'reciprocal relationships', where all research partners including patients learn collaboratively. https://www.pcori.org/sites/default/files/Engagement-Rubric.pdf

Box 3. Overview of how to involve patients in research
A clinician wants to involve patients in a trial of treatment for migraine. Here are steps for involving patients when preparing a research funding application. ii. how will they add value and are there risks?
iii. will they be employed?
iv. who will mentor and provide supervision?  improve their understanding of a topic. Focus group discussions or qualitative interviews are audio-recorded and transcribed. Researchers collate, analyse and interpret text data from carefully sampled patients to produce valid new knowledge and generate hypotheses. Qualitative and survey research have systematic methodological quality standards. However, the researcher holds the power and patients may express strong views which may not be reported. In any research, the PPI and the data collection to gain wider patient perspectives can be separate, combined or overlap in some study phases, or they can be completely integrated throughout ( Figure 1). Any combination is possible (Supplementary File 2, Example 1). They are often combined and integrated in equitable partnership research methods like action research, and 'co-' prefixes to research terms, e.g. co-learning and co-production (Box 2).
Action research, co-design, co-learning and co-production Action research historically precedes co-production and gathered momentum in the 1940's as a community-led action in research initiative 19,20,30 . The UK National Institute of Health Research (NIHR) who fund research advocate co-production 27 as a method of involving patients meaningfully from start to finish of the research process. Differences in definitions (Box 2) are subtle, vary internationally and researchers may apply the approaches flexibly in practice. 'Partnership approaches' is used in this article as an umbrella term because it acknowledges the changing roles of patients beyond being 'participants' or 'subjects'. Partnership research methods involve patients, clinicians, academics and other relevant stakeholders as equal mutually respected partners in the research team. Being a patient partner implies equal opportunity and equal voice. Equal power in decision-making is sometimes implied, however there are structural and economic power differentials between different types of partner in terms of pay, employment contracts, status and workplace environments. As language is evolving internationally it is more helpful to describe actual patient roles, tasks and responsibilities explicitly rather than use a label for an approach that is open to misinterpretation. For example, co-production 27 may mean consulting patients regularly or patients may actively collect and interpret research data. Terms like 'Participatory Action Research' confuse because the definition of 'participation' in a study means to contribute data, rather than active involvement in research decisions. Partnership research teams decide who has access to participant level data, how to share data securely and how decisions will be made collectively. Partnership approaches can be resource intensive require leadership skills to balance equity of decision-making with a strong scientific rationale. Negotiation skills are required to accommodate different perspectives in order to reach consensus in a timely manner. An important limitation to consider is how the partnership approach is interacting with the intervention: for example action research can become an active intervention component (Supplementary File 2, Example 2).

When starting to design a study about migraine, understand how PPI will add value to the research and which uncertainties about patient perspectives might benefit from additional analysis of patient data from a survey or qualitative interviews.
Find out what research questions are priorities for patients Many funders require researchers to justify that their research question addresses what is important to patients 31-33 . If a research question is of low priority to the people affected by the condition, or important outcomes are not considered, and/or the intervention in question is considered unacceptable to patients, then further research is wasteful 11 .
A starting point for researchers is to find out if patients' priorities already exist for their topic. Many national and international organisations involve patients to identify and publish research priorities specific to a healthcare condition. In the UK, the James Lind Alliance (JLA) specifically identifies and prioritises research questions for funders and there is a register 34,35 . JLA establish Priority Setting Partnerships which involve collaborations between patients, carers and clinicians. The NIHR funds JLA advisors and the infrastructure, but a Priority Setting Partnership is responsible for its own funding. The JLA has a guidebook which provides step by step processes to identify research uncertainties and prioritise a top 10 list for different conditions 36 . Researchers are advised to evaluate how priorities were established and the rigour of processes, as priorities can change with time and some groups may not have had an opportunity to be involved.
If patients' priorities are unknown, and a Priority Setting Partnership is not available, contacting the potential funder to discuss options may be helpful. When researchers plan bespoke methods to prioritise research, it is important to find patients as soon as possible to identify the topic and refine the research question to ensure relevance.

Researchers wanting to study migraine may consider the pros and cons of patients doing aspects of the research and the governance issues.
Working together ethically Consider how to work with patients ethically. PPI can be empowering for individuals and communities, but there are tensions and risks, including exploitation 25 , and the burden and resource implications can be considerable 10 . Some ethical principles for researchers to consider when involving patients in research include: -avoiding discrimination, undue persuasion, excessive burden or creating a sense of obligation to be involved in the study -the distribution of power in research -valuing patient contributions and fair financial compensation -conflicts of interest, research integrity and respect for intellectual property -the confidentiality of data and protecting anonymity of research participants -advancing science through honest and accurate reporting.
INVOLVE 3 states that UK ethics committee approval is not required when patients advise research teams, prioritise research questions, make choices relating to design, share decisionmaking or disseminate research findings. However, there can be grey areas particularly in relation to defining 'data collection'. NHS or University Ethics committee approval is required in the UK if personal information, i.e. data as defined in the Data Protection Act 61 , is collected, shared and stored for future analysis and reporting. For iterative partnership research approaches like co-production, the current ethics committee processes create many challenges 62 . Researchers can request informed consent from participants to share anonymised data with patient partners, so that they can be involved in analysis and interpretation as members of the study team.
There are international differences in requirements for research ethical and governance approvals, and particular challenges with digital health research 25 which are beyond the remit of this article. New EU General Data Protection Regulation 63 commenced in May 2018, and requires transparency about the source of personal data, the purpose and who data will be shared with.
Audio-recording of PPI meetings in order to write accurate but not verbatim minutes, does not require ethics committee approval. However, it does require at least verbal consent from all present at the start of the meeting and the recording should be destroyed as soon as the minutes are agreed. People should receive forewarning of the intention to audio-record, know the purpose, what will happen to the recording and to the content, and be able to object or withdraw. If audio-recordings are stored for longer than is necessary, transcribed verbatim or if there is an intention to report or publish potentially identifiable quotations or content arising from PPI activities, then ethics committee approval is required. Ethics committees have lay committee members, who consider the ethical issues relating to patient involvement.

A researcher wanting to study migraine should consider the ethical issues when involving patients in the design and conduct of their study. Consider patient burden, equity and power, fair and respectful arrangements, confidentiality and the purpose, processes and consequences of any data collected or stored.
Involve patients in writing a grant application Patients sit on research prioritisation committees and funding panels, alongside clinicians and academics, to decide which research is commissioned and which grants are awarded. See Supplementary File 2, Example 10. Many UK funding panels expect to read convincing and meaningful accounts of how patients have had an impact at key stages: preparatory work to inform the planned research; writing the application form particularly the lay summary; and the proposed PPI activity during the study. Expect to be challenged if PPI appears tokenistic. It is important to consider the trade-offs between specifying a plan for PPI in a research protocol and building in some flexibility for change as the research progresses. This may be challenging in countries where regulatory approvals for amending protocols is time consuming.
Patients can help researchers to write the whole grant application in an engaging, easy to understand language. The lay summary is often one of the first sections in a grant application that funding committee members read to gain an overview of the study. Reviewers like to understand exactly what study participants will experience from start to finish. Describe PPI clearly so that the reader understands who, why, how many, how often, what methods and what impact patients have already had on the grant application and will have in contributing to future research decisions. For example, decisions about recruitment methods, intervention delivery or components, which outcomes will be primary or secondary and how to collect data. It helps to use language precisely and to understand how involving,

Conclusion
This article provides a starting point for researchers and patient partners who are planning to seek funding for research. There is no current international consensus on best practice or terminology and guidance is evolving across countries and research disciplines. A crucial distinction when gaining patient perspectives is between patient involvement in research and patients participating by providing data in surveys, qualitative interviews or group discussions. The ethical governance implications differ particularly regarding data protection.
Researchers and patient partners can choose a wide range of different approaches to PPI and each study will require consideration of the optimal approach. Rigour is needed because patients' lived experience and persuasive narratives can influence important research decisions and the outcomes are not always predictable. Evidence is needed about how different methods of involving people can improve research decisions, healthcare outcomes and impact. A more collaborative and reciprocal partnership approach with patients has the potential to ensure that research undertaken matters to a wider tranche of society and involves those who stand most to benefit from it.

Key messages
Important questions for researchers about including PPI in their research: • How can I find people in society (patients, patient groups, carers, the taxpaying public, lay organisations) who can make important contributions to research design, conduct and dissemination?
• How will PPI help me to access the perspectives of those who the research potentially will impact on?
• How can different approaches to involving patients as consultants, collaborators or partners improve the relevance, quality, future implementation and sustainability of research?
• How can patients contribute to three key functions: research decision-making; enhance researchers' understanding of different perspectives; and knowledge capture?
• How can PPI, qualitative research and surveys of patient opinion be optimally combined?

Data availability
No data are associated with this article.  68 . PH has been a chair and deputy chair of a research ethics committee. PH is guarantor and affirms that the manuscript is an honest, accurate, and transparent account of the analysis reported.

Competing interests
No competing interests were declared. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. This paper originated from discussions between the authors and observations from our work. In particular observations of conflation between PPI and qualitative research in grant applications and confusion about how they differ. We would like to acknowledge and thank:

Grant information
• Peer reviewers of three earlier versions of this paper, and the editors and editorial boards of a leading peer reviewed UK medical journal. Their comments have helped to improve this article. Post-publication peer review of this article provides them with the option of sharing their comments openly.
• Co-authors from two articles on which this article builds, in particular Heather Morgan, Gill Thomson, Nicola Crossland and PPI groups involved in the BIBS study 47,69 which combined PPI with qualitative research; and the co-authors of a paper considering how qualitative research contributes to feasibility and pilot studies 70 .

Supplementary material
Supplementary File 1: Section A, Additional terminology and international information relevant for patient and public involvement in research at the funding application stage; Section B, Questions asked by UK National Institute for Health Research (NIHR) on funding application forms.
Click here to access the data.
Supplementary File 2: Examples of different approaches to incorporating patient and public perspectives into research design and conduct.
Click here to access the data. Thank you for the opportunity to review this paper. The aim of this paper is to discuss how researchers can involve patients when applying for research funding, and to outline opportunities and pitfalls. It uses a combination of knowledge about this process from literature and the authors' wide ranging experience.

Open Peer Review
This article provides a general background to patient and public involvement in research. The article very usefully includes in Box 1 and Table 1 current resources and guidelines that are available. The information is very comprehensive and relevant to the aims of the article -which is how to to incorporate PPI into the design and conduct of research. This overview is unique and in my view will be widely used. However can it be moved to later in the article? The tables do break up the flow of the paper and break up what I think is an important section about the difference between PPI and patient participation in research.
The section on the difference between patient involvement and patient participation in research is much needed as PPI is often conflated with qualitative research. This is still a common problem. I wondered if Figure 1 could come earlier, or if Box 2 can show the differences more explicitly instead of providing a list of definitions, can the differences be displayed side by side, using maybe column headings of differences and similarities. This will enable the reader to access the important distinctions more easily.
On page 10 in the section about action research, co-design and co-learning there is a sentence about how a partnership approach may interact with an intervention. I think this is the first time intervention studies are mentioned as previous text has stated that patient participation in research may be through interviews, focus groups and surveys. I wondered if this could be expanded to include that these can be undertaken in the context of trials, as then the text would align better with the material in the supplementary documents.
There are a lot of examples in the supplementary files and this can be quite difficult to navigate.
I have an observation on how the steps are characterised. At first I thought the text in italics are the recommended steps for researchers? For example on page 11 the first step for researchers outlined is to search the internet for organisations and guidelines to see if research priorities have been established in their area of interest. There is however some advice given earlier (written in italics) that recommends understanding what PPI will add and which uncertainties about patient perspectives may benefit from more exploration.
Then I realised that the steps are characterised in the blue subheadings. These are a mix of 1. advice Then I realised that the steps are characterised in the blue subheadings. These are a mix of 1. advice (e.g. understand what PPI is (this is a conceptual issue)), 2. questions (e.g. how does PPI differ from patient participation?) and then 3. labels for different issues (e.g. patients doing research, which is a practical issue). Could there be more consistency across the paper? I wondered if it would work if they were all translated into specific statements / recommendations particularly as the aim is to provide guidance on how to incorporate PPI.
Related to the above -I wondered whether the section headings of "how does PPI differ from research participation" and action research, co-design, co-learning and co-production" should be at the same level as the others. They are very important -but I think are fundamentally about understanding what PPI is.
The text in italics is also a mix I think of statements of principles -e.g. seek to balance positive and negative views, rather than a statement of how to do that (the title of the paper is how to incorporate PPI....) The section on how many patients to involve is again really helpful as this is the question that people often ask. This is usually asked in the context of "representativeness" of the patients' views. Can this section be expanded -with some examples of what teams can say when the "representativeness" of their patient advisors is questioned. It links to a later section on bias and conflict of interest.
The authors recommend that researchers select patients with balanced views. It is difficult to know what view people hold at the outset. Can this recommendation again be expanded upon -with ideas about how you may do this, or how you may get fresh perspectives over time.
Towards the end there is a heading about involving patients in writing a grant application. This made me realise that the advice spans both the process of PPI at pre-award and then outlining what to do post award. I wondered therefore whether this section should come earlier. PPI advisors will have an input into the design of the PPI throughout and the research design throughout, and therefore impact on decision making about earlier topics in the article.
I think this is a really valuable paper in providing such a comprehensive overview of resources and guidelines for PPI and in highlighting the key challenges of good PPI practice. My suggestions are really about structuring the content to make it more accessible to the reader by highlighting maybe two aspects. First the need to understand what PPI is and is not. Second the practical issues -how to do it. Thank you for the opportunity to review this manuscript. This article gives an overview of steps research teams and patient partners can take when working together to prepare a research funding application. We believe a guide for inclusive and meaningful involvement of patients and public partners, particularly in the planning stages of research, is a highly useful and timely resource for health researchers. We thought it was appropriate that the authors included a variety of resources from different continents, and appreciated the attempt to summarize the varied (and often confusing) terms and concepts that exist around patient and public involvement. It was great to see involvement of patient partners in authorship of the paper. We also wonder if patient partners have been involved in providing review comments as well.

Are all factual statements correct and adequately supported by citations? Yes
We did struggle with the overall flow of the article, due to its non traditional structure, thereby making it challenging to follow for the reader. Despite the valiant attempt to describe differences in terminology around patient and public involvement in research and key concepts that need to be considered for meaningful patient and public involvement, terms are at times dispersed, seem to be used interchangeably throughout the article, and do not follow a step-by-step order that would be expected from an overview of 'steps' in a process. In addition, many key considerations that should be made prior to engaging patients or members of the public are left to the end of the article (such as training for patients involved in research, or working together ethically). In general, the article operationalizes PPI into a series of steps and although mentioned at some points later in the article, the theory and principles of equity and social justice that are critical to meaningful and inclusive engagement are presented later in the article. However, to ensure researchers are incorporating these principles, we suggest they be introduced sooner and have a more central role as the lens through which PPI is viewed. We feel it would be more beneficial to begin with terminology and key concepts, and then provide an overview of the subsequent steps involved in engagement.
Given that the article appears to be an attempt to synthesize information and provide an overview of the steps to take when involving patients and the public in preparing a research grant application, a clear description of the methods used to gather, sort, and summarize the sources and information contained in the article is missing. Such information is critical for demonstrating the strength and rigor of the recommendations. Although we recognize this is an opinion article that is not intended as a formal synthesis, we feel that including a description of the literature review methods would provide a landmark for others who may seek to undertake a full synthesis in the future.
The tables and boxes included are excellent resources for researchers seeking to learn more about PPI and how it is done in different contexts and regions. However, aside from the definitions, their placement may be better suited to the supplementary material, rather than embedded within the article itself 1 2 1 2 may be better suited to the supplementary material, rather than embedded within the article itself (especially Table 1, as it is in landscape orientation making it difficult to read in digital form).
As we have heard from our stakeholders and patient and public partners, language is a critical aspect of ensuring that PPI is successful, inclusive, and respectful. Many of the decisions that should be made through dialogue with patient partners are presented as decisions for the research team to make and share with patient partners. We recommend directing researchers to dialogue with patient and public partners about topics such as ground rules, preferences for compensation, preferences for feedback, and other matters in which patient partners are involved, in order to ensure that decision-making power in engagement activities is equally shared. A specific example is on page 12 under the heading "Agree appropriate funding for patient involvement", where the authors state that researchers "Negotiate with patients the costs: payment for patient time, any special needs…" and "Researchers are advised to spell out to patients the best case and worst case scenarios…" -we caution to consider carefully how language can challenge or perpetuate power differentials that often tend to place the researcher above the patient as having more expertise and education. We suggest that terms like 'discuss' and 'dialogue', in place of 'negotiate' and 'spell out', can help work towards equality and mutual respect in the research partnership. Some specific points the authors may want to address are detailed below.
partners to meaningfully engage -for example, determining the willingness of the research team to change directions based on patient and public partner input; determining what resources are available to support engagement activities; and determining the underlying purpose for engaging.
Point 3 indicates researchers should identify patients to involve "as early as possible" -this may be unclear to those unfamiliar with the PPI process, and should be more explicit -i.e. engaging before the research questions and methods have been determined -something we do not see mentioned elsewhere in this article.
Under 3.i, it is suggested that researchers consider identifying a "professional or lay link worker", however many research teams will not (for whatever reason) choose to hire an outside professional to assist with engagement activities, and will embark on it themselves. For this reason, we feel it is critical to mention the need for reflexive practice and trauma-informed approaches for those who are not familiar with these approaches and their need when engaging with people with lived experience of health issues, and the potential retraumatization associated with sharing those experiences.
Under 5, there is a minor grammar error -"agree an approach" should read "agree an approach". on Under 6, we believe the use of language such as 'negotiate' may perpetuate power imbalance between research teams and patient and public partners. This language posits researchers and patients on opposing sides, when perhaps what the authors intended to suggest with 'negotiate' was a method of working together to determine appropriate funding and compensation -with which we agree. Perhaps using a term such as 'researchers and patient partners should work together' instead of 'negotiate' would be more appropriate. In relation to this point, it seems that a key component missing from this article's discussion of compensation for patient and public partners is determining the preference of patients and what they want in terms of level of involvement, supports, payment, etc. Similarly, actually asking them under 8, the authors suggest that researchers "consider whether patients or patient groups will 'do' any research" and subsequently assess their skills, ability to add value, status as employees, and mentorship/supervision -we again suggest that these considerations should be made patient and with public partners, rather than decided for them.

Page 10:
From the title of the section "Action research, co-design, co-learning and co-production", it seemed as though the authors would be providing a discussion of these different principles, but instead was a discussion of the differences between definitions that should probably have been included at the beginning of the article. We would suggest moving this information to before the 'steps' of involvement are discussed, and potentially changing the heading since it is somewhat misleading given the content of the section that follows. Additionally, the first sentence discusses the historical context of 'action research', yet this is the only mention of the chronology of terms and seems somewhat out of place. Without discussion of the origin of other terms mentioned in the article, we would suggest that this is unnecessary to the reader's understanding of concepts.
Under "Action research, co-design, co-learning and co-production", there is a minor grammatical error -in the sentence "Equal power in decision-making is sometimes implied, however there are structural and economic power differentials between different types of partner in terms of pay…", the word "partner" should be pluralized to read "partners". the article, particularly regarding the dynamics of relationship-building in research partnerships, where researchers may hold more 'power' and need to engage in reflexive practice to understand how this impacts the process of engagement. We also recommend a discussion of safe spaces, a key consideration for addressing power imbalances in research partnerships. As a reference, we would suggest reviewing the 2017 BMC Health Services Research article by Shimmin et al. .
In the second column on this page, the sentence "Partnership approaches can be resource intensive require leadership skills…" appears to contain a grammatical error and should perhaps read "Partnership approaches can be resource intensive and require leadership skills" or "Partnership approaches can be resource intensive, requiring leadership skills".
The authors mention "negotiation skills are required to accommodate different perspectives in order to reach consensus in a timely manner", however we are not sure that this aligns with the principles of meaningful and inclusive PPI in research -though consensus and timeliness are obviously of importance to researchers, we are not convinced that these are central tenets of engagement, and would argue that respect for patients' stories and lived experiences should be valued over 'timeliness'.
Under "Find out what research questions are priorities for patients", the authors state that "If a research question is of low priority to the people affected by the condition, or important outcomes are not considered, and/or the intervention in question is considered unacceptable to patients, then further research is wasteful". Although we do not disagree, this is a bold statement that we would suggest tempering, given that basic, fundamental research is often of low priority to patients, and may not have a direct intervention for treating disease or improving health, though it often serves as critical foundations for future discoveries.

Figure 1:
The figure demonstrates the people involved in different types of activities. It seem as though the integrated partnership approaches are intended to be positioned as the 'overlap' between PPI and qualitative and survey research. Generally, we found this figure is somewhat difficult to interpret and aesthetically unpleasing. We also question the exclusion of health and social care staff from PPI, and wonder why people in these roles would not be able to participate in PPI.

Page 11:
In determining patient priorities, we are pleased to see mention of JLA, an important methodology. We also think this would be an excellent opportunity to introduce the Patient-Led Research Hub, mentioned later in the article.
The use of survey methodology as an example of identifying patient priorities seems overused -there is a lack of discussion of other potential priority-setting methods (such as patient journey mapping and digital storytelling). This is particularly important for researchers who are new to the concept of PPI who would benefit greatly from information about alternatives to the survey methodology.
Mention of hard-to-reach groups should be mentioned initially under "Identify patients and/or patient groups to involve as early as possible", to ensure it is at the forefront of researchers' minds when considering their recruitment strategy.

Page 12:
Under "Consider the potential for bias and conflicts of interest", we would recommend mentioning that this should be a consideration for discussing ground rules and/or terms of reference. This would also be a good time to mention that the research team and patient partners should discuss together what to do in 3 good time to mention that the research team and patient partners should discuss together what to do in cases of potential conflicts of interest, and how those will be addressed in the context of the research partnership.
Within the heading "Negotiate and agree an approach" there appears to be a word missing, such that the heading should read "Negotiate and agree an approach". Additionally, in the first sentence of the on paragraph under this heading, "Once patients are involved, it is advisable to agree clear boundaries…" should read "Once patients are involved, it is advisable to agree clear boundaries…". upon In discussing the 'scope of the role', it is also important to note that the role of the researcher in the partnership should also be discussed, and this should be indicated in the article.

Sally Crowe
Crowe Associates Ltd, Oxford, UK Cochrane, London, UK Thank you for the opportunity to review this article -which I think is an important mile stone publication in the business of patient and public involvement in the activity that prepares and contributes to research applications. This is a timely and useful review of the literature and the experiences of the research authors, one day we might look back and realise that investing in this stage of research yields benefits and savings in research down the line. I was interested in the reasons for the authors writing this article and was not surprised to see that it was about the conflation of qualitative research and patient and public involvement in research. This article helpfully, and in a practical way helps to 'de couple' the most pertinent issues in this domain. The use of an illustrative example in migraine research helps to bring the review of evidence and discussion in each section to a useful and practical conclusion for the reader, and could be easily re interpreted for their own research context. It is also a strength in this article that research is considered as both primary studies and research synthesis and reviews, both of which require careful patient and public involvement -in the section Page 3 How does patient involvement differ from patient participation in research? "the context and outcomes from listening (to patients) differ. PPI means that researchers are in a continuing and reciprocal relationship with patients and make decisions with them about the research. In qualitative research, researchers listen to patients in order to improve their understanding of a topic". This is probably the most helpful sentence I have read in a while! The further discussions about the choices of researchers that may or may not include qualitative perspectives in their analysis further underlines that in PPI the power dynamic is different and it is an important difference. Figure 1 is helpful I have a problem with the word iterative -and would suggest a plainer language option especially as the rest of the language used in the diagram is of the non-research variety. Table 2 I struggled with this table maybe because I am not a researcher -I would prefer to see the differences between the two rather than the strengths and limitations of each but can appreciate that for researchers making choices this might be very helpful information and analysis.

Boxes 1 and 2
For an article such as this I think that the are more helpful for readers untangling what is contents of Box 2 meant by PPI in pre funded research and -this information is would put Box 1 as a supplementary file more easily found for a curious researcher and Box 2 really adds value to the article as a whole as authors have collected the (sadly) rather large collection of terminology used in PPI and sought to differentiate it. I like the fact that the authors have stated and used their preferred terminology and articulated the reasons for their choice (partnership approaches). Table 1 I imagine this could be immensely useful to researchers and it is a useful comparison tool but positioning in the middle of the narrative is a shame I think as it breaks up the flow of reading.
Find out what research questions are priorities for patients I think that there is a step before initiating priority setting exercises that encompass PPI -increasingly there are published accounts of priority setting that may or may not include the perspectives of patients and the public. There is also an emerging checklist to support the quality assessment of these accounts and specifically the degree of PPI in them Tong A., Sautenet B., Chapman JR., Appraisal checklist used in a systematic review of priority 1,2 1 2 Tong A., Sautenet B., Chapman JR., Appraisal checklist used in a systematic review of priority setting partnerships in health research (Currently being reworked as a priority setting reporting checklist and will be renamed REPRISE Checklist).

Consider equity of opportunity, unheard perspectives and health inequalities
This is a particularly important and pertinent section and offers practical advice and ideas for researchers, especially the use of outreach models that offer more scope for addressing these issues more directly, but may put researchers out of their comfort zones.

Consider the potential for bias and conflicts of interest
This feels an underwritten first paragraph. It feels more important to instigate transparency and declaration of financial and other interests research participants -than the more in-depth for all description of the 'experienced patient?'. Additionally some text about how to manage these conflicts of interests in patients/public may really help readers address this issue.

Agree appropriate funding for patient involvement
There are important considerations here especially around equity of involvement -its good to see these spelled out.
Training for patients involved in research I was very pleased to see this in the text for this section "For patients new to a PPI role, support to develop their abilities and confidence to express their views and question researchers may be relevant." A much under-appreciated aspect of PPI, I would suggest that it doesn't just apply to those new to a role who may find it harder (as am embedded part of the research team) to challenge the research orthodoxy… Also the training needs to be two way -for research teams as well as involved patients and the public.

Key messages Important questions
I like these but I would include a challenging first question to researchers -'Why do they want to do PPI?' I think that there is an important aspect of self-discovery in PPI in research and it helps to understand self/organisational motivations to doing this preparatory work. These reasons may encompass rational or outcome-based reasons (which the current list addresses) but it is also important to understand motivations on a human and relational level.  This article gives a comprehensive overview of the involvement of patients, carers and members of the public in health research. It is well written and addresses some important thorny issues in regards to involvement. The article is likely to be useful for researchers new to involvement. Also, as someone who has been facilitating involvement for some time it is very useful to see all this information pulled together. Partly due to the comprehensiveness, I suggest some of the information in boxes and supplements could be cut down. The sheer volume of information might confuse novice readers.

Are all factual statements correct and adequately supported by citations? Yes
For example, the authors point to the many different and overlapping terms used in this field. For a new author it might be more helpful if the information about all these terms was reduced so that the article focuses on the substantial differences rather than the terms used. I found boxes 1 and 2 quite overwhelming to read through at the beginning of the article. Perhaps it would help if they were supplementary files instead of in-text?
Another example is the definition of 'participating' -I found the introduction to participatory action research here quite confusing because this can be a study design where involvement and participation happen in tandem or are intersected.
Overall, I would have liked to see less of the detailed information on various websites and terms, and more incorporation of the thorny issues, e.g. where research and involvement intersect, and discussion about what we can do about this. It is these aspect of the article that are most interesting, in my view. But the attention to this would depend on the purpose of the article. Some other comments: In Box 3, which gives an overview of the involvement process, I would suggest not using the term bias, because this is commonly used for samples. I agree with the point made -and it is important as often ignored in involvement guidelines -but I think it would be better not to use bias and instead say something about whether the topic of the research is contentious amongst different groups of patients (or something to that effect). Also in Box 3, bullet number 7 mentions roles and responsibilities. I would suggest the agreement of these needs to come much earlier in the process. For example, this could come under point 1 -when the researcher familiarises themselves with involvement. What roles would they like patients, carers or members of the public to have? I really like Tables 1 and 2 -these are super useful overviews. I am not clear what purpose Supplement 1  I really like Tables 1 and 2 -these are super useful overviews. I am not clear what purpose Supplement 1 has, this relates to my previous point on the brevity and detail of information.
In Supplement 2 I find some of the examples lacking in purpose and clarity beyond saying that involvement can happen. I don't find that most of the examples provide new information that isn't available elsewhere in many different formats. However, some examples are very interesting and could merit more space. These are: Example 2 about the cautionary tale -this is a new point that I have not heard before; Example 6 which gives a very interesting example of when qualitative research and involvement overlap (again, would merit more discussion); Example 9 which is very brief but points to how protectionist policies can exclude people from participating (a common reason for not involving children, young people, people with disabilities, the frail elderly and other vulnerable populations) -I don't think this has been considered in-depth by policies intended to increase participation; Example 10 is good on details on how researchers can work collaboratively with patients/carers/members of the public and will be of interest to people looking for new involvement ideas.
You describe patients as primarily fulfilling three functions when they are involved in research: research decision-making, enhancing the patient perspective, how to capture knowledge. I suspect many involved patients/public advisors will object to this as being too limiting. In my own work I have seen at least three additional kinds of input: 1) public advisors helping researchers plan involvement in their research, 2) public advisors helping with dissemination and collaboration strategies (recently a public partner pointed out that the researchers had not asked for a letter of support to a very key national charity which could really help with dissemination), and 3) what I'd call 'hidden or obvious talents'. Examples of the latter are patients drawing on their previous careers or hobbies, or other talents for seeing new aspects of an established research method. A parent carer we work with decided to use VideoScribe to disseminate some research she had initiated, she then presented about this software at a research seminar, and the research programme bought the software as a direct result of hearing her talk about it. In this example the function of the involved parent was to influence the dissemination and communication strategy of a whole programme of work.
I hope these comments are useful. A helpful article. A few recommendations that will help improve the accuracy and help avoid any confusion.
One step that is missing I think is 'Preparing the research team for PPI'. Support and training are mentioned in relation to the public but this also applies to the research team. p1. 'The research methods of PPI...' I don't think 'methods' is the right word (eg 'informal discussions' is not a research method). Perhaps 'techniques' or 'The ways in which patients and public are involved in research...' p3. I think the sentence in the first para beginning 'In the UK, INVOLVE states...' should come at the end of that para.
p3. 'Steps for how etc'. Are these steps? Or are they 'Issues to consider'? p3. 'Understand what patient and public involvement is'. A more accurate heading would be 'Understand your rationale for patient and public involvement.' And in the next two sentences you mention 'theory'. I don't think these are theories -I think it is about 'understanding the rationale or motivation for patient and public involvement.' p3 'How does patient involvement differ from patient participation?' The only step that is posed as a question -I'd change to 'Understanding how patient involvement differs etc'. And it would be worth adding in again to this section (I know it's already in elsewhere) INVOLVE's distinction between involvement and participation.
p7 The list of acronyms includes both terminology and organisations. Confusing. Take out the organisations -if they're in the main body of the article then they should be written in full anyway. p11 'A first step....'. I'd reword to make less like an instruction and consistent with the rest of the article. So 'A researcher could search the internet etc. Another approach would be to contact migraine charities etc.' p11 Para beginning 'Lay or professional coordinators or link people...' Consider explaining what these terms mean. And replace 'whereas' with 'and'. Lose the sentence 'The qualitative research and PPI become synergistic' -it confuses the point being made in this section and I'm nort sure it's accurate. p11 'Decide who and how many patients to involve' -we've moved from people to patients. Needs to be 1,2 1 2 p11 'Decide who and how many patients to involve' -we've moved from people to patients. Needs to be consistent throughout. Also add in something about why you might want to consider having more than one person ie a) public can support each other b) helps redress the power balance in the room and c) the public can not always make a meeting and so, if you have more than one person, it reduced the likelihood that the public voice will be absent at any given time.
p11 Replace the sentence 'Aim to find patients who represent the demogaphics etc' with 'Consider the demographic of those affected etc'. Some readers will take the first sentence to the extreme and it may become a barrier to involving people. I would also suggest losing the sentence 'It can be challenging to access 'typical' members etc' -I'm not sure what is meant by 'typical' here. Need to give some consideration here to the issues of 'representativeness' -when you have only one or two people involved in your research it is unlikely that can be 'the' voice of everyone but they can be 'a' voice. If you want something more representative then surveys etc might be a more appropriate answer.
p11 The authors say that 'the more confident and financially secure are more likely to volunteer'. Need to add in something about researchers have struggled to access certain groups. The sentence 'A lack of resources' -not sure that 'less privileged' is a phrase I would use . Perhaps 'less well off' or something similar? p11 Need to be careful with the sentence 'Yet this is important..' -some ethnic minority groups might be offended that you are asserting that they tend' to experience lower health status' . Perhaps 'some groups tend to experience etc' p12 'Patients can contribute to three key functions etc'. Lose this sentence. I'm not sure that it's true. For example they can also be included in data collection (also applies to the penultimate bullet point in 'key messages' on page 15).
p12 'AT an early stage a researcher is advised to discuss roles and task etc' -I would also add in here behaviours or responsibilities.
p13 'INVOLVE has a Patient-Led-Research-Hub to support patients who want to puruse their own research.' Are you referring to INVODirect? This is more a list of organisations who support active PPI in research.
p13 'Researchers wanting to study migraine etc'. Add on to the end of the sentence 'this entails.' (I read it at first as the patients doing aspects of the governance issues).
p15 Either lose the final bullet point on the key messages -I found this confusing -or add in something like 'to ensure you get the public view'.

Are arguments sufficiently supported by evidence from the published literature? Yes
Are the conclusions drawn balanced and justified on the basis of the presented arguments?