How We Work

Section 4

Section 4

Writing and registering a Protocol

 Last updated September 27th, 2017

4.1 Purpose of the Protocol

CEE Evidence Syntheses require the publication of their Protocol (project plan) as an independent document, before the synthesis is conducted. There are several reasons for doing so:

  • Data are not evidence unless accompanied by a Protocol and analysis.
  • Within the CEE approach the Protocol acts as a formal registration of intent by the Review Team to conduct a CEE Evidence Synthesis on a given topic. It allows CEE to inform the scientific community of this project, and to let the Review Team know about any prior similar projects, or ongoing ones.
  • The Protocol acts as an a priori guide and reference to the conduct of the synthesis that reflects views of stakeholders and that the Review Team and their commissioners agreed upon during the planning stage (including the scoping exercise).
  • The Protocol is essential to minimise reviewer bias (e.g. resulting from ad-hoc decisions made or ‘mission creep’ during the synthesis process) and make the review process as rigorous, transparent, and well-defined as possible.
  • The Protocol enables explicit and compulsory recording of any change that may occur during the conduct of the synthesis that would not have been foreseen. This is particularly important to ensure the confidence of the consumers, commissioners and stakeholders about the reasons for changes.

IMPORTANT: COMMITMENT TO REGISTER AND PUBLISH WITH CEE. By registering and publishing your Protocol with CEE you are registering your intent to conduct, and submit to for publication, a CEE Systematic Review/Map. You will be asked to confirm that you and your co-authors are aware of and agree with this commitment when you submit your Protocol for publication in Environmental Evidence.

4.2 Developing and writing a Protocol

 The Protocol’s background section should present the problem being addressed and the rationale for why a Systematic Review or Systematic Map is required. Where possible, a ‘theory of change’ or conceptual model should be presented that explains the process(es) whereby the intervention or exposure factor is thought to have an impact or cause a change in the subject population (see www.theoryofchange.org/what-is-theory-of-change/). In more complex situations a proposed causal chain, linking intervention(s) to outcome(s), may be helpful. The structure of an Evidence Synthesis Protocol mirrors the structure of the Systematic Review or Systematic Map that it guides. Beside a formal presentation of the question and its background (the “real world” context), a Protocol sets out (informed by the scoping process – see above) the strategy for searching for relevant studies and defines eligibility criteria for article screening. The question elements defined in the question formulation stage provide the a priori inclusion criteria important for the objectivity and transparency of the synthesis. They should also lead to a description of the kinds of evidence (e.g. study designs) that you would consider valid to include in the synthesis. An Evidence Synthesis Protocol should also detail, with rationale, the likely methods to be used for eligibility screening, data coding/extraction, critical appraisal (Systematic Review only), and data synthesis, and state any conflicts of interest including details of any funding sources.

Since the Protocol sets out what the synthesis aims to achieve, it is useful for getting the engagement of experts who may have data to contribute. Anyone reading the Protocol should clearly understand the nature of the question and what type of evidence will inform it. Registering and posting of Protocols on the CEE website provides transparency and also acts as a record of which syntheses are in progress, enabling others to see if a synthesis is being conducted that may be of interest to them, or to prevent the initiation of a synthesis on a topic that is already underway. An example of Protocol development is given in Box 4.1. For examples of recently completed Protocols, visit the Environmental Evidence Library at: http://www.environmentalevidence.org/reviews-in-progress.

Once an Evidence Synthesis Protocol has been peer-reviewed and published as final, changes are discouraged. However, it may become necessary during the course of an evidence synthesis to make revisions because of deviations from the proposed methods. These changes should be clearly documented within the final synthesis report so that transparency and repeatability can be maintained. If a major change is necessary to a Protocol part-way through a Systematic Review or Systematic Map (e.g. change of question or major change in scope) then the Protocol should be updated in consultation with CEE, and the change should then be applied to all references, or studies, as appropriate, to avoid introducing bias. The final Evidence Synthesis report should explicitly state how the final Systematic Review or Map methods differed from the Protocol.

Protocols are plans of conduct and can rarely be fully comprehensive. They are judged in this context during the CEE peer review process. Consequently, the acceptance and publication by CEE of a synthesis Protocol does not guarantee acceptance of the resulting synthesis report. Problems with the latter may occur due to conduct that was not mentioned or not fully transparent in the Protocol.

As a general rule, a Protocol should set out the plan for a single report (a Systematic review or a Systematic map). Exceptionally, where there is a strong logical case made, a single Protocol may set out a plan for multiple reports. This should be anticipated and fully explained in the Protocol and should not be a post-hoc decision. Whether the multiple report route is permissible will be decided by the CEE Editorial Board and is a trade-off between the efficiency provided by the publication of one Protocol and the legitimacy and feasibility of combining several reviews or maps together.

Box 4.1 Example of Protocol Development

 

 

The following sections set out the expected format, reporting and standards to be met by CEE Evidence Synthesis Protocols.

4.3 Systematic Review Protocol template

4.3.1 General criteria

Protocols set out the plans for the conduct of CEE Systematic Reviews. They are a requirement of CEE registration to conduct a CEE Systematic Review and a Protocol should be submitted in advance of conducting the review.

Protocols should normally be no longer than 8000 words (including references but excluding appendices and additional files).

4.3.2 Title page and Abstract 

The title page should state the anticipated title of the Systematic Review and the title should accurately reflect the review question. The title should also indicate the type of Evidence Review and the fact that it is a Protocol, for example: ‘What is the effectiveness of intervention A in producing change in subject B? A systematic review Protocol’.

The full names, institutional addresses, and email addresses for all authors must be included on the title page. The corresponding author should also be indicated.

The abstract should not exceed 350 words and must be structured into separate sections:

Background – the context and purpose of the review, including the review question; Methods – how the review will be conducted and the outputs that are expected. All stages of the methods should be briefly described here, including searching, eligibility screening, data extraction, synthesis and presentation.

Three to ten keywords representing the main content of the article should be given. Keywords should not repeat terms mentioned in the title. Keywords may be organised alphabetically. 

4.3.3 Main text

Background

This section should be written in a way that is accessible to readers without specialist knowledge in that area and must clearly state – and, if helpful, illustrate – the background to the review and its aims. You should indicate why this study is necessary and what it aims to contribute to the field. If possible, a theory of change and/or conceptual model should be presented that links the key elements of the question (e.g. explaining how an intervention or exposure is linked to the outcome). The role of commissioners and other stakeholders in the formulation of the question should be described and explained. Any proposed role of stakeholders throughout the review process should also be described. It should logically lead the reader to the primary question stated in the next section.

Objective of the Systematic Review

You should describe the primary question and any secondary questions when applicable. The primary question is the main question of the Systematic Review and should be the same or very similar to the Protocol title. The rationale for any secondary questions (usually linked to possible subgroup and/or heterogeneity analyses) should be explained.

Methods
Searches

Here the proposed searches are described in sufficient detail so as to be repeatable. Details of how the search strategy was developed through scoping should be provided (possibly in an additional file). The following list is a guide to the detail required on what will be searched and how the search will be conducted.

  • Search terms and languages
  • Search strings and/or combinations of searches (search strings refer to combinations of terms using Boolean characters, combinations are methods used to set-up and pool different searches run separately).
  • Estimating the comprehensiveness of the search
  • Bibliographic databases and services to be searched (e.g. ISI Web of Science, Scopus, CAB Abstracts)
  • Internet searches to be conducted (e.g. Google Scholar)
  • Specialist searches – Searches for grey literature: contacts, searches of organisational websites, use of specific search terms or strings, filtering or limitations.
  • Supplementary searches such as Bibliographical searches and literature provided directly by stakeholders
  • Any software that will be used to conduct and record your search should be described.
Eligibility criteria and screening process

State here the criteria you will use to decide whether articles and the studies they report are eligible for inclusion in the review. Normally you should use the following subheadings;

  • Eligible population(s)
  • Eligible intervention(s) or exposure(s)
  • Eligible comparator(s) (if appropriate)
  • Eligible outcomes
  • Eligible study designs

If a worksheet or checklist is to be used for the eligibility screening process this should be provided in an appendix.

State here the article screening process in terms of

  • Title, abstract and full text screening methodology and how you will record and report the process from number of articles considered to number of studies included. Please note here the distinction between articles and studies (articles can report on more than one study and studies can be reported in more than one article).
  • Test(s) for consistency of decisions regarding eligibility, at title, abstract, full-text level.

Describe how the Review Team members who have also authored articles to be considered within the review will be prevented from influencing decisions regarding inclusion or critical appraisal of their own work.

Please state clearly what you will provide in your report to conform with CEE Standards (e.g. a list of studies excluded at the full text screening step with reasons for exclusion).

Study critical appraisal

Describe here the approach you propose to use to critically appraise and assess internal and external validity of included studies. Consideration should be given to ensuring that important aspects of validity which would influence the reliability of the evidence are taken into account, such as internal validity (freedom from bias or confounding) and external validity (generalisability of a study’s results to the current Systematic Review question). If a worksheet or checklist is used for the assessing study validity this should be provided in a table or an appendix. Describe how the information from critical appraisal will be used in any synthesis. Describe how repeatability of individual reviewer critical appraisal of study validity will be tested.

Data extraction strategy

Describe here how you will collect and record outcome data and associated meta-data from included studies. How will the repeatability of coding, meta-data and data extraction be tested? Please state that your extracted data records will be made available as additional files. Describe any processes for obtaining and confirming missing or unclear information or data from authors.

Potential effect modifiers and reasons for heterogeneity

Provide a list of those effect modifiers to be considered and coded in the review and details of how the list was compiled (including consultation of external experts). The list should not be exhaustive but should state those variables thought to be most important and amenable to analysis.

Data synthesis and presentation

Describe here the methods you propose to synthesise the collected data and any subsequent manipulation of the data set, such as sub-group or sensitivity analyses. Narrative synthesis should always be attempted and included studies described and tabulated as a minimum. The potential for meta-analysis (or any alternative form of synthesis such as qualitative or mixed methods) should have been assessed and the preferred synthesis methodology should be stated in sufficient detail for reviewers to comment. Appropriate subgroup analyses, sensitivity analyses and tests for publication bias should be planned where possible.

4.3.4 Declarations

Potential Conflicts of Interest and Sources of Support

The independence of CEE Systematic Reviews is a high priority. Consequently, the potential competing interests of the Review Team should be transparent. A competing interest exists when your interpretation of data or presentation of information may be influenced by your personal or financial relationship with other people or organizations. Authors must disclose any financial competing interests; they should also reveal any non-financial competing interests that may cause them embarrassment were they to become public after the publication of the manuscript. Authors are required to complete a declaration of competing interests. All competing interests that are declared will be listed at the end of published articles. Where an author gives no competing interests, the listing will read

‘The author(s) declare that they have no competing interests’. When completing your declaration, please consider the following questions:

Financial competing interests

In the past five years have you received reimbursements, fees, funding, or salary from an organization that may in any way gain or lose financially from the publication of this manuscript, either now or in the future? Is such an organization financing this manuscript (including the article-processing charge)? If so, please specify. Do you hold any stocks or shares in an organization that may in any way gain or lose financially from the publication of this manuscript, either now or in the future? If so, please specify. Do you hold or are you currently applying for any patents relating to the content of the manuscript? Have you received reimbursements, fees, funding, or salary from an organization that holds or has applied for patents relating to the content of the manuscript? If so, please specify.

Do you have any other financial competing interests? If so, please specify.

Non-financial competing interests

Are there any non-financial competing interests (political, personal, religious, ideological, academic, intellectual, commercial or any other) to declare in relation to this manuscript? If so, please specify

Funding

All sources of funding for the research reported should be declared. The role of the funding body in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript should be declared.

Authors’ contributions

The individual contributions of authors to the manuscript should be specified in this section. Guidance and criteria for authorship can be found in our editorial policies.

Please use initials to refer to each author’s contribution in this section, for example: “FC analyzed and interpreted the data regarding the hematological disease and the transplant. RH performed the histological examination of the kidney, and was a major contributor in writing the manuscript. All authors read and approved the final manuscript.”

Acknowledgements

Please acknowledge anyone who contributed towards the article who does not meet the criteria for authorship, including anyone who provided professional writing services or materials.

Authors should obtain permission to acknowledge from all those mentioned in the Acknowledgements section. See our journal editorial policies for a full explanation of acknowledgements and authorship criteria. If you do not have anyone to acknowledge, please write “Not applicable” in this section.

Authors’ information

This section is optional. You may choose to use this section to include any relevant information about the author(s) that may aid the reader’s interpretation of the article, and understand the standpoint of the author(s). This may include details about the authors’ qualifications, current positions they hold at institutions or societies, or any other relevant background information. Please refer to authors using their initials. Note this section should not be used to describe any competing interests.

4.3.5 References

 All references cited in the above sections should be listed here and formatted according to journal instructions.

 4.3.6 Figures, Tables and Additional Files

See the Journal’s general formatting guidelines for information on how to format figures, tables and additional files.

4.4 Systematic Map Protocol template

4.4.1 General criteria

Protocols set out the plans for the conduct of CEE Systematic Maps. They are a requirement of CEE registration to conduct a CEE Systematic Map and a Protocol should be submitted in advance of conducting the mapping.

Protocols should normally be no longer than 8000 words (including references but excluding appendices and additional files).

4.4.2 Title page and Abstract 

The title page should state the anticipated title of the Systematic Map and the title should accurately reflect the map question. The title should also indicate the type of Evidence Synthesis and the fact that it is a Protocol, for example: ‘What evidence is available on the effectiveness of interventions of type A in producing change in subjects of type B? A systematic map Protocol.’

The full names, institutional addresses, and email addresses for all authors must be included on the title page. The corresponding author should also be indicated.

The abstract should not exceed 350 words and must be structured into separate sections:

Background – the context and purpose of the map, including the review question; Methods – how the mapping will be conducted and the outputs that are expected. All stages of the methods should be briefly described here, including searching, eligibility screening, meta-data extraction and coding, study mapping and presentation.

Three to ten keywords representing the main content of the article should be given. Keywords should not repeat terms mentioned in the title. Keywords may be organised alphabetically.

4.4.3 Main text

Background

This section should be written in a way that is accessible to readers without specialist knowledge in that area and must clearly state – and, if helpful, illustrate – the background to the map and its aims. You should indicate why this study is necessary and what it aims to contribute to the field. If possible, a theory of change and/or conceptual model should be presented that links the key elements of the question (e.g. explaining how multiple interventions or exposures are potentially linked to the outcome). The role of commissioners and other stakeholders in the formulation of the question and review should be described and explained. Any proposed role of stakeholders throughout the review process should also be described. It should logically lead the reader to the primary question stated in the next section.

Objective of the Systematic Map

You should describe the primary question and any secondary questions when applicable. The primary question is the main question of the Systematic Map and should be the same or very similar to the Protocol title. The rationale for any secondary questions (usually linked to possible subgroup analyses) should be explained.

Methods
Searches

Here the proposed searches should be described in sufficient detail so as to be repeatable. Details of how the search strategy was developed through scoping should be provided (possibly in an additional file). The following subsections are a guide to the detail required on what will be searched and how the search will be conducted.

  • Search terms and languages
  • Search strings and/or combinations of searches (search strings refer to combinations of terms using Boolean characters, combinations are methods used to set-up and pool different searches run separately).
  • Estimating the comprehensiveness of the search
  • Bibliographic databases and services to be searched (e.g. ISI Web of Science, Scopus, CAB Abstracts)
  • Internet searches to be conducted (e.g. Google Scholar)
  • Specialist searches – Searches for grey literature: contacts, searches of organisational websites, use of specific search terms or strings, filtering or limitations.
  • Supplementary searches such as bibliographic searches and literature provided directly by stakeholders

Any software that will be used to conduct and record your search should be described.

Eligibility criteria and screening process

State here the criteria you will use to decide whether articles and the studies they report are eligible for inclusion in the Systematic Map. Normally you should use the following subheadings;

  • Eligible population(s)
  • Eligible intervention(s) or exposure(s)
  • Eligible comparator(s) (if appropriate)
  • Eligible outcomes
  • Eligible study designs

If a worksheet or checklist is to be used for the eligibility screening process this should be provided in an appendix.

State here the article screening process in terms of

  • Title, abstract and full text screening methodology and how you will record and report the process from number of articles considered to number of studies included. Please note here the distinction between articles and studies (articles can report on more than one study and studies can be reported in more than one article).
  • Test(s) for consistency of decision regarding eligibility, at title, abstract, full-text level.

Describe how the Review Team members who have also authored articles to be considered within the map will be prevented from influencing decisions regarding inclusion or critical appraisal of their own work.

Please state clearly what you will provide in your report to conform with CEE Standards (e.g. a list of studies excluded at the full text screening step with reasons for exclusion).

Data Coding Strategy

Describe here the data that you intend to extract from each study and how they are to be recorded as an element of the map. How will the repeatability of this process be tested? Please state that your extracted data records will be made available as additional files. Describe any processes for obtaining and confirming missing or unclear information or data from authors. If a worksheet or checklist is used for the data coding this should be provided in an appendix.

A full critical appraisal of studies is not required for a Systematic Map but some preliminary assessment of study validity can be informative regarding the reliability of the evidence base. If any aspects of study validity are planned (e.g. basic coding of certain aspects of internal validity such as study design) this should be explained in the data coding strategy part of the Protocol.

Study mapping and presentation

Describe here the methods you propose to use to map and present the collected studies and the data they contain. You should be explicit about the form in which you will make the map and the coded data available.

4.4.4 Declarations

Potential Conflicts of Interest and Sources of Support

The independence of CEE Systematic Maps is a high priority. Consequently, the potential competing interests of the Review Team should be transparent. A competing interest exists when your interpretation of data or presentation of information may be influenced by your personal or financial relationship with other people or organizations. Authors must disclose any financial competing interests; they should also reveal any non-financial competing interests that may cause them embarrassment were they to become public after the publication of the manuscript. Authors are required to complete a declaration of competing interests. All competing interests that are declared will be listed at the end of published articles. Where an author gives no competing interests, the listing will read

‘The author(s) declare that they have no competing interests’. When completing your declaration, please consider the following questions:

Financial competing interests

In the past five years have you received reimbursements, fees, funding, or salary from an organization that may in any way gain or lose financially from the publication of this manuscript, either now or in the future? Is such an organization financing this manuscript (including the article-processing charge)? If so, please specify. Do you hold any stocks or shares in an organization that may in any way gain or lose financially from the publication of this manuscript, either now or in the future? If so, please specify. Do you hold or are you currently applying for any patents relating to the content of the manuscript? Have you received reimbursements, fees, funding, or salary from an organization that holds or has applied for patents relating to the content of the manuscript? If so, please specify.

Do you have any other financial competing interests? If so, please specify.

Non-financial competing interests

Are there any non-financial competing interests (political, personal, religious, ideological, academic, intellectual, commercial or any other) to declare in relation to this manuscript? If so, please specify

Funding

All sources of funding for the research reported should be declared. The role of the funding body in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript should be declared.

Authors’ contributions

The individual contributions of authors to the manuscript should be specified in this section. Guidance and criteria for authorship can be found in our editorial policies.

Please use initials to refer to each author’s contribution in this section, for example: “FC analyzed and interpreted the data regarding the hematological disease and the transplant. RH performed the histological examination of the kidney, and was a major contributor in writing the manuscript. All authors read and approved the final manuscript.”

Acknowledgements

Please acknowledge anyone who contributed towards the article who does not meet the criteria for authorship, including anyone who provided professional writing services or materials.

Authors should obtain permission to acknowledge from all those mentioned in the Acknowledgements section. See our journal editorial policies for a full explanation of acknowledgements and authorship criteria. If you do not have anyone to acknowledge, please write “Not applicable” in this section.

Authors’ information

This section is optional. You may choose to use this section to include any relevant information about the author(s) that may aid the reader’s interpretation of the article, and understand the standpoint of the author(s). This may include details about the authors’ qualifications, current positions they hold at institutions or societies, or any other relevant background information. Please refer to authors using their initials. Note this section should not be used to describe any competing interests.

4.3.5 References

 All references cited in the above sections should be listed here and formatted according to journal instructions.

4.3.6 Figures, Tables and Additional Files

See Journal General formatting guidelines for information on how to format figures, tables and additional files.

 

Continue to Section 5 – Conducting a Search

Follow Us On Twitter Connect On Linkedin Make A Donation