Open Protocols

Hello and welcome to this short talk on open protocols. We’ll begin by reviewing what “research protocols” are, and how sharing them — especially before you carry out a research study — is an open research practice that contributes to both transparency and research quality.

What are open protocols?

A protocol is basically a recipe for carrying out a research project, or, to put that more formally, an “explanation of all the aspects of a future research project given in a precise, understandable manner.” An open protocol, accordingly, is a research methodology that has been shared in such a way that other researchers can easily replicate the exact procedures that it describes — and, at least potentially, arrive at similar results. As such, open protocols are not only linked to the concept of “openness” per se, but to another aspect of good research: “replicability”.

The difference between method and documentary

Crucially, a protocol is not a play-by-play description of what happened in a research project, but a methodology that is written down in advance. The protocol will likely talk about the overall shape of the data to be gathered, and may include a data management plan. However, due to variability in contextual factors, simply having a protocol in place does not guarantee that the results of running two studies with the same methods will actually be the same: “methods replicability” is necessary but not sufficient for “results replicability”. A good test of the readiness of a protocol is whether or not it could be followed by someone who wasn’t previously familiar with the research project — indeed, when it has reached that level of detail and accuracy, it may well be ready to publish as a stand-alone document. We’ll talk more about that shortly.

Associated practices of pre-registration and registered reports

Assuming you’ve created a protocol in advance of your study (which is in and of itself a good practice — because it constitutes a logical check on how you expect the project to develop), you then have to decide where to put it. One option is to publish it before you actually carry out the work it describes. Preregistration is the practice of specifying your research plan and registering it to a public repository in advance of undertaking your study. Registered Reports are a further evolution of this practice. A registered report is a form of journal article in which peer review of the study protocol and the (in-principle) decision to publish the results occur before the study is run (this is called “in principle acceptance” or “IPA”). In this case, the research is submitted in two stages: one part is pre-registered and reviewed before the research is carried out, and the other submitted after the study has completed. Reviewers will check whether the protocol was actually followed. Journals have different approaches as to when the Stage 1 material is published.

Why do this?

Of course, you’re free to write up your methods, implement them, and only publish them after the results are in. One reason not to do things in that order — and also a reason for all formality around Registered Reports — is to avoid any possibility of “HARKing” or Hypothesizing After the Results are Known. Another benefit is that sharing your methods early on can allow them to be scrutinised and improved by others. While there’s a chance that someone could “scoop” your methods and carry out the data gathering before you have a chance to complete your study, preregistration itself is evidence of the priority of an investigator’s hypotheses. (Furthermore, in the case of Registered Reports, publication is guaranteed no matter how many similar papers are published between the two stages.)

Working with open methods

Not all fields or research projects lend themselves to the full formality of preregistration, however, it’s worth mentioning that it may be possible to share your “methods” openly, even when the “data” you gather cannot be shared (for whatever reason). Aspects of your process may be both rigorous and relatively easy for others to adopt — for example, a structured debrief protocol that you might prepare in advance of carrying out field work — even if other aspects of your methodology are deeply inductive or interpretive.

Initial meta-research suggests that preregistration helps to counteract the bias towards publishing primarily “positive” and “statistically significant” findings, i.e., findings that appear to confirm the researcher’s hypothesis. Registered reports may provide the antidote to other “Questionable Research Practices”. Here’s one example. A 2021 paper found that non-replicable papers tend to be cited more frequently than replicable papers. As the authors point out, registered reports could help reverse this rather discouraging trend: this is because registered reports are reviewed before data collection has taken place, and so they will not be accepted simply because they happen to include some remarkable (but in the end non-replicable) finding.

One place you can find examples is on the “Peer Community In” platform (https://rr.peercommunityin.org/). PCI is an open peer reviewing service that both runs its own multi-disciplinary journal, and also maintains relationships with other journals that are willing to accept PCI-reviewed registered reports. In various examples hosted on the platform, you’ll see that “Stage 1” of a Registered Report can include multiple rounds of reviewing. For example, the review of “Voice preferences across contrasting singing and speaking styles” took four rounds, spanned 15 months and “resulted in largely a new manuscript.” After getting familiar with the medium, another practical step could be to find a protocol describing a project you’re interested in, and carry out a replication study, possibly working together with colleagues or students. Have a look at “Ten simple rules for designing and conducting undergraduate replication projects” for further ideas.

Link

• Questionable Research Practices: Definition, Detection, and Recommendations for Better Practices

References

Serra-Garcia, M., & Gneezy, U. (2021). Nonreplicable publications are cited more than replicable ones. In Science Advances (Vol. 7, Issue 21). American Association for the Advancement of Science (AAAS). https://doi.org/10.1126/sciadv.abd1705

Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports. In Advances in Methods and Practices in Psychological Science (Vol. 4, Issue 2, p. 251524592110074). SAGE Publications. https://doi.org/10.1177/25152459211007467

Moreau, D., & Wiebels, K. (2023). Ten simple rules for designing and conducting undergraduate replication projects. In R. Schwartz (Ed.), PLOS Computational Biology (Vol. 19, Issue 3, p. e1010957). Public Library of Science (PLoS). https://doi.org/10.1371/journal.pcbi.1010957

Henderson, E. L., & Chambers, C. D. (2022). Ten simple rules for writing a Registered Report. In PLOS Computational Biology (Vol. 18, Issue 10, p. e1010571). Public Library of Science (PLoS). https://doi.org/10.1371/journal.pcbi.1010571