Writing a Reproducible and Replicable Literature Review

What we hope you'll find in this article is that a literature review of any size can reap the benefits of legitimacy and clarity that come from documenting your research methods. Creating reproducible methods and replicable results benefits undergraduate students on their first research paper as well as systematic reviewers on their 100th review.

A reproducible literature review means that if another researcher wanted to write a literature review with the same substance, they would be able to reproduce your methods. Because the total body of academic literature is expanding quickly, reproducing the exact results will be difficult as more papers are consistently published (Cram et al. 2020). What we can guarantee in our literature reviews is using a searching, screening, and extraction strategy that will hold up with other researchers conducting the same review.

Reproducibility differs from replicability. Replicability is how well a researcher could test the same method to obtain results that may agree or contradict the study. A replicable study will have results that are clearly derived from the methods used. A reproducible study will have a methodology that can be clearly redone.

How do we write a literature review that is both reproducible and replicable? We start with the search for literature. From the beginning of the literature review, the search terms and database should be documented so that anyone conducting a research search will derive similar articles to this review.

An example of a literature search using an advanced (boolean) search demonstrates that it could be reproduced using the OpenAlex database and the specific keywords. In this example, only one database was used, so there was no deduplication necessary:

Searches were run in the OpenAlex database. The following keywords were searched for titles and abstracts 'pond pH change with depth' OR 'lake pH change with depth', and 'Electrical Conductivity' OR 'Specific Conductance'. A total of 168 articles were retrieved and after deduplication 168 articles were screened.

The next step is to create a methodological screening procedure and document it accurately. The screening stage is where we can quickly determine which articles have full text that we want to analyze.

If you're doing this process entirely by hand, you would find similar papers through the citations found as you're reading the full papers. For very large systematic reviews, you may choose not to add similar papers because of how comprehensive your boolean search is.

Documenting the initial search and screening to show how we made these decisions looks like this:

The following keywords were searched for titles and abstracts 'pond pH change with depth' OR 'lake pH change with depth', and 'Electrical Conductivity' OR 'Specific Conductance'. A total of 168 articles were retrieved and after deduplication 168 articles were screened. The search terms 'depth' and 'pH' were used across the full text and 50 thematically similar papers [using the FoundThat software]. 41 articles containing these terms were selected for this review (36 from the initial search, 5 from FoundThat's similar papers).

Reproducible data extraction is where researchers can hit a wall. Your method for extraction is reproducible if you can point to your criteria for selecting data and you can ensure you pulled data that was accurate to what the article is saying.

If you're doing the data extraction by hand, the Cochrane group recommends you have a second reviewer check all data. A study by Smela et al. found that having a second reviewer was more important for verifying the quantitative data rather than the article description (Smela et al. 2023). You can keep extraction reproducible by using non-AI tools like FoundThat for the quantitative data extraction from every article's tables at once. If doing the work by hand, best practice is to have a second reviewer ensure you're extracting information accurate to each article.

In conclusion, writing a reproducible and replicable literature review is something that can be achieved through intentional documentation and the use of software products that are deterministic. Adding, or tightening, the precise methods used in a literature review adds credibility to everything you're writing.

This article was written using the FoundThat research software. We searched Google Scholar for "reproducibility" and "literature review"; 3 papers were chosen from title and abstract screening from the top 10 results. The search terms "literature review" and "reproducibility" were used across the full text and 50 thematically similar papers [using the FoundThat software]. 7 articles containing these terms were selected for this review (3 from the initial search, 4 from FoundThat's similar papers).

• Smela, B., Toumi, M., Świerk, K., Francois, C., Biernikiewicz, M., Clay, E., & Boyer, L. (2023). Rapid Literature Review: Definition and Methodology. Journal of Market Access & Health Policy, 11(1), 2241234. https://doi.org/10.1080/20016689.2023.2241234
• Cram, W. A., Templier, M., & Pare, G. (2020). (Re)considering the Concept of Literature Review Reproducibility. Journal of the Association for Information Systems, 21(5). https://doi.org/10.17705/1jais.00630