đ R2S2 Knowledge Base
Reproducible Research Support Service (R2S2) Knowledge Base
The R2S2 was a temporary pilot conducted in collaboration by the project Opening Reproducible Research (https://o2r.info/) and the University and State Library of MĂźnster (ULB).
Currently, there are no consultations possible.
To learn more about the concept and offerings of the R2S2, take a look a the R2S2 page.
Reproducible research in general
This page is a growing list of potentially useful material if you want to become familiar with the topic of reproducible computational research.
Start with the basic topics at the top and find more detailed, special, and advanced content further below.
If youâre looking for guidelines or surveys for your discipline/area of research, see Reproducible research in âŚ, and if youâre looking for help with a specific piece of softwareor programming languages, visit Reproducible research with âŚ
đ¤ What to know? The basics of reproducible research
The concept of a âresearch compendiumâ. Start with this Turing Way chapter - https://book.the-turing-way.org/reproducible-research/compendia.html, skim the original paper coining the term (https://doi.org/10.1198/106186007X178663), and then check out more information at https://research-compendium.science/, e.g., something from your discipline.
What to read, watch, or listen to?
I have only 20 minutes
I have only one hour
I have half a day
All of the above, then:
I have one full day
All of the above, then:
I have all the time I need
All of the above (though not necessarily in that order), then:
- The Turing Wayâs Guide for Reproducible Research from top to bottom: https://book.the-turing-way.org/
- Chiarelli, A., Loffreda, L., & Johnson, R. (2021). The Art of Publishing Reproducible Research Outputs: Supporting emerging practices through cultural and technological innovation. Zenodo. https://doi.org/10.5281/zenodo.5521077
- Arguillas, F., Christian, T.-M., Gooch, M., Honeyman, T., Peer, L., & CURE-FAIR WG. (2022). 10 Things for Curating Reproducible and FAIR Research (1.1). Zenodo. https://doi.org/10.15497/RDA00074
- https://repro4everyone.org/pages/explore-topics/ and a preprint describing the project: Auer, S., Haelterman, N., Weissgerber, T. L., Erlich, J. C., Susilaradeya, D., Julkowska, M., & Jadavji, N. M. (2021). A community-led initiative for training in reproducible research. eLife 2021;10:e64719. https://doi.org/10.7554/eLife.64719
- Benureau, F. C. Y., & Rougier, N. P. (2018). Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions. Frontiers in Neuroinformatics, 11. https://doi.org/10.3389/fninf.2017.00069
- Stodden, V. and Miguez, S., 2014. Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research. Journal of Open Research Software, 2(1), p.e21. DOI: http://doi.org/10.5334/jors.ay
- David L. Donoho, An invitation to reproducible computational research, Biostatistics, Volume 11, Issue 3, July 2010, https://doi.org/10.1093/biostatistics/kxq028
-
- Peer, L., Orr, L., & Coppock, A. (2021). âActive Maintenance: A Proposal for the Long-Term Computational Reproducibility of Scientific Results.â PS: Political Science & Politics, 1-5. https://doi.org/10.1017/S1049096521000366
- Gute (digitale) wissenschaftliche Praxis und Open Science (Helmholtz Open Science Briefing): https://doi.org/10.2312/os.helmholtz.012
- If you like watching videos, the RIOT Science YouTube channel has a lot of interesting material: https://www.youtube.com/c/RIOTScienceClub/videos
- Selected articles from the Ten Simple Rules collection, for example:
- Principles for data analysis workflows: https://doi.org/10.1371/journal.pcbi.1008770
- Look for a study from your field on the Awesome Reproducible Research list and try to learn from it
- Sharing data and code during peer review: https://www.cambridge.org/core/blog/2019/08/19/how-to-make-the-data-and-code-for-your-manuscript-available-to-peer-reviewers-before-making-it-public/
I have no time but a long commute
Journals to know
Other knowledge bases, lists, and courses
What to do at my group/lab/institute/department?
- Lowndes, J., Best, B., Scarborough, C. et al. Our path to better science in less time using open data science tools. Nat Ecol Evol 1, 0160 (2017). https://doi.org/10.1038/s41559-017-0160
- Organise a ReproHack (also works remotely!)
What about research software
Reproducible research in âŚ
Discipline-specific material and resources, in additionto the general material on the section âReproducible research: general materialâ.
Are you interested in a different discipline and know about or look for useful links? Open an issue or a PR: https://github.com/o2r-project/o2r-project.github.io/
How about: Physics, Chemistry, Zoology, Space Science/Astronomy, Linguistics, âŚ
Earth science & geosciences / GIScience / geography & spatial sciences / Ecology / Hydrology / Remote Sensing
- NĂźst, Daniel, and Pebesma, Edzer. 2020. Practical reproducibility in geography and geosciences. Annals of the American Association of Geographers. doi:10.1080/24694452.2020.1806028;
- PDF: NĂźst-and-Pebesma_2020_AAM_Practical-Reproducibility-in-Geography-and-Geosciences.pdf
- NĂźst, D., Ostermann, F. O., Sileryte, R., Hofer, B., Granell, C., Teperek, M., & Hettne, K. M. (2020, January 7). AGILE Reproducible Paper Guidelines. https://doi.org/10.17605/OSF.IO/CB7Z8
- Alston, J. M., & Rick, J. A. (2021). A Beginnerâs Guide to Conducting Reproducible Research. The Bulletin of the Ecological Society of America. https://doi.org/10.1002/bes2.1801
- Reproducible Research Techniques - Data Training by the National Ecological Observatory Network (NEON), covers documentation, R, git, tidy data, spatialdata analysis
- A Hydrologistâs Guide to Open Science by Caitlyn A. Hall, Sheila M. Saia, Andrea Popp, Nilay Dogulu, Stan Schymanski, Niels Drost, Tim van Emmerik, Rolf Hut. https://doi.org/10.31223/X58P62
- Boeing, G., and Arribas-Bel, D. (2020). GIS and Computational Notebooks. The Geographic Information Science & Technology Body of Knowledge (1st Quarter 2021 Edition), John P. Wilson (Ed.). DOI: 10.22224/gistbok/2021.1.2.
- Cross-disciplinary data practices in earth system science: Aligning services with reuse and reproducibility priorities. AnYan, Caihong Huang, Jian-Sin Lee, Carole L. Palmer First published: 22 October 2020. https://doi.org/10.1002/pra2.218
- Crystal-Ornelas, Robert, Brandon Edwards, Katherine H=C3=A9bert, Emma J. Hudgins, Luna L. Sanchez-Reyes, Eric R. Scott, Matthew Grainger, et al. 2022. Not Just for Programmers: How Github Can Accelerate Collaborative and Reproducible Research in Ecology and Evolution. MetaArXiv. July 13. https://doi.org/10.31222/osf.io/x3p2q
- Reproducible Medical Research with R: <https://bookdown.org/pdr_higgins/rmrwr/
- Ten simple rules on how to create open access and reproduciblemolecular simulations of biological systems: https://doi.org/10.1371/journal.pcbi.1006649
- http://reproducible-bioinformatics.org
- https://www.protocols.io/
- Reagent sharing (overview slidedeck): https://repro4everyone.org/docs/reagent-sharing/
- https://kbroman.org/pages/teaching.html
- Eglen, S., Marwick, B., Halchenko, Y. et al. Toward standard practices for sharing computer code and programs in neuroscience. Nat Neurosci 20, 770=E2=80=93773 (2017). https://doi.org/10.1038/nn.4550
- Research Reproducibility & Replicability Webinar in clinical informatics in the midst of the COVID-19 pandemic (U of A Research Reproducibility & Replicability Committee, Nov 2020): https://scholarworks.uark.edu/oreievt/1/
- Portable Encapsulated Projects (PEP) for standardized sample metadata structure for biological research: http://pep.databio.org/
- Practical Resources for Enhancing the Reproducibility of Mechanistic Modeling in Systems Biology by Michael L. Blinov, John H. Gennari, Jonathan R. Karr, Ion I. Moraru, David P. Nickerson, Herbert M. Sauro. arXiv:2104.04604 [q-bio.QM]
- Veronica Porubsky, Lucian Smith, Herbert M Sauro, Publishing reproducible dynamic kinetic models, Briefings in Bioinformatics, Volume 22, Issue 3, May 2021, bbaa152, https://doi.org/10.1093/bib/bbaa152 (systems biology)
- Juan Luis Riquelme & Julijana Gjorgjieva, Towards readable code in neuroscience: https://doi.org/10.1038/s41583-021-00450-y
- Heil, B.J., Hoffman, M.M., Markowetz, F. et al. Reproducibility standards for machine learning in the life sciences. Nat Methods 18, 1132=E2=80=931135 (2021). https://doi.org/10.1038/s41592-021-01256-7
Archaeology
- Reproducibility and Transparency in Archaeological Science (workshop by B. Marwick, using rrtools)
- Marwick, B., & Wang, L. (2019, August 22). How to align disciplinary ideals with actual practices: Transparency and openness in archaeological science. https://doi.org/10.31235/osf.io/s8z6u
- Schmidt, S. C., & Marwick, B. (2019, January 5). Tool-driven Revolutions in Archaeological Science. https://doi.org/10.31235/osf.io/4nkxv
Mathematics
- JĂśrg Fehr, Jan Heiland, Christian Himpe, Jens Saak. Best practices for replicability, reproducibility and reusability of computer-based experiments exemplified by model reduction software[J]. AIMS Mathematics, 2016, 1(3): 261-281. https://10.3934/Math.2016.3.261
- Gray C.T., Marwick B. (2019) Truth, Proof, and Reproducibility: Thereâs No Counter-Attack for the Codeless. In: Nguyen H. (eds) Statistics and Data Science. RSSDS 2019. Communications in Computer and Information Science, vol 1150. Springer, Singapore. https://doi.org/10.1007/978-981-15-1960-4_8
Psychology
- Open Science Initiative der Institute fĂźr Psychologie der Universit=C3=A4t MĂźnster
- Hardwicke, T. E., Bohn, M., MacDonald, K. E., Hembacher, E., Nuijten, M. B., Peloquin, B., & Frank, M. C. (2020, July 13). Analytic reproducibility in articles receiving open data badges at Psychological Science: An observational study. https://doi.org/10.31222/osf.io/h35wt
- Data & file management in Psychology (source): Videos, slides (âUnit5â)
- Rouder, Jeffrey N., Julia M. Haaf, and Hope K. Snyder. =E2=80=9CMinimizing Mistakes in Psychological Science.=E2=80=9D Advances in Methods and Practices in Psychological Science 2,no. 1 (March 2019): 3=E2=80=9311. https://doi.org/10.1177/2515245918801915
- Wiebels, K., & Moreau, D. (2021). Leveraging Containers for Reproducible Psychological Research. Advances in Methods and Practices in Psychological Science, 4(2), 251524592110178. https://doi.org/10.1177/25152459211017853
Economics
- Christensen, Garret, and EdwardMiguel. 2018. âTransparency, Reproducibility, and the Credibility of Economics Research.â Journal of Economic Literature, 56 (3): 920-80. https://doi.org/10.1257/jel.20171350
Political Science
- Peer, L., Orr, L., & Coppock, A. (2021). âActive Maintenance: A Proposal for the Long-Term Computational Reproducibility of Scientific Results.â PS: Political Science & Politics, 1-5. https://doi.org/10.1017/S1049096521000366
Philosophy
Computer science & =E2=9A=99=EF=B8=8F Engineering
Social science & humanities
Communication
- Tobias Dienlin, et al. An Agenda for Open Science in Communication, Journal of Communication, Volume 71, Issue 1, February 2021, Pages 1=E2=80=9326, https://doi.org/10.1093/joc/jqz052
Reproducible research with âŚ
Resources about reproducibility with specific tools, languages, or methods.
Computers & data
R
Python
Matlab
A concise guide to reproducible MATLAB projects, David Wilby: https://rse.shef.ac.uk/blog/2022-05-05-concise-guide-to-reproducible-matlab/
Machine Learning (ML) / Artificial Intelligence (AI)
Blind peer review
GitHub
- Perez-Riverol Y, Gatto L, Wang R, Sachsenberg T, Uszkoreit J, Leprevost FdV, et al. (2016) Ten Simple Rules for Taking Advantage of Git and GitHub. PLoS Comput Biol 12(7): e1004947. https://doi.org/10.1371/journal.pcbi.1004947
- Crystal-Ornelas, Robert, Brandon Edwards, Katherine H=C3=A9bert, Emma J. Hudgins, Luna L. Sanchez-Reyes, Eric R. Scott, Matthew Grainger, etal. 2022. Not Just for Programmers: How Github Can Accelerate Collaborative and Reproducible Research in Ecology and Evolution. MetaArXiv. July 13. https://doi.org/10.31222/osf.io/x3p2q
Software and data citation and licensing
At some point the citation and licensing of software can become important for researchers. One structural problem of current science is, that very often scientific research software and research data is not covered by the commonly used âsuccess metricsâ for scientific careers. Ensuring proper citation of software and data should thus be of high importance for developersand researchers alike. This holds for both your own software and data (making it citable, citing it) and software you use. Data sharing can benefit your scientific career by leading to greater collaboration, increased confidence in findings and goodwill between researchers (https://doi.org/10.1038/d41586-019-01506-x). Furthermore, several studies have shown that articles making data available have a citation benefit and data are actually reused (https://doi.org/10.7717/peerj.175, https://doi.org/10.1371/journal.pone.0230416).The same can be argued for software.
Licensing is important to keep in mind when starting toshare, collaboratively develop, or reuse code and data. Itâs worth gettinga quick overview of what copyright is (https://en.wikipedia.org/wiki/Copyright) and to acknowledge that (i) copyright law is very diverse across legal jurisdictions, (ii) the laws and their application for âmodernâ things like data and software are partly still evolving, and (iii) we need copyright law to be able to allow people to use our work. Important disclaimer: the information provided here is not legal advice. If youare unsure about copyright and licensing, consult your lawyer.
âThe Legal Framework for Reproducible Scientific Research - Licensing and Copyrightâ (https://doi.org/10.1109/MCSE.2009.19, public PDF at https://academiccommons.columbia.edu/doi/10.7916/D8GH9TD8/download) gives you a good overview and provides clear recommendations on practices and licenses, as is âA Quick Guide to Software Licensing for the Scientist-Programmerâ (https://doi.org/10.1371/journal.pcbi.1002598). If you want to make sure the licenses of software you use or share supports your intentions and do not stand in conflict with each other, TL;DR Legal can help you out: https://tldrlegal.com/. The website http://forschungslizenzen.de informs about rights and licenses for research data (German only) with a special focus on the humanities.
The Software Sustainability Instituteâs page âHow to cite and describe softwareâ is a great starting point for software citation (https://www.software.ac.uk/how-cite-software), albeit being a bit outdated. A more current article ais âRecognizing the value of software: a software citation guideâ (https://doi.org/10.12688/f1000research.26932.2), as it includes recent initiatives such as Software Heritage (https://www.softwareheritage.org/). If you use a modern reference managers, the biblatex-software style (https://www.softwareheritage.org/2020/05/26/citing-software-with-style/) might be useful. The GitHub-Zenodo integration makes getting a citable DOI for every release very easy (https://guides.github.com/activities/citable-code/), but manual publishing from GitLab(.com, ZIV-GitLab) is almost as simple. Pro-tip: look for .zenodo.json files on GitHub to automate the metadata insertion on Zenodo and consider publishing a software paper in JOSS (https://joss.theoj.org/) or JORS (https://openresearchsoftware.metajnl.com/).
For data citation, university libraries and data repositories are your places to go. Data publication is part of more established practices around research data management (RDM, Forschungsdatenmanagement -FDM) and often is required by funders. Therefore, all universities have services in this area (https://www.uni-muenster.de/Forschungsdaten/) and the more static and less evolving nature of data, compared to software, makes some things easier as well. Generic information can be found at DataCite (âCite your Dataâ, https://datacite.org/cite-your-data.html) and DataVerse (https://dataverse.org/best-practices/data-citation). Open Data Commons (https://opendatacommons.org/) provides established licenses to use and an excellent FAQ (https://opendatacommons.org/faq/licenses/).
In a nutshell
- Make your own data and software citable and provide the desired citation in your README.
- Make your data and software usable by others by using open licenses (data licenses for data, software licenses for software).
- Put software on Zenodo and/or Software Heritage.
- Put data in a suitable research data repository, which you can find onhttps://www.re3data.org/.
- Cite all data and software that you use with their proper version and DOI. If data use reuse does not have a DOI, ask the author to make it citable.
Research Software Engineering & Software publishing
- Adding software to package management systems can increase their citation by 280%: <https://doi.org/10.1101/2020.11.16.385211
- Ten simple rules for quick and dirty scientific programming: <https://doi.org/10.1371/journal.pcbi.1008549
- Twelve quick tips for software design: <https://doi.org/10.1371/journal.pcbi.1009809
- The Good Research Code Handbook: <https://goodresearch.dev/ =
Containers