R&R Workshop at SPARC

The capabilities of containerisation and the concept of the Executable Research Compendium form the basis for o2r’s reproducibility service. But the demonstration how latest technology may support a more open and transparent scholarly publication alone is only one half the battle. Breakthroughs in tools and infrastructure must be accompanied by outreach activities to highlight the need for opening reproducible research to all stakeholders (scientists, editors, publishers, funding agencies). That is why I was extremely glad to join some of the most renowned researchers of geography and GI Science at the “Replicability and Reproducibility Workshop” in Tempe, Arizona, on February 11 and 12, 2019. The event was organised by the Spatial Analysis Research Center (SPARC) at Arizona State University (ASU).

Daniel Nust from the University of Munster presenting at our Replicability and Reproducibility in Geospatial Research: Open is not enough for reproducibility! pic.twitter.com/PwpVmwdQ7b

— SPARC at ASU (@SPARC_ASU) 11. Februar 2019

The events featured four full talks. I was invited to go first and report on the activities of o2r as well as the Reproducible AGILE conference series and initiative for developing new submission and reviewing guidelines. The expectations were high, but after three years of intense work by the o2r team, the allotted time was easily filled. After introducing challenges which disrupt scholarly publication practices, I reported on observations made by our own surveys and others on reproducibility, including the two reproduction campaigns let be the o2r team. The painted picture unsurprisingly left a lot of room for improvement, for which the talk provided technical as well as organisational approaches.

Just finished my talk on challenges, observations and approaches towards reproducible research at the SPARC workshop #RandR #geospatial @ASU Find the slides and material (including speaker notes in the source file with all the stuff I forgot!) at https://t.co/cSRZ1DkwhO pic.twitter.com/FXjNj0hT2K

— Daniel Nüst (@nordholmen) 11. Februar 2019

In the second talk, ASU’s own Peter Kedron took a step back and surveyed the existing definitions of replicability and reproducibility. Peter then excellently connected and extended these terms with the intricacies and specifics of geospatial sciences. The technical and theoretical groundwork was layed, so the discussion following the first talks set the bar for the remainder of the workshop quite high. Many critical and thoughtful comments were made and viewpoints shared. One of the main take-home messages for me was that while geography/GI Science/related disciplines may take advantage of the hard lessons learned in other domains (which faced a ‘replication crisis’), the uniqueness of geography as a science that always had to deal with uncertainty and context may also contribute a unique perspective on replicability and reproducibility. It was great to see that the topic of reproducibility is widely acknowledged as a relevant challenge and the interest to initiate improvement was unilateral.

The first day continued with lightning talks. As could be expected, the diverse backgrounds (eScience, ecology, political geography, …) let to a very useful diversity in topics and perspectives. Afterwards the participants split up into three groups to tackle technical, organisational, and institutional aspects of replicability and reproducibility, which gave input for yet another thoughtful debate in the assembly to conclude day one. The discussions continued between old colleagues and new friends during a delightful evening reception and dinner.

Day two kicked of in a similar fashion with talks by Daniel Sui, University of Arkansas, and Esri’s Dawn Wright and Kevin Butler. Again two very different takes on the topic, with valuable new ideas. The following discussion was lively and included potential venues for the newly formed group to continue the collaboration, most importantly to increase the awareness of the topic across all communities working with spatial data. The whole meeting was nicely guided and framed by contributions from ASU’s Mike Goodchild and Stewart Fotheringham. All participants were united in their interest to advance transparency and openness and a realisation that there is a need for action from many different angles, including education and evaluation, if a ‘crisis’ shall be avoided. Despite some concerns how the topic might be received by critics, the meeting ended in a positive mood of newfound mutual support and of acknowledging the value of the work ahead.

My personal opinion is that the disruptions in science are more pressing than a traditional scholarly approach (organising a special issue for 2020, writing an editorial) can answer. Yet the old-school way may be able to bridge across the divide and different skill-set/mindset/needs between computational/junior/young/technical and theoretical/senior researchers, and is as such worth pursuing. For future discussions, I plan to frame reproducibility as an ideal that is worth striving for and worth to reward (e.g. in evaluations, during reviews, using badges, in funding schemes), but to be careful with too simple checklists and dos/don’ts, because there will always be corner cases and limitations for specific circumstances. This is a core difference between reproducibility and openness - you can not be a little or partially open, but being almost reproducible is still an important achievement. Reproducibility and replicability will not be helped by whataboutism nor by pointing fingers, but the positive spirit of preproducibility.

Luckily there is no need to echo all insights by talks and during the discussions: the sessions were recorded on video they will be published together with slides and position papers in an OSF project soon. This post will be updated then. Follow us on Twitter to not miss it. Until then you can take a look at my position paper on GitLab, even the speaker notes in the presentation source file if you dare.

This post would be incomplete without a big Thank You to the sponsoring and excellent organisation provided by Esri and the hosting School of Geographical Sciences and Urban Planning (SGSUP). I am confident this workshop may spark new collaborations and be able to put replicability and reproducibility on the map for more researchers in geography and related disciplines.

On the last flight back home after a great week with @SPARC_ASU. They brought together a great group of people to talk about #reproducibleresearch. Thank you!
I enjoyed learning more about geospatial specialities with R&R, hiking Arizona, and starting new collaborations. #PhDlife pic.twitter.com/mb1lu9Nc0Q

— Daniel Nüst (@nordholmen) 15. Februar 2019