August 2021 MDAP Seminar
The repliCATS project
Abstract: The repliCATS project crowdsources evaluations of the reliability of published research claims in social and behavioural sciences from small groups of experts through a structured elicitation protocol delivered by a custom-built online platform. This elicitation provides ‘Confidence Scores’ for papers across a range of dimensions, such as plausibility, transparency, replicability and generalizability. The repliCATS project is part of the SCORE program, a multi-team based international research project with the goal of generating automated Confidence Scores for research papers. A longer-term goal for the repliCATS project is to reimagine peer-review as a structured deliberation process. A particular feature of the repliCATS project is that it generates a considerable volume of qualitative data in which experts justify their assessments. This dataset provides a unique opportunity to understand how scientists reason about the work of other scientists through both mixed-methods and purely qualitative techniques. In this talk we outline the repliCATS project, progress to date and future possibilities with a particular focus on describing how the qualitative data is collected, processed and analysed.
Speaker bios: Dr Fallon Mody is a Research Fellow in the interdisciplinary MetaMelb lab and the School of Historical and Philosophical Studies at the University of Melbourne. Fallon helps coordinate the repliCATS project, and leads the Community & Engagement team who oversee data collection and engagement of a global participant pool of researchers. Having completed a doctoral thesis in history of medicine, Fallon is interested in the history of contemporary metaresearch communities and the challenges of interdisciplinary reform movements.
Dr Martin Bush is a Research Fellow in the MetaMelb lab at the School of Historical and Philosophical Studies at the University of Melbourne. Martin coordinates the Reasoning Analysis Team for the repliCATS project, undertaking qualitative analysis of expert reasoning. Other metaresearch interests focus on public trust in science and public reasoning practices and beyond this, Martin Bush is a cultural historian of popular science.