Results Reproducibility and Resources Citation in Science and Technology of Language Workshop Programme 09 : 00 – 10 : 30 Reproducibility

This paper presents the idea of applying an open source, web-based platform – Gonito.net – for hosting challenges for researchers in the field of natural language processing. Researchers are encouraged to compete in well-defined tasks by developing tools and running them on provided test data. The researcher who submits the best results becomes the winner of the challenge. Apart from the competition, Gonito.net also enables collaboration among researchers by means of source code sharing mechanisms. Gonito.net itself is fully open source, i.e. its source is available for download and compilation, as well as a running instance of the system, available at gonito.net. The key design feature of Gonito.net is using Git for managing solutions of problems submitted by competitors. This allows for research transparency and reproducibility.

[1]  Ross Ihaka,et al.  Gentleman R: R: A language for data analysis and graphics , 1996 .

[2]  Robert Wilensky,et al.  A framework for distributed digital object services , 2006, International Journal on Digital Libraries.

[3]  Jelena Kovacevic,et al.  How to Encourage and Publish Reproducible Research , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[4]  Gil Francopoulo,et al.  TagParser: well on the way to ISO-TC37 conformance , 2008 .

[5]  Huai Liu,et al.  An innovative approach for testing bioinformatics programs using metamorphic testing , 2009, BMC Bioinformatics.

[6]  Ted Pedersen,et al.  Empiricism Is Not a Matter of Faith , 2008, Computational Linguistics.

[7]  Elena Beisswanger,et al.  Building a BioWordNet Using WordNet Data Structures and WordNet’s Software Infrastructure–A Failure Story , 2008, SETQALNLP.

[8]  Dragomir R. Radev,et al.  The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics , 2008, LREC.

[9]  Gail E. Kaiser,et al.  Properties of Machine Learning Applications for Use in Metamorphic Testing , 2008, SEKE.

[10]  Hong Yu,et al.  TRANSLATING BIOLOGY: TEXT MINING TOOLS THAT WORK. , 2008, Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing.

[11]  R. Dellavalle,et al.  The Processes of Life: An Introduction to Molecular Biology , 2009 .

[12]  Baowen Xu,et al.  Application of Metamorphic Testing to Supervised Classifiers , 2009, 2009 Ninth International Conference on Quality Software.

[13]  Markus Rupp,et al.  Reproducible research in signal processing , 2009, IEEE Signal Processing Magazine.

[14]  M. Garcia-Blanco,et al.  Factors Affecting Reproducibility between Genome-Scale siRNA-Based Screens , 2010, Journal of biomolecular screening.

[15]  Baowen Xu,et al.  Testing and validating machine learning classifiers by metamorphic testing , 2011, J. Syst. Softw..

[16]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[17]  F. Prinz,et al.  Believe it or not: how much can we rely on published data on potential drug targets? , 2011, Nature Reviews Drug Discovery.

[18]  Wanxiang Che,et al.  A Graph-based Method for Entity Linking , 2011, IJCNLP.

[19]  W. Bisang.  Variation and reproducibility in linguistics , 2011 .

[20]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[21]  Claudia Soria,et al.  The LRE Map. Harmonising Community Descriptions of Resources , 2012, LREC.

[22]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[23]  Patrick Paroubek,et al.  Rediscovering 25 years of discoveries in spoken language processing: a preliminary ISCA archive analysis , 2013, INTERSPEECH.

[24]  Zhiyong Lu,et al.  PubTator: a web-based text mining tool for assisting biocuration , 2013, Nucleic Acids Res..

[25]  Antske Fokkens,et al.  Offspring from Reproduction Problems: What Replication Failure Teaches Us , 2013, ACL.

[26]  Gil Francopoulo,et al.  Global Atlas: Proper Nouns, From Wikipedia to LMF , 2013 .

[27]  Karthik Ram,et al.  Git can facilitate greater reproducibility and increased transparency in science , 2013, Source Code for Biology and Medicine.

[28]  Dragomir R. Radev,et al.  The ACL anthology network corpus , 2009, Language Resources and Evaluation.

[29]  Suzanne K. Linder,et al.  A Survey on Data Reproducibility in Cancer Research Provides Insights into Our Limited Ability to Translate Findings from the Laboratory to the Clinic , 2013, PloS one.

[30]  Roberto Navigli,et al.  Entity Linking meets Word Sense Disambiguation: a Unified Approach , 2014, TACL.

[31]  Christian Collberg,et al.  Measuring Reproducibility in Computer Systems Research , 2014 .

[32]  Patrick Paroubek,et al.  Facing the Identification Problem in Language-Related Scientific Data Analysis. , 2014, LREC.

[33]  Kenneth Heafield,et al.  N-gram Counts and Language Models from the Common Crawl , 2014, LREC.

[34]  Anupam Joshi,et al.  PROB: A tool for Tracking Provenance and Reproducibility of Big Data Experiments , 2014, HPCA 2014.

[35]  Paul Buitelaar,et al.  Hot Topics and Schisms in NLP: Community and Trend Analysis with Saffron on ACL and LREC Proceedings , 2014, LREC.

[36]  Jean-François Méhaut,et al.  Modeling and Simulation of a Dynamic Task-Based Runtime System for Heterogeneous Multi-core Architectures , 2014, Euro-Par.

[37]  Patrick Paroubek,et al.  NLP4NLP: The Cobbler's Children Won't Go Unshod , 2015, D Lib Mag..

[38]  Arnaud Legrand,et al.  An Effective Git And Org-Mode Based Workflow For Reproducible Research , 2015, OPSR.

[39]  Jyoti Rani,et al.  pubmed.mineR: An R package with text-mining algorithms to analyse PubMed abstracts , 2015, Journal of Biosciences.

[40]  Alex M. Warren.  Repeatability and Benefaction in Computer Systems Research — A Study and a Modest Proposal , 2015 .

[41]  Joseph Mariani,et al.  Language Matrices and a Language Resource Impact Factor , 2015 .

[42]  Filip Gralinski,et al.  "He Said She Said" ― a Male/Female Corpus of Polish , 2016, LREC.

[43]  Patrick Paroubek,et al.  Rediscovering 15 + 2 years of discoveries in language resources and evaluation , 2016, Lang. Resour. Evaluation.