RepeatabilityCentral - Revision history

Juliana at 16:01, 22 August 2012

2012-08-22T16:01:13Z

← Older revision		Revision as of 16:01, 22 August 2012
Line 1:		Line 1:
			"This page is no longer maintained. Please see http://www.reproduciblescience.org"

	== News ==		== News ==

Juliana: /* News */

2012-05-30T19:57:52Z

News

← Older revision		Revision as of 19:57, 30 May 2012
Line 6:		Line 6:

	* The VisTrails 2.0 has been released and it is available for download from http://www.vistrails.org/index.php/Downloads. VisTrails now includes support for the creation of reproducible papers.		* The VisTrails 2.0 has been released and it is available for download from http://www.vistrails.org/index.php/Downloads. VisTrails now includes support for the creation of reproducible papers.


	* The discipline involved in the publication of reproducible results can actually lead to higher quality science: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html		* The discipline involved in the publication of reproducible results can actually lead to higher quality science: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html
Line 19:		Line 18:
	** [http://www.nytimes.com/2010/09/24/science/24retraction.html?\_r=1\&emc=eta1 Nobel Laureate Retracts Two Papers Unrelated to Her Prize]		** [http://www.nytimes.com/2010/09/24/science/24retraction.html?\_r=1\&emc=eta1 Nobel Laureate Retracts Two Papers Unrelated to Her Prize]
	** [http://www.nytimes.com/2011/06/26/opinion/sunday/26ideas.html?\_r=2 It's Science, but Not Necessarily Right]		** [http://www.nytimes.com/2011/06/26/opinion/sunday/26ideas.html?\_r=2 It's Science, but Not Necessarily Right]


	* There is an interesting new site that is tracking paper retractions, check it out:		* There is an interesting new site that is tracking paper retractions, check it out:

Juliana: /* Presentations */

2012-05-30T19:57:34Z

Presentations

← Older revision		Revision as of 19:57, 30 May 2012
Line 111:		Line 111:

	== Presentations ==		== Presentations ==

			* [http://vgc.poly.edu/~juliana/talks/reproducibility-tutorial2012.pdf Computational reproducibility: state-of-the-art, challenges, and database research opportunities]. Juliana Freire, Philippe Bonnet and Dennis Shasha. Tutorial presented at ACM SIGMOD 2012.

	* [http://vgc.poly.edu/~juliana/talks/2011-amp-reproducible-research.pdf Provenance-Rich Science.] Juliana Freire. UFRJ, Brazil. August 1, 2011.		* [http://vgc.poly.edu/~juliana/talks/2011-amp-reproducible-research.pdf Provenance-Rich Science.] Juliana Freire. UFRJ, Brazil. August 1, 2011.

Juliana: /* Publications */

2012-05-30T19:56:16Z

Publications

← Older revision		Revision as of 19:56, 30 May 2012
Line 101:		Line 101:

	== Publications ==		== Publications ==

			* [http://vgc.poly.edu/~juliana/pub/freire-sigmod2012.pdf Computational reproducibility: state-of-the-art, challenges, and database research opportunities], by Juliana Freire, Philippe Bonnet, Dennis Shasha. In SIGMOD Conference, 593-596, 2012.

	* [http://www.sigmod.org/publications/sigmod-record/1106/pdfs/08.report.bonnet.pdf Repeatability and Workability Evaluation of SIGMOD 2011], by Philippe Bonnet, Stefan Manegold, Matias Bjorling, Wei Cao, Javier Gonzales, Joel Granados, Nancy Hall, Stratos Idreos, Milena Ivanova, Ryan Johnson, David Koop, Tim Kraska, René Müller, Dan Olteanu, Paolo Papotti, Christine Reilly, Dimitris Tsirogiannis, Cong Yu, Juliana Freire and Dennis Shasha. In SIGMOD Record, vol. 40, no. 2, pp. 45-48, 2011.		* [http://www.sigmod.org/publications/sigmod-record/1106/pdfs/08.report.bonnet.pdf Repeatability and Workability Evaluation of SIGMOD 2011], by Philippe Bonnet, Stefan Manegold, Matias Bjorling, Wei Cao, Javier Gonzales, Joel Granados, Nancy Hall, Stratos Idreos, Milena Ivanova, Ryan Johnson, David Koop, Tim Kraska, René Müller, Dan Olteanu, Paolo Papotti, Christine Reilly, Dimitris Tsirogiannis, Cong Yu, Juliana Freire and Dennis Shasha. In SIGMOD Record, vol. 40, no. 2, pp. 45-48, 2011.

Juliana: /* News */

2012-05-30T19:49:42Z

News

← Older revision		Revision as of 19:49, 30 May 2012
Line 1:		Line 1:
	== News ==		== News ==

			* We have given a tutorial at SIGMOD 2012 on reproducibility: [http://vgc.poly.edu/~juliana/talks/reproducibility-tutorial2012.pdf Computational reproducibility: state-of-the-art, challenges, and database research opportunities]

	* The DataBase experiments Repository (DBXR ) is now up and running, thanks to [http://www.itu.dk/people/phbo Philippe Bonnet]. This site serves as a repository for experiments related to database research. Currently, it supports the submission and review of results published at PVLDB and ACM Sigmod. For more details see http://www.dbxr.org.		* The DataBase experiments Repository (DBXR ) is now up and running, thanks to [http://www.itu.dk/people/phbo Philippe Bonnet]. This site serves as a repository for experiments related to database research. Currently, it supports the submission and review of results published at PVLDB and ACM Sigmod. For more details see http://www.dbxr.org.

Juliana at 19:47, 30 May 2012

2012-05-30T19:47:55Z

← Older revision		Revision as of 19:47, 30 May 2012
Line 1:		Line 1:
	== News ==		== News ==

	* [[More News]]

	* The DataBase experiments Repository (DBXR ) is now up and running, thanks to [http://www.itu.dk/people/phbo Philippe Bonnet]. This site serves as a repository for experiments related to database research. Currently, it supports the submission and review of results published at PVLDB and ACM Sigmod. For more details see http://www.dbxr.org.		* The DataBase experiments Repository (DBXR ) is now up and running, thanks to [http://www.itu.dk/people/phbo Philippe Bonnet]. This site serves as a repository for experiments related to database research. Currently, it supports the submission and review of results published at PVLDB and ACM Sigmod. For more details see http://www.dbxr.org.
Line 7:		Line 5:
	* The VisTrails 2.0 has been released and it is available for download from http://www.vistrails.org/index.php/Downloads. VisTrails now includes support for the creation of reproducible papers.		* The VisTrails 2.0 has been released and it is available for download from http://www.vistrails.org/index.php/Downloads. VisTrails now includes support for the creation of reproducible papers.

	* "U.S. Department of Energy’s Energy Sciences Network, or ESnet, provides reliable high-bandwidth network services to thousands of researchers tackling some of the most pressing scientific and engineering problems, such as finding new sources of clean energy, increasing energy efficiency, understanding climate change, developing new materials for industry and discovering the nature of our universe."
	~~http://www.hpcwire.com/hpcwire/2012-04-26/esnet_launches_architecture_to_help_researchers_deliver_on_data-intensive_science.html~~

	* The discipline involved in the publication of reproducible results can actually lead to higher quality science:

	''In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set. For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups. Investigators frequently presented the results of one experiment, such as a single Western-blot analysis. They sometimes said they presented specific experiments that supported their underlying hypothesis, but that were not reflective of the entire data set. . .''

	~~(from~~ http://www.nature.com/nature/journal/v483/n7391/full/483531a.html)		* The discipline involved in the publication of reproducible results can actually lead to higher quality science: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html

	* There has been increased concern about reproducibility in clinical trials:		* There has been increased concern about reproducibility in clinical trials:
Line 26:		Line 18:
	** [http://www.nytimes.com/2011/06/26/opinion/sunday/26ideas.html?\_r=2 It's Science, but Not Necessarily Right]		** [http://www.nytimes.com/2011/06/26/opinion/sunday/26ideas.html?\_r=2 It's Science, but Not Necessarily Right]

	* A recent study discusses the important of publications that report on negative results:
	** http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0010271

	* There is an interesting new site that is tracking paper retractions, check it out:		* There is an interesting new site that is tracking paper retractions, check it out:
Line 35:		Line 25:

	* The [http://www.vistrails.org/index.php/ExecutablePapers VisTrails methodology and infrastructure for creating provenance-rich, executable publications] has been selected as a finalist of the [http://www.executablepapers.com Executable Paper Grand Challenge]. We presented this work at the [http://www.iccs-meeting.org/iccs2011/index.html ICCS Meeting], in Singapore.		* The [http://www.vistrails.org/index.php/ExecutablePapers VisTrails methodology and infrastructure for creating provenance-rich, executable publications] has been selected as a finalist of the [http://www.executablepapers.com Executable Paper Grand Challenge]. We presented this work at the [http://www.iccs-meeting.org/iccs2011/index.html ICCS Meeting], in Singapore.

			* [[More News]]

	== Project Description ==		== Project Description ==

Juliana at 19:46, 30 May 2012

2012-05-30T19:46:09Z

← Older revision		Revision as of 19:46, 30 May 2012
Line 1:		Line 1:
	== News ==		== News ==

			* [[More News]]

	* The DataBase experiments Repository (DBXR ) is now up and running, thanks to [http://www.itu.dk/people/phbo Philippe Bonnet]. This site serves as a repository for experiments related to database research. Currently, it supports the submission and review of results published at PVLDB and ACM Sigmod. For more details see http://www.dbxr.org.		* The DataBase experiments Repository (DBXR ) is now up and running, thanks to [http://www.itu.dk/people/phbo Philippe Bonnet]. This site serves as a repository for experiments related to database research. Currently, it supports the submission and review of results published at PVLDB and ACM Sigmod. For more details see http://www.dbxr.org.

Juliana: /* Project Description */

2012-05-11T22:28:41Z

Project Description

← Older revision		Revision as of 22:28, 11 May 2012
Line 38:		Line 38:
	A hallmark of the scientific method has been that experiments should be described in enough detail that they can be repeated and perhaps generalized. This implies the ability to redo experiments in nominally equal settings and also to test the generalizability of a claimed conclusion by trying similar experiments in different settings. In principle, this should be easier for computational experiments than for natural science experiments, because not only can computational processes be automated but also computational systems do not suffer from the 'biological variation' that plagues the life sciences. Unfortunately, the state of the art falls far short of this goal. Most computational experiments are specified only informally in papers, where experimental results are briefly described in figure captions; the code that produced the results is seldom available; and configuration parameters change results in unforeseen ways. Because important scientific discoveries are often the result of sequences of smaller, less significant steps, the ability to publish results that are fully documented and reproducible is necessary for advancing science. While concern about repeatability and generalizability cuts across virtually all natural, computational, and social science fields, no single field has identified this concern as a target of a research effort.		A hallmark of the scientific method has been that experiments should be described in enough detail that they can be repeated and perhaps generalized. This implies the ability to redo experiments in nominally equal settings and also to test the generalizability of a claimed conclusion by trying similar experiments in different settings. In principle, this should be easier for computational experiments than for natural science experiments, because not only can computational processes be automated but also computational systems do not suffer from the 'biological variation' that plagues the life sciences. Unfortunately, the state of the art falls far short of this goal. Most computational experiments are specified only informally in papers, where experimental results are briefly described in figure captions; the code that produced the results is seldom available; and configuration parameters change results in unforeseen ways. Because important scientific discoveries are often the result of sequences of smaller, less significant steps, the ability to publish results that are fully documented and reproducible is necessary for advancing science. While concern about repeatability and generalizability cuts across virtually all natural, computational, and social science fields, no single field has identified this concern as a target of a research effort.

	This ~~collaborative~~ project ~~between the University of Utah and New York University consists~~ of tools and infrastructure that supports the process of sharing, testing and re-using scientific experiments and results by leveraging and extending the infrastructure provided by provenance-enabled scientific workflow systems. The project explores three key research questions: 1) How to package and publish compendia of scientific results that are reproducible and generalizable. 2) What are appropriate algorithms and interfaces for exploring, comparing, re-using the results or potentially discovering better approaches for a given problem? 3) How to aid reviewers to generate experiments that are most informative given a time/resource limit.		This project is developing a suite of tools and infrastructure that supports the process of sharing, testing and re-using scientific experiments and results by leveraging and extending the infrastructure provided by provenance-enabled scientific workflow systems. The project explores three key research questions: 1) How to package and publish compendia of scientific results that are reproducible and generalizable. 2) What are appropriate algorithms and interfaces for exploring, comparing, re-using the results or potentially discovering better approaches for a given problem? 3) How to aid reviewers to generate experiments that are most informative given a time/resource limit.

	An expected result of this work is a software infrastructure that allows authors to create workflows that encode the computational processes that derive the results (including data used, configuration parameters set, and underlying software), publish and connect these to publications where the results are reported. Testers (or reviewers) can repeat and validate results, ask questions anonymously, and modify experimental conditions. Researchers, who want to build upon previous works, are able to search, reproduce, compare and analyze experiments and results. The infrastructure helps scientists in any discipline to construct, publish and share reproducible results.		An expected result of this work is a software infrastructure that allows authors to create workflows that encode the computational processes that derive the results (including data used, configuration parameters set, and underlying software), publish and connect these to publications where the results are reported. Testers (or reviewers) can repeat and validate results, ask questions anonymously, and modify experimental conditions. Researchers, who want to build upon previous works, are able to search, reproduce, compare and analyze experiments and results. The infrastructure helps scientists in any discipline to construct, publish and share reproducible results.

Juliana: /* What scientists are doing about reproducibility */

2012-05-11T18:41:27Z

What scientists are doing about reproducibility

← Older revision		Revision as of 18:41, 11 May 2012
Line 47:		Line 47:

	* Professor Matthias Troyer (ETH Zurich) and his collaborators have published a number of papers whose results are fully reproducible. He is using VisTrails to both carry out the experiments and to package them for publication. Here are some of this reproducible papers:		* Professor Matthias Troyer (ETH Zurich) and his collaborators have published a number of papers whose results are fully reproducible. He is using VisTrails to both carry out the experiments and to package them for publication. Here are some of this reproducible papers:
	** [http://iopscience.iop.org/1742-5468/2011/05/P05001 The ALPS project release 2.0: open source software for strongly correlated systems], also available at http://arxiv.org/pdf/1101.2646.pdf		** [http://iopscience.iop.org/1742-5468/2011/05/P05001 The ALPS project release 2.0: open source software for strongly correlated systems], (by Bela Bauer et al.). also available at http://arxiv.org/pdf/1101.2646.pdf
	** [http://arxiv.org/abs/1106.3267 Galois Conjugates of Topological Phases]		** [http://arxiv.org/abs/1106.3267 Galois Conjugates of Topological Phases] (by Michael H. Freedman, Jan Gukelberger, Matthew B. Hastings, Simon Trebst, Matthias Troyer, Zhenghan Wang)

	== Infrastructure to Create Provenance-Rich Papers ==		== Infrastructure to Create Provenance-Rich Papers ==

Juliana: /* What scientists are doing about reproducibility */

2012-05-11T18:39:59Z

What scientists are doing about reproducibility

← Older revision		Revision as of 18:39, 11 May 2012
Line 46:		Line 46:
	* The journal Biostatistics has an associate editor for reproducibility who can assign grades of merit to conditionally accepted papers: D: data are available, C: code is available, and R: the AE could run the code and reproduce the results without much effort. (http://magazine.amstat.org/blog/2011/01/01/scipolicyjan11)		* The journal Biostatistics has an associate editor for reproducibility who can assign grades of merit to conditionally accepted papers: D: data are available, C: code is available, and R: the AE could run the code and reproduce the results without much effort. (http://magazine.amstat.org/blog/2011/01/01/scipolicyjan11)

	* Professor Matthias Troyer (ETH Zurich) and his collaborators have published a number of papers whose results are fully reproducible. He is using VisTrails to both carry out the experiments and to package them for publication.		* Professor Matthias Troyer (ETH Zurich) and his collaborators have published a number of papers whose results are fully reproducible. He is using VisTrails to both carry out the experiments and to package them for publication. Here are some of this reproducible papers:
			** [http://iopscience.iop.org/1742-5468/2011/05/P05001 The ALPS project release 2.0: open source software for strongly correlated systems], also available at http://arxiv.org/pdf/1101.2646.pdf
			** [http://arxiv.org/abs/1106.3267 Galois Conjugates of Topological Phases]

	== Infrastructure to Create Provenance-Rich Papers ==		== Infrastructure to Create Provenance-Rich Papers ==