Toward better benchmarking: challenge-based methods assessment in cancer genomics

Boutros, Paul C; Margolin, Adam A; Stuart, Joshua M; Califano, Andrea; Stolovitzky, Gustavo

doi:10.1186/s13059-014-0462-7

Table 1 Non-comprehensive list of important and current challenge efforts and platforms

From: Toward better benchmarking: challenge-based methods assessment in cancer genomics

Challenge	Scope	Assessment type	Organizers	Website
Assemblathon1&2	Sequence assembly	Objective scoring	UC Davis Genome Center	http://assemblathon.org/
CAFA	Protein function prediction	Objective scoring	Community collaboration	http://biofunctionprediction.org/node/8
CAGI	Systems biology	Objective scoring	UC Berkley/University of Maryland	http://genomeinterpretation.org/
CAPRI	Protein docking	Objective scoring	Community collaboration	http://www.ebi.ac.uk/msd-srv/capri/
CASP	Structure prediction	Objective scoring	Community collaboration	http://predictioncenter.org/
ChaLearn	Machine learning	Objective scoring	ChaLearn Organization (non-for profit)	http://www.chalearn.org/
CLARITY	Clinical genome interpretation	Objective scoring and evaluation by judges	Boston Children's Hospital	http://www.childrenshospital.org/research-and-innovation/research-initiatives/clarity-challenge
DREAM	Network inference and systems biology	Objective scoring	Community collaboration & Sage Bionetworks	https://www.synapse.org/#!Challenges:DREAM
FlowCAP	Flow cytometry analysis	Objective scoring	Community collaboration	http://flowcap.flowsite.org/
IGCG-TCGA DREAM Somatic Mutation Calling	Sequence analysis	Objective evaluation	Community collaboration & Sage Bionetworks	https://www.synapse.org/#!Synapse:syn312572
IMPROVER	Systems biology	Objective evaluation and crowd-verification	Phillip Morris International	https://sbvimprover.com/
Innocentive	Topics in various industries	Objective scoring and evaluation by judges	Commercial platform	http://www.innocentive.com/
Kaggle	Topics in various industries	Objective scoring and evaluation by judges	Commercial platform	http://www.kaggle.com/
RGASP	RNA-seq analyses	Objective scoring	European Bioinformatics Institute	http://www.gencodegenes.org/rgasp/
Sequence Squeeze	Sequence compression	Objective scoring and evaluation by judges	Pistoia Alliance	http://sequencesqueeze.org/
X-Prize	Technology	Evaluation by judges	X-Prize Organization (non-for-profit)	http://www.xprize.org/

The challenges were chosen based on relevance to cancer genomics or the representativeness of a type of challenge. Different challenges specialize in specific areas of research (see `Scope’), and may use different assessment types such as objective scoring against a gold standard, evaluation by judges, or community consensus (`crowd-verification’). Organizers can be researchers from specific institutions (such as universities or hospitals), a group of diverse researchers from academia and industry collaborating in the challenge organization (community collaboration), not-for-profit associations, or commercial platforms that run challenges as their business model (such as Innocentive and Kaggle). Initiatives such as CAFA, CAGI, CAPRI, CASP, ChaLearn, DREAM, FlowCAP and IMPROVER organize several challenges each year, and only the generic project is listed in this table, with the exception of DREAM, for which we also show the IGCG-TCGA DREAM Somatic Mutation Calling Challenge because of its relevance to this paper. More information about these efforts can be found on the listed websites.

Back to article page

ISSN: 1474-760X

Contact us

Submission enquiries: editorial@genomebiology.com
General enquiries: info@biomedcentral.com

Genome Biology

Contact us