Skip to main content

Table 1 Non-comprehensive list of important and current challenge efforts and platforms

From: Toward better benchmarking: challenge-based methods assessment in cancer genomics

Challenge Scope Assessment type Organizers Website
Assemblathon1&2 Sequence assembly Objective scoring UC Davis Genome Center http://assemblathon.org/
CAFA Protein function prediction Objective scoring Community collaboration http://biofunctionprediction.org/node/8
CAGI Systems biology Objective scoring UC Berkley/University of Maryland http://genomeinterpretation.org/
CAPRI Protein docking Objective scoring Community collaboration http://www.ebi.ac.uk/msd-srv/capri/
CASP Structure prediction Objective scoring Community collaboration http://predictioncenter.org/
ChaLearn Machine learning Objective scoring ChaLearn Organization (non-for profit) http://www.chalearn.org/
CLARITY Clinical genome interpretation Objective scoring and evaluation by judges Boston Children's Hospital http://www.childrenshospital.org/research-and-innovation/research-initiatives/clarity-challenge
DREAM Network inference and systems biology Objective scoring Community collaboration & Sage Bionetworks https://www.synapse.org/#!Challenges:DREAM
FlowCAP Flow cytometry analysis Objective scoring Community collaboration http://flowcap.flowsite.org/
IGCG-TCGA DREAM Somatic Mutation Calling Sequence analysis Objective evaluation Community collaboration & Sage Bionetworks https://www.synapse.org/#!Synapse:syn312572
IMPROVER Systems biology Objective evaluation and crowd-verification Phillip Morris International https://sbvimprover.com/
Innocentive Topics in various industries Objective scoring and evaluation by judges Commercial platform http://www.innocentive.com/
Kaggle Topics in various industries Objective scoring and evaluation by judges Commercial platform http://www.kaggle.com/
RGASP RNA-seq analyses Objective scoring European Bioinformatics Institute http://www.gencodegenes.org/rgasp/
Sequence Squeeze Sequence compression Objective scoring and evaluation by judges Pistoia Alliance http://sequencesqueeze.org/
X-Prize Technology Evaluation by judges X-Prize Organization (non-for-profit) http://www.xprize.org/
  1. The challenges were chosen based on relevance to cancer genomics or the representativeness of a type of challenge. Different challenges specialize in specific areas of research (see `Scope’), and may use different assessment types such as objective scoring against a gold standard, evaluation by judges, or community consensus (`crowd-verification’). Organizers can be researchers from specific institutions (such as universities or hospitals), a group of diverse researchers from academia and industry collaborating in the challenge organization (community collaboration), not-for-profit associations, or commercial platforms that run challenges as their business model (such as Innocentive and Kaggle). Initiatives such as CAFA, CAGI, CAPRI, CASP, ChaLearn, DREAM, FlowCAP and IMPROVER organize several challenges each year, and only the generic project is listed in this table, with the exception of DREAM, for which we also show the IGCG-TCGA DREAM Somatic Mutation Calling Challenge because of its relevance to this paper. More information about these efforts can be found on the listed websites.