Skip to main content

Table 2 Some advantages and limitations of challenge-based methods assessment, along with barriers to participation in them

From: Toward better benchmarking: challenge-based methods assessment in cancer genomics

Advantages Limitations Participation barriers
Reduction of over-fitting Narrower scope compared to traditional open-ended research Incentives not strong enough to promote participation
Benchmarking individual methods Ground truth needed for objective scoring No funding available to support time spent participating in challenges
Impartial comparison across methods using same datasets Mostly limited to computational approaches Fatigue resulting from many ongoing challenges
Fostering collaborative work, including code sharing Requires data producers to share their data before publication Time assigned by organizers to solve a difficult challenge question may be too short
Acceleration of research Sufficient amount of high-quality data needed for meaningful results Lack of computing capabilities
Enhancing data access and impact Large number of participants not always available New data modality or datasets that are too complex or too big poses entry barrier
Determination of problem solvability Challenge questions may not be solvable with data at hand Challenge questions not interesting or impactful enough
Tapping the `Wisdom of Crowds’ Traditional grant mechanisms not adequate to fund challenge efforts Cumbersome approvals to acquire sensitive datasets
Objective assessment Difficulties to distribute datasets with sensitive information  
Standardizes experimental design