Table 2 Some advantages and limitations of challenge-based methods assessment, along with barriers to participation in them

Reduction of over-fitting

Narrower scope compared to traditional open-ended research

Incentives not strong enough to promote participation

Benchmarking individual methods

Ground truth needed for objective scoring

No funding available to support time spent participating in challenges

Impartial comparison across methods using same datasets

Mostly limited to computational approaches

Fatigue resulting from many ongoing challenges

Fostering collaborative work, including code sharing

Requires data producers to share their data before publication

Time assigned by organizers to solve a difficult challenge question may be too short

Acceleration of research

Sufficient amount of high-quality data needed for meaningful results

Lack of computing capabilities

Enhancing data access and impact

Large number of participants not always available

New data modality or datasets that are too complex or too big poses entry barrier

Determination of problem solvability

Challenge questions may not be solvable with data at hand

Challenge questions not interesting or impactful enough

Tapping the `Wisdom of Crowds’

Traditional grant mechanisms not adequate to fund challenge efforts

Cumbersome approvals to acquire sensitive datasets

Objective assessment

Difficulties to distribute datasets with sensitive information

Standardizes experimental design