A living benchmark for single-cell analysis.
Open Problems turns the hardest computational challenges in single-cell biology into formal, reproducible benchmarks, and continuously ranks the methods that solve them. Here's the idea, the machinery, and why it matters.
Make method choice in single-cell analysis an evidence-based decision.
Open Problems exists so that researchers can answer one question with confidence: for my data and my task, which method should I actually use? We turn that question into open, living benchmarks the whole field can trust and extend.
Openness
Everything we make (code, data, results) is public and inspectable.
Collaboration
Built by 50+ institutions working in the open, together.
Reproducibility
Containerized components and declarative workflows by construction.
Standardization
A strict task API keeps every comparison fair.
Community
Anyone can contribute a method, metric, dataset or task.
Rigor
Quantitative metrics and controls, never vibes.
Single-cell benchmarking is fragmented.
Single-cell data is large, sparse and noisy, and ground truth is scarce. A fast-growing ecosystem of methods has emerged, but the field often can't tell which one is best for a given job.
published for single-cell analysis as of Feb 2024; tools are created faster than the field can evaluate them.
in datasets and metrics between independent benchmarks of the same task, so results rarely agree.
of batch integration each used different data and metrics, and each named a different best method.
Source: Luecken et al., Nature Biotechnology (2025). Counts are point-in-time.
Borrow the common task framework from machine learning.
Progress accelerates when a field agrees on shared tasks, gold-standard data, quantitative metrics and a public leaderboard. It worked for computer vision and NLP; Open Problems brings it to single-cell biology.
Every challenge becomes a task.
A task defines a strict input/output API. Within it, every method runs on every dataset and is scored by every metric, then ranked by a scaled overall mean.
Datasets
Gold-standard public data in a ready-to-use AnnData format, with a known biological solution.
Methods
Components implementing the task API: given the input, produce an output.
Metrics
Quantitative scores comparing a method's output against the solution.
Positive controls (near-best by construction) and negative controls (naive or random) set the upper and lower bounds. Every real method should beat the negative control and trail the positive one, and the controls double as quality control for the metrics themselves.
Reproducible by construction.
Open Problems deliberately separates scientific contribution from infrastructure plumbing. Contributors write a script; the platform handles containers, execution and CI.
Viash
Wraps Python / R / Bash scripts into modular, Dockerized components, with no deep Docker or Nextflow expertise required.
Nextflow
Executes benchmark and dataset workflows portably across laptop, HPC and cloud.
AnnData
The standardized I/O format (CELLxGENE schema + OP metadata), interoperable across R and Python.
Docker + ghcr.io
A pinned image per component guarantees reproducibility and avoids dependency conflicts.
GitHub Actions
Continuous integration runs unit tests and builds containers on every contribution.
Seqera Platform
Orchestrates runs at scale on cloud backends (e.g. AWS Batch + S3), portable across providers.
What a living benchmark gives you.
Continuity
Leaderboards update as methods, datasets and metrics are added; they never freeze at publication.
Standardization
A strict component API and shared datasets prevent the custom-tuning performance inflation seen in one-off comparisons.
Reproducibility
Containers, versioned components, declarative workflows and built-in QC reports: reproducible by construction.
Extensibility
Modular components and a shared dataset/compute core cut boilerplate, so contributors focus on the science.
Neutrality
An independent, community-governed effort, not a single lab promoting its own method.
A two-way bridge
Reconciles ML culture with bioinformatics culture, lowering the barrier for AI researchers to contribute to genomics.
For newcomers
You don't need to be a single-cell expert to contribute. Pick a task, scaffold a component with one command, and your method is benchmarked against the field on shared data, fairly and automatically.
- 1 Browse the benchmarks and pick a task that interests you.
- 2 Scaffold a method with Viash (Python, R or Bash).
- 3 Open a pull request; CI runs the benchmark for you.
For funders
Open Problems is independent, reproducible and community-governed, backed by the Chan Zuckerberg Initiative as part of the Human Cell Atlas Data Ecosystem. It de-risks method evaluation for the whole field.
- Neutral. Governed by a community, not a vendor, so results are credible.
- Durable. A living platform that compounds in value, not a one-off paper.
- Efficient. Shared infrastructure amortizes benchmarking cost across the field.
Read more, and cite the project.
If you use Open Problems, please cite the platform paper. To reference a specific dataset or competition, cite the corresponding work below.
Defining and benchmarking open problems in single-cell analysis
Malte D. Luecken, Scott Gigante, Daniel B. Burkhardt et al.
@article{luecken2025openproblems,
title = {Defining and benchmarking open problems in single-cell analysis},
author = {Malte D. Luecken and Scott Gigante and Daniel B. Burkhardt and others},
year = {2025},
doi = {10.1038/s41587-025-02694-w}
} A sandbox for prediction and integration of DNA, RNA, and proteins in single cells
Malte Luecken, Daniel Burkhardt, Robrecht Cannoodt et al.
@inproceedings{luecken2021neurips,
title = {A sandbox for prediction and integration of DNA, RNA, and proteins in single cells},
author = {Malte Luecken and Daniel Burkhardt and Robrecht Cannoodt and Christopher Lance and Aditi Agrawal and Hananeh Aliee and Ann Chen and Louise Deconinck and Angela Detweiler and Alejandro Granados and Shelly Huynh and Laura Isacco},
year = {2021}
} Multimodal single cell data integration challenge: results and lessons learned
Christopher Lance, Malte D. Luecken, Daniel B. Burkhardt et al.
@article{lance2022multimodal,
title = {Multimodal single cell data integration challenge: results and lessons learned},
author = {Christopher Lance and Malte D. Luecken and Daniel B. Burkhardt and Robrecht Cannoodt and Pia Rautenstrauch and Anna Laddach and Aidyn Ubingazhibov and Zhi-Jie Cao and Kaiwen Deng and Sumeer Khan and Qiao Liu and Nikolay Russkikh},
year = {2022},
doi = {10.1101/2022.04.11.487796}
}