Task structure

An OpenProblems benchmarking task is a self-contained repository that evaluates methods on a biological problem. Each task contains four types of components: data processors, methods, control methods, and metrics. These components communicate through AnnData (.h5ad) files with a well-defined format.

Expand

The structure of an OpenProblems task. Legend: grey rectangles are AnnData (.h5ad) files; purple parallelograms are Viash components.

The structure of an OpenProblems task. Legend: grey rectangles are AnnData (.h5ad) files; purple parallelograms are Viash components.

File and component formats (src/api)

The src/api/ directory defines the interfaces shared across all components in the task. It contains two types of YAML files:

• file_*.yaml - describe the format of each AnnData file (layers, obs columns, var columns, obsm slots, uns entries). Every field that a component reads or writes must be declared here.
• comp_*.yaml - describe the interface of each component type (namespace, arguments, a reference to the unit test script). Each method, control method, and metric imports exactly one comp_*.yaml via __merge__.

Keeping the format definitions separate from the component definitions means that changing a file format automatically propagates to every component and to the generated documentation.

Data processors (src/data_processors)

A data processor transforms a common OpenProblems dataset into one or more task-specific AnnData files. It is the entry point of the task pipeline.

Typical outputs include a training set, a test set, and a held-out solution file. By separating the solution from the inputs, the processor guarantees that methods cannot access ground-truth labels during inference.

Methods (src/methods)

Method components implement candidate solutions to the task. Each method reads the task-specific input files produced by the data processor and writes a prediction or output file. Methods must conform to the interface defined in src/api/comp_method.yaml and must not read the solution file.

Control methods (src/control_methods)

Control methods have the same input/output interface as methods but may also receive the solution file. They serve as reference points for interpreting benchmark results:

• A positive control uses the solution directly to produce a near-perfect score. It establishes the upper bound a real method could theoretically achieve.
• A negative control produces a random or trivially simple output. It establishes the lower bound and helps detect degenerate metrics.

Every task should include at least one positive and one negative control.

Metrics (src/metrics)

Metric components measure how well a method’s output matches the solution. Each metric reads both the method output and the solution file and writes a score file containing one or more scalar values. Metrics must conform to the interface defined in src/api/comp_metric.yaml.

Benchmarking pipeline (src/workflows)

The src/workflows/ directory contains a Nextflow workflow that wires all components together into a full benchmark run. The workflow calls the data processor, fans out across all methods and control methods in parallel, and then evaluates each output with all metrics. Results are aggregated into a single score table.

See Create Workflow for instructions on setting up the pipeline.