Task structure

Each task lives under src/tasks/<task_id> and contains a dataset processor, methods, control methods, and metrics.

Overview of a typical benchmarking workflow. Legend: grey rectangles are AnnData (.h5ad) files; purple parallelograms are Viash components.

Directory structure

Root path: src/tasks/<task_id>.

Dataset processor (src/process_dataset): Transforms a common dataset into task-specific files. In supervised tasks this typically yields a solution, a training set, and a test set. In unsupervised tasks it yields a solution and a masked dataset.
Control methods (src/control_methods): Control components with the same interface as regular methods that also receive the solution as input.
- A positive control achieves the best possible metric score and sets an upper performance bound.
- A negative control returns random or trivial output and sets a lower performance bound.
Methods (src/methods): Method components that produce output from the (training and) test dataset.
Metrics (src/metrics): Metric components that score a method output against the solution.
Benchmarking pipeline (src/workflows): A Nextflow pipeline defining the full benchmarking workflow.
File and component formats (src/api): Task-specific file format specifications (file_*.yaml), component interfaces (comp_*.yaml), task metadata (task_info.yaml), and a thumbnail (thumbnail.svg).
Resource generation scripts (create_datasets/resources.sh, create_datasets/test_resources.sh): Scripts for generating benchmarking and test resources.

For a step-by-step guide to creating a new task, see Create task. For the task template repository, see task_template.