Dataset workflows

The dataset processing pipeline uses dataset loaders to create raw dataset files. Raw datasets are then processed through normalization and common processing steps to produce common dataset files used across multiple benchmarking tasks.

Overview of the dataset processing workflow. Legend: grey rectangles are AnnData (.h5ad) files; purple parallelograms are Viash components.

Directory structure

Dataset file and component formats (src/api): Specifications for dataset file formats and component interfaces. See the datasets README for the full API reference.
Dataset loaders (src/datasets/loaders): Components to load and format datasets from various sources. See Create a dataset loader for how to add a new one.
Dataset normalization (src/datasets/normalization): Various dataset normalization methods (Log CP10k, L1 sqrt, Log scran pooling, Sqrt CP10k).
Dataset processors (src/datasets/processors): Components for processing datasets — HVG selection, PCA, KNN graph computation, and dataset subsetting.
Resource generation scripts (src/common/resources_scripts): Scripts for generating datasets using the loaders, normalization methods, and processors.
Test resource generation scripts (src/common/resources_test_scripts): Scripts for generating test resources.

For the full pipeline documentation including component API specifications, see the datasets repository README.