Skip to content

Add a metric

A metric is a quantitative measure used to evaluate the performance of the different methods in solving a specific task problem.

This guide shows how to create a new Viash component for both Python and R. The Task template repo is used throughout; replace any occurrence of task_template with your task of interest.

Step 1: Create a new component

Section titled “Step 1: Create a new component”

Use the create_*_metric.sh script to start creating a new metric. Open the script and update the name parameter.

scripts/create_component/create_python_metric.sh
common/scripts/create_component \
  --name my_python_metric \
  --language python \
  --type metric

Run it from the repository root:

bash scripts/create_component/create_python_metric.sh 2>&1 | grep -v '^\[notice\]'
Check inputs
Check language
Check API file
Read API file
Create output dir
Create config
Create script
Done!

This creates a new folder at src/metrics/my_python_metric/:

src/metrics/my_python_metric/
    ├── script.py         Script for running the metric.
    ├── config.vsh.yaml   Config file for the metric.
    └── ...               Optional additional resources.

Change --name to a unique name for your metric. It must match the regex [a-z][a-z0-9_]* (snake_case).

  • A config file contains the component’s metadata and its runtime dependencies.
  • A script contains the code to run the metric.

Step 2: Fill in metadata

Section titled “Step 2: Fill in metadata”

The Viash config contains your metric’s metadata, the script used to run it, and its dependencies. The metrics config can define multiple metric values, each listed in info.metrics.

Generated config.vsh.yaml 
cat src/metrics/my_python_metric/config.vsh.yaml
# The API specifies which type of component this is.
# It contains specifications for:
#   - The input/output files
#   - Common parameters
#   - A unit test
__merge__: ../../api/comp_metric.yaml


# A unique identifier for your component (required).
# Can contain only lowercase letters or underscores.
name: my_python_metric






# Metadata for your component
info:
  metrics:
      # A unique identifier for your metric (required).
      # Can contain only lowercase letters or underscores.
    - name: my_python_metric
      # A relatively short label, used when rendering visualisarions (required)
      label: My Python Metric
      # A one sentence summary of how this metric works (required). Used when
      # rendering summary tables.
      summary: "FILL IN: A one sentence summary of this metric."
      # A multi-line description of how this component works (required). Used
      # when rendering reference documentation.
      description: |
        FILL IN: A (multi-line) description of how this metric works.
      # references:
      #   doi:
      #     - 10.1000/xx.123456.789
      #   bibtex:
      #     - |
      #       @article{foo,
      #         title={Foo},
      #         author={Bar},
      #         journal={Baz},
      #         year={2024}
      #       }
      links:
        # URL to the documentation for this metric (required).
        documentation: https://url.to/the/documentation
        # URL to the code repository for this metric (required).
        repository: https://github.com/organisation/repository
      # The minimum possible value for this metric (required)
      min: 0
      # The maximum possible value for this metric (required)
      max: 1
      # Whether a higher value represents a 'better' solution (required)
      maximize: true


# Component-specific parameters (optional)
# arguments:
#   - name: "--n_neighbors"
#     type: "integer"
#     default: 5
#     description: Number of neighbors to use.


# Resources required to run the component
resources:
  # The script of your component (required)
  - type: python_script
    path: script.py
  # Additional resources your script needs (optional)
  # - type: file
  #   path: weights.pt


engines:
  # Specifications for the Docker image for this component.
  - type: docker
    image: openproblems/base_python:1.0.0
    # Add custom dependencies here (optional). For more information, see
    # https://viash.io/reference/config/engines/docker/#setup .
    # setup:
    #   - type: python
    #     packages: numpy<2


runners:
  # This platform allows running the component natively
  - type: executable
  # Allows turning the component into a Nextflow module / pipeline.
  - type: nextflow
    directives:
      label: [midtime,midmem,midcpu]

Required metadata fields

Section titled “Required metadata fields”

Edit the info section of the config file:

  • .merge: Specifies which type of component this is. Defines input/output files, common parameters, and a unit test.
  • .name: A unique identifier. Can only contain lowercase letters, numbers, or underscores.
  • .label: A short, human-readable label for summary tables and visualizations.
  • .summary: A one-sentence summary of purpose and methodology.
  • .description: A longer description (one or more paragraphs) for reference documentation.

Step 3: Add dependencies

Section titled “Step 3: Add dependencies”

Each component has its own set of dependencies, since different components may have conflicting requirements.

Base images

Section titled “Base images”

Several pre-built base images are available in the OpenProblems Docker repository:

  • openproblems/base_python — base image for Python scripts.
  • openproblems/base_r — base image for R scripts.
  • openproblems/base_pytorch_nvidia — base image for PyTorch with NVIDIA GPU support.
  • openproblems/base_tensorflow_nvidia — base image for TensorFlow with NVIDIA GPU support.

Custom image

Section titled “Custom image”

If you use a custom image, add the following minimum setup to the engines section:

engines:
  - type: docker
    image: your_custom_image
    setup:
      - type: apt
        packages:
          - procps
      - type: python
        packages:
          - anndata~=0.10.0
          - scanpy~=1.10.0
          - pyyaml
          - requests
          - jsonschema
        github:
          - "openproblems-bio/core#subdirectory=packages/python/openproblems"

See the Viash dependency guide for more information.

After changing dependencies, rebuild the Docker container:

viash run src/metrics/my_python_metric/config.vsh.yaml -- ---setup cachedbuild

Step 4: Edit script

Section titled “Step 4: Edit script”

A component’s script typically has five sections:

a. Imports and libraries b. Argument values (the Viash placeholder block) c. Read input data d. Generate results e. Write output data to file

Generated script 
cat src/metrics/my_python_metric/script.py
import anndata as ad


## VIASH START
# Note: this section is auto-generated by viash at runtime. To edit it, make changes
# in config.vsh.yaml and then run `viash config inject config.vsh.yaml`.
par = {
  'input_solution': 'resources_test/.../solution.h5ad',
  'input_prediction': 'resources_test/.../prediction.h5ad',
  'output': 'output.h5ad'
}
meta = {
  'name': 'my_python_metric'
}
## VIASH END


print('Reading input files', flush=True)
input_solution = ad.read_h5ad(par['input_solution'])
input_prediction = ad.read_h5ad(par['input_prediction'])


print('Compute metrics', flush=True)
# metric_ids and metric_values can have length > 1
# but should be of equal length
uns_metric_ids = [ 'my_python_metric' ]
uns_metric_values = [ 0.5 ]


print("Write output AnnData to file", flush=True)
output = ad.AnnData(


)
output.write_h5ad(par['output'], compression='gzip')

a. Imports and libraries

Section titled “a. Imports and libraries”

Define which packages the metric needs at the top of the script. If you add a new package, also add it to the setup field in config.vsh.yaml (see Step 3).

b. Argument block

Section titled “b. Argument block”

The Viash code block enables you to run the script directly for prototyping. Everything between ## VIASH START and ## VIASH END is removed and replaced with a CLI argument parser when Viash builds the component.

The par dictionary holds all arguments defined in config.vsh.yaml (including those from the __merge__ file). When you add an argument to par, also add it to the arguments section of the config.

c. Read input data

Section titled “c. Read input data”

Read any input AnnData files passed to the component.

d. Generate results

Section titled “d. Generate results”

The core section: calculates the metric values from the input data.

e. Write output data to file

Section titled “e. Write output data to file”

Store the output in an AnnData object and write it to an .h5ad file. The format is specified by the API file referenced in the __merge__ field of the config.

Step 5: Try component

Section titled “Step 5: Try component”

Your component’s API file contains unit tests to check whether the component runs and whether the output has the correct format. Test your component with:

viash test src/metrics/my_python_metric/config.vsh.yaml

See Run tests for example output and common error messages.

You can also run your component on local files:

viash run src/metrics/my_python_metric/config.vsh.yaml -- \
  --input_prediction resources_test/task_template/cxg_mouse_pancreas_atlas/prediction.h5ad \
  --input_solution resources_test/task_template/cxg_mouse_pancreas_atlas/solution.h5ad \
  --output output.h5ad

Next steps

Section titled “Next steps”

If your component works, create a pull request.