Predict Modality
Predicting the profiles of one modality (e.g. protein abundance) from another (e.g. mRNA expression).
Experimental techniques to measure multiple modalities within the same single cell are increasingly becoming available. The demand for these measurements is driven by the promise to provide a deeper insight into the state of a cell. Yet, the modalities are also intrinsically linked. We know that DNA must be accessible (ATAC data) to produce mRNA (expression data), and mRNA in turn is used as a template to produce protein (protein abundance). These processes are regulated often by the same molecules that they produce: for example, a protein may bind DNA to prevent the production of more mRNA. Understanding these regulatory processes would be transformative for synthetic biology and drug target discovery. Any method that can predict a modality from another must have accounted for these regulatory processes, but the demand for multi-modal data shows that this is not trivial.
Leaderboard
Methods ranked by scaled overall mean. Each cell encodes a score from 0 to 1 by size and intensity.
QC: Normalisation Visualisation 8 plots
Per metric: points placed by control-anchored scaled score (x); dashed lines mark scaled 0 and 1 (worst/best control); the lower axis shows the raw score. Points beyond [-0.2, 1.2] are clamped to the edge as triangles. Hover a dot or line to highlight it and read details.
- MAElower better
- Mean pearson per cellhigher better
- Mean pearson per genehigher better
- Mean spearman per cellhigher better
- Mean spearman per genehigher better
- Overall pearsonhigher better
- Overall spearmanhigher better
- RMSElower better
QC: Indicator table 1 error11 warnings
Automated checks on the benchmark run and its results: missing values, score scaling, metric ranges and similar. Errors are high-severity issues that usually need a maintainer's attention; warnings are lower-severity signals. Findings that are expected for this task are listed separately as silenced.
1 high-severity issue need review. 182 of 194 checks passed.
- error Raw results Dataset 'openproblems_neurips2022/pbmc_multiome/swap' %missing
Percentage of missing results should be less than 10%. Task id: task_predict_modality dataset id: openproblems_neurips2022/pbmc_multiome/swap Percentage missing: 36%
Show 11 warnings
- warning Method info Pct 'paper_reference' missing
Method metadata field 'paper_reference' should be defined Task id: task_predict_modality Field: paper_reference
- warning Metric info Pct 'paper_reference' missing
Metric metadata field 'paper_reference' should be defined Task id: task_predict_modality Field: paper_reference
- warning Dataset info Pct 'task_id' missing
Dataset metadata field 'task_id' should be defined Task id: task_predict_modality Field: task_id
- warning Raw results Method 'zeros' %missing
Percentage of missing results should be less than 10%. Task id: task_predict_modality method id: zeros Percentage missing: 25%
- warning Raw results Method 'guanlab_dengkw_pm' %missing
Percentage of missing results should be less than 10%. Task id: task_predict_modality method id: guanlab_dengkw_pm Percentage missing: 25%
- warning Scaling Worst score lmds_irlba_rf overall_pearson
Method lmds_irlba_rf performs much worse than baselines. Task id: task_predict_modality Method id: lmds_irlba_rf Metric id: overall_pearson Worst score: -2.4102%
- warning Raw results Metric 'overall_pearson' %missing
Percentage of missing results should be less than 10%. Task id: task_predict_modality Metric id: overall_pearson Percentage missing: 17%
- warning Raw results Metric 'overall_spearman' %missing
Percentage of missing results should be less than 10%. Task id: task_predict_modality Metric id: overall_spearman Percentage missing: 17%
- warning Raw results Method 'knnr_py' %missing
Percentage of missing results should be less than 10%. Task id: task_predict_modality method id: knnr_py Percentage missing: 12%
- warning Raw results Method 'lm' %missing
Percentage of missing results should be less than 10%. Task id: task_predict_modality method id: lm Percentage missing: 12%
- warning Raw results Dataset 'openproblems_neurips2022/pbmc_multiome/normal' %missing
Percentage of missing results should be less than 10%. Task id: task_predict_modality dataset id: openproblems_neurips2022/pbmc_multiome/normal Percentage missing: 14%
Method info 3
A kernel ridge regression method with RBF kernel.
This is a solution developed by Team Guanlab - dengkw in the Neurips 2021 competition to predict one modality from another using kernel ridge regression (KRR) with RBF kernel. Truncated SVD is applied on the combined training and test data from modality 1 followed by row-wise z-score normalization on the reduced matrix. The truncated SVD of modality 2 is predicted by training a KRR model on the normalized training matrix of modality 1. Predictions on the normalized test matrix are then re-mapped to the modality 2 feature space via the right singular vectors.
K-nearest neighbor regression in Python.
Linear model regression.
A linear model regression method.
Control method info 4
Returns the mean expression value per gene.
Returns random training profiles.
Returns the ground-truth solution.
Returns a prediction consisting of all zeros.
Metric info 8
The mean absolute error.
The average difference between the expression values and the predicted expression values.
The mean of the pearson values of per-cell expression value vectors.
The mean of the pearson values of per-gene expression value vectors.
The mean of the spearman values of per-cell expression value vectors.
The mean of the spearman values of per-gene expression value vectors.
The mean of the pearson values of vectorized expression matrices.
The mean of the spearman values of vectorized expression matrices.
The root mean squared error.
The square root of the mean of the square of all of the error.
Dataset info 6
Single-cell CITE-Seq (GEX+ADT) data collected from bone marrow mononuclear cells of 12 healthy human donors.
Single-cell CITE-Seq data collected from bone marrow mononuclear cells of 12 healthy human donors using the 10X 3 prime Single-Cell Gene Expression kit with Feature Barcoding in combination with the BioLegend TotalSeq B Universal Human Panel v1.0. The dataset was generated to support Multimodal Single-Cell Data Integration Challenge at NeurIPS 2021. Samples were prepared using a standard protocol at four sites. The resulting data was then annotated to identify cell types and remove doublets. The dataset was designed with a nested batch layout such that some donor samples were measured at multiple sites with some donors measured at a single site.
Single-cell CITE-Seq (GEX+ADT) data collected from bone marrow mononuclear cells of 12 healthy human donors.
Single-cell CITE-Seq data collected from bone marrow mononuclear cells of 12 healthy human donors using the 10X 3 prime Single-Cell Gene Expression kit with Feature Barcoding in combination with the BioLegend TotalSeq B Universal Human Panel v1.0. The dataset was generated to support Multimodal Single-Cell Data Integration Challenge at NeurIPS 2021. Samples were prepared using a standard protocol at four sites. The resulting data was then annotated to identify cell types and remove doublets. The dataset was designed with a nested batch layout such that some donor samples were measured at multiple sites with some donors measured at a single site.
Single-cell Multiome (GEX+ATAC) data collected from bone marrow mononuclear cells of 12 healthy human donors.
Single-cell CITE-Seq data collected from bone marrow mononuclear cells of 12 healthy human donors using the 10X Multiome Gene Expression and Chromatin Accessibility kit. The dataset was generated to support Multimodal Single-Cell Data Integration Challenge at NeurIPS 2021. Samples were prepared using a standard protocol at four sites. The resulting data was then annotated to identify cell types and remove doublets. The dataset was designed with a nested batch layout such that some donor samples were measured at multiple sites with some donors measured at a single site.
Single-cell Multiome (GEX+ATAC) data collected from bone marrow mononuclear cells of 12 healthy human donors.
Single-cell CITE-Seq data collected from bone marrow mononuclear cells of 12 healthy human donors using the 10X Multiome Gene Expression and Chromatin Accessibility kit. The dataset was generated to support Multimodal Single-Cell Data Integration Challenge at NeurIPS 2021. Samples were prepared using a standard protocol at four sites. The resulting data was then annotated to identify cell types and remove doublets. The dataset was designed with a nested batch layout such that some donor samples were measured at multiple sites with some donors measured at a single site.
Single-cell CITE-Seq (GEX+ADT) data collected from bone marrow mononuclear cells of 12 healthy human donors.
Single-cell CITE-Seq data collected from bone marrow mononuclear cells of 12 healthy human donors using the 10X 3 prime Single-Cell Gene Expression kit with Feature Barcoding in combination with the BioLegend TotalSeq B Universal Human Panel v1.0. The dataset was generated to support Multimodal Single-Cell Data Integration Challenge at NeurIPS 2022. Samples were prepared using a standard protocol at four sites. The resulting data was then annotated to identify cell types and remove doublets. The dataset was designed with a nested batch layout such that some donor samples were measured at multiple sites with some donors measured at a single site.
Single-cell CITE-Seq (GEX+ADT) data collected from bone marrow mononuclear cells of 12 healthy human donors.
Single-cell CITE-Seq data collected from bone marrow mononuclear cells of 12 healthy human donors using the 10X 3 prime Single-Cell Gene Expression kit with Feature Barcoding in combination with the BioLegend TotalSeq B Universal Human Panel v1.0. The dataset was generated to support Multimodal Single-Cell Data Integration Challenge at NeurIPS 2022. Samples were prepared using a standard protocol at four sites. The resulting data was then annotated to identify cell types and remove doublets. The dataset was designed with a nested batch layout such that some donor samples were measured at multiple sites with some donors measured at a single site.
- Chai, T., & Draxler, R. R. (2014). Root mean square error (RMSE) or mean absolute error (MAE)? 10.5194/gmdd-7-1525-2014 ↗
- Fix, E., & Hodges, J. L. (1989). Discriminatory Analysis. Nonparametric Discrimination: Consistency Properties. International Statistical Review / Revue Internationale de Statistique, 57(3), 238. 10.2307/1403797 ↗
- KENDALL, M. G. (1938). A new measure of rank correlation. Biometrika, 30(1–2), 81–93. 10.1093/biomet/30.1-2.81 ↗
- Lance, C., Luecken, M. D., Burkhardt, D. B., Cannoodt, R., Rautenstrauch, P., Laddach, A., Ubingazhibov, A., Cao, Z.-J., Deng, K., Khan, S., Liu, Q., Russkikh, N., Ryazantsev, G., Ohler, U., Pisco, A. O., Bloom, J., Krishnaswamy, S., & Theis, F. J. (2022). Multimodal single cell data integration challenge: results and lessons learned. bioRxiv. 10.1101/2022.04.11.487796 ↗
- Pearson, K. (1895). VII. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58(347–352), 240–242. 10.1098/rspl.1895.0041 ↗
- Wilkinson, G. N., & Rogers, C. E. (1973). Symbolic Description of Factorial Models for Analysis of Variance. Applied Statistics, 22(3), 392. 10.2307/2346786 ↗