Cell-Cell Communication
Detect interactions between ligands and target cell types
The growing availability of single-cell data has sparked an increased interest in the inference of cell-cell communication (CCC), with an ever-growing number of computational tools developed for this purpose.
Different tools propose distinct preprocessing steps with diverse scoring functions, that are challenging to compare and evaluate. Furthermore, each tool typically comes with its own set of prior knowledge. To harmonize these, Dimitrov et al, 2022 recently developed the LIANA framework, which was used as a foundation for this task.
The challenges in evaluating the tools are further exacerbated by the lack of a gold standard to benchmark the performance of CCC methods. In an attempt to address this, Dimitrov et al use alternative data modalities, including the spatial proximity of cell types and downstream cytokine activities, to generate an inferred ground truth. However, these modalities are only approximations of biological reality and come with their own assumptions and limitations. In time, the inclusion of more datasets with known ground truth interactions will become available, from which the limitations and advantages of the different CCC methods will be better understood.
This subtask evaluates the methods' ability to predict interactions, the corresponding of cytokines of which, are inferred to be active in the target cell types. This subtask focuses on the prediction of interactions from steady-state, or single-context, single-cell data.
Leaderboard
Methods ranked by scaled overall mean. Each cell encodes a score from 0 to 1 by size and intensity.
QC: Normalisation Visualisation 2 plots
Per metric: points placed by control-anchored scaled score (x); dashed lines mark scaled 0 and 1 (worst/best control); the lower axis shows the raw score. Points beyond [-0.2, 1.2] are clamped to the edge as triangles. Hover a dot or line to highlight it and read details.
- Odds Ratiohigher better
- Precision-recall AUChigher better
QC: Indicator table all clear
Automated checks on the benchmark run and its results: missing values, score scaling, metric ranges and similar. Errors are high-severity issues that usually need a maintainer's attention; warnings are lower-severity signals. Findings that are expected for this task are listed separately as silenced.
No high-severity issues. 109 of 109 checks passed.
Method info 7
CellPhoneDBv2 calculates a mean of ligand-receptor expression as a measure of interaction magnitude, along with a permutation-based p-value as a measure of specificity. Here, we use the former to prioritize interactions, subsequent to filtering according to p-value less than 0.05.
Connectome uses the product of ligand-receptor expression as a measure of magnitude, and the average of the z-transformed expression of ligand and receptor as a measure of specificity.
logFC (implemented in LIANA and inspired by iTALK) combines both expression and magnitude, and represents the average of one-versus-the-rest log2-fold change of ligand and receptor expression per cell type.
RobustRankAggregate generates a consensus rank of all methods implemented in LIANA providing either specificity or magnitude scores.
NATMI uses the product of ligand-receptor expression as a measure of magnitude. As a measure of specificity, NATMI proposes $specificity.edge = \frac{l}{l_s} \cdot \frac{r}{r_s}$; where $l$ and $r$ represent the average expression of ligand and receptor per cell type, and $l_s$ and $r_s$ represent the sums of the average ligand and receptor expression across all cell types. We use its specificity measure, as recommended by the authors for single-context predictions.
SingleCellSignalR provides a magnitude score as $LRscore = \frac{\sqrt{lr}}{\mu+\sqrt{lr}}$; where $l$ and $r$ are the average ligand and receptor expression per cell type, and $\mu$ is the mean of the expression matrix.
RobustRankAggregate generates a consensus rank of all methods implemented in LIANA providing either specificity or magnitude scores.
Control method info 2
Random generation of cell-cell communication events by random selection of ligand, receptor, source, target, and score
Perfect prediction of cell-cell communication events from target data
Metric info 2
The odds ratio represents the ratio of true and false positives within a set of prioritized interactions (top ranked hits) versus the same ratio for the remainder of the interactions. Thus, in this scenario odds ratios quantify the strength of association between the ability of methods to prioritize interactions and those interactions assigned to the positive class.
Area under the precision-recall curve for the binary classification task predicting interactions.
Dataset info 1
Human breast cancer atlas (Wu et al., 2021), with cytokine activities, inferred using a multivariate linear model with cytokine-focused signatures, as assumed true cell-cell communication (Dimitrov et al., 2022). 42512 cells x 28078 features with 29 cell types from 10 patients
- Bland, J. M. (2000). Statistics Notes: The odds ratio. BMJ, 320(7247), 1468–1468. 10.1136/bmj.320.7247.1468 ↗
- Cabello-Aguilar, S., Alame, M., Kon-Sun-Tack, F., Fau, C., Lacroix, M., & Colinge, J. (2020). SingleCellSignalR: inference of intercellular networks from single-cell transcriptomics. Nucleic Acids Research, 48(10), e55–e55. 10.1093/nar/gkaa183 ↗
- Open Problems for Single Cell Analysis Consortium. (2022). Open Problems. link ↗
- Davis, J., & Goadrich, M. (2006). The relationship between Precision-Recall and ROC curves. Proceedings of the 23rd International Conference on Machine Learning - ICML ’06. 10.1145/1143844.1143874 ↗
- Dimitrov, D., Türei, D., Garrido-Rodriguez, M., Burmedi, P. L., Nagai, J. S., Boys, C., Flores, R. O. R., Kim, H., Szalai, B., Costa, I. G., Valdeolivas, A., Dugourd, A., & Saez-Rodriguez, J. (2022). Comparison of methods and resources for cell-cell communication inference from single-cell RNA-Seq data. Nature Communications, 13(1). 10.1038/s41467-022-30755-0 ↗
- Efremova, M., Vento-Tormo, M., Teichmann, S. A., & Vento-Tormo, R. (2020). CellPhoneDB: inferring cell–cell communication from combined expression of multi-subunit ligand–receptor complexes. Nature Protocols, 15(4), 1484–1506. 10.1038/s41596-020-0292-x ↗
- Hou, R., Denisenko, E., Ong, H. T., Ramilowski, J. A., & Forrest, A. R. R. (2020). Predicting cell-to-cell communication networks using NATMI. Nature Communications, 11(1). 10.1038/s41467-020-18873-z ↗
- Raredon, M. S. B., Yang, J., Garritano, J., Wang, M., Kushnir, D., Schupp, J. C., Adams, T. S., Greaney, A. M., Leiby, K. L., Kaminski, N., Kluger, Y., Levchenko, A., & Niklason, L. E. (2022). Computation and visualization of cell–cell signaling topologies in single-cell systems data using Connectome. Scientific Reports, 12(1). 10.1038/s41598-022-07959-x ↗