How scMultiBench works¶
scMultiBench is a benchmark of single-cell multimodal integration methods. The
multibench package is its Python API.
What scMultiBench covers¶
The benchmark groups datasets into four categories (category in the API)
by how the data were measured: vertical, diagonal, mosaic and cross.
The four integration categories
defines each.
Details
Its datasets are real and simulated, and cover RNA, ADT and ATAC.
How it evaluates¶
The benchmark scores dimension reduction, batch correction, clustering and
classification. In the package, grand_score combines the scores: on each
dataset, each metric ranks a method's value among the stored methods' values.
The mean of a method's ranks is min-max scaled, so that the best stored method
gets 1 and the lowest 0, and your own rows are placed on that scale.
grand_score is the mean over the datasets a method was scored on. It is a
rank, not a quality score. For the full ranking per category, see the paper's
decision tree (Fig. 6) or the interactive explorer.
Details
The metric panels are:
- Dimension reduction, batch correction and clustering: scIB metrics such as kBET, iLISI, ASW, graph connectivity, ARI and NMI.
- Classification: OCA, ACA, Spec, Sens and F1.
- Every method: runtime and peak memory.
In the package, mtb.evaluate computes the scIB metrics and the
classification metrics for an embedding.
Next steps¶
- Install:
pip install multibench-sc, thenimport multibench. See Installation. - First run: Quickstart.
- Tutorials: vertical, diagonal, mosaic, and cross.
- Explore: interactive Shiny explorer.
- API:
multibench.