Skip to content

How scMultiBench works

scMultiBench is a benchmark of single-cell multimodal integration methods. The multibench package is its Python API.

What scMultiBench covers

The benchmark groups datasets into four categories (category in the API) by how the data were measured: vertical, diagonal, mosaic and cross. The four integration categories defines each.

Details

Its datasets are real and simulated, and cover RNA, ADT and ATAC.

How it evaluates

The benchmark scores dimension reduction, batch correction, clustering and classification. In the package, grand_score combines the scores: on each dataset, each metric ranks a method's value among the stored methods' values. The mean of a method's ranks is min-max scaled, so that the best stored method gets 1 and the lowest 0, and your own rows are placed on that scale. grand_score is the mean over the datasets a method was scored on. It is a rank, not a quality score. For the full ranking per category, see the paper's decision tree (Fig. 6) or the interactive explorer.

Details

The metric panels are:

  • Dimension reduction, batch correction and clustering: scIB metrics such as kBET, iLISI, ASW, graph connectivity, ARI and NMI.
  • Classification: OCA, ACA, Spec, Sens and F1.
  • Every method: runtime and peak memory.

In the package, mtb.evaluate computes the scIB metrics and the classification metrics for an embedding.

Next steps