Skip to content

Score an embedding

mtb.evaluate scores an embedding against cell-type labels with the paper's metrics and returns them as a long table. An embedding is a table with one row per cell, like adata.obsm["X_pca"].

Summary

import scanpy as sc
import multibench as mtb

mtb.data.fetch("D11")
mtb.data.write_example("D11")     # my_file.h5ad: cell types in obs["celltype"]
adata = sc.read_h5ad("my_file.h5ad")
sc.pp.normalize_total(adata)
sc.pp.log1p(adata)
sc.pp.pca(adata)                  # the embedding to score: adata.obsm["X_pca"]
scores = mtb.evaluate(adata, labels="celltype", obsm="X_pca", method="PCA",
                      category="vertical", metrics="clustering")
print(scores)

Choose the metrics

metrics= takes a family ("clustering", "batch", "classification", "all"), a metric code, or a list of codes and families. The default computes every metric your inputs allow except kBET.

mtb.evaluate(adata, labels="celltype", obsm="X_pca", metrics="cLISI")
mtb.evaluate(adata, labels="celltype", obsm="X_pca", metrics=["ARI", "NMI"])

The protocol is the paper's. ARI, NMI, ASW, iASW and iF1 score Leiden clusters searched to the number of cell types. ARI_batch and NMI_batch score Leiden clusters searched to the number of batches.

Details

The search bisects the Leiden resolution on [0, 3] for at most 30 steps until the clustering has the number of clusters wanted. When no step gets there, the last step's clusters are scored. The clusters come from scanpy's default 15-neighbour graph of the embedding, or of its first 50 principal components when it is wider, as in the paper. cLISI, iLISI and GC use the 15-neighbour graph of the whole embedding.

A selection without the seven metrics above skips the search, such as metrics="cLISI".

kBET is computed only when a list names it, and can take hours on large datasets.


Inputs

Pass the embedding and one cell-type label per cell, in the embedding's row order. Both can be paths or in-memory objects. The reference lists the accepted forms.

# embedding and labels in one AnnData
mtb.evaluate(adata, labels="celltype", obsm="X_pca")

A column is named with or without obs:, so labels="obs:celltype" is the same. To score a method's output, use evaluate(dataset=, category=, method=) and no labels=: the embedding.h5 that run writes names its cells, and evaluate matches the label files to them.

A missing label is not a cell type: evaluate raises when a label is missing, and so does a missing batch. Score the labelled cells, as adata[adata.obs["celltype"].notna()].

Details

The labels of a dataset folder, in its cell order:

import pandas as pd
folder = mtb.data.fetch("D52mini")
order = {"rna": 0, "adt": 1, "atac": 2}
files = sorted(folder.glob("*cty*.csv"),
               key=lambda f: (order.get(f.stem.split("_")[0], -1), len(f.stem), f.stem))
labels = pd.concat([pd.read_csv(f).iloc[:, 0] for f in files]).to_numpy()

They follow the folder's cells as mtb.io.read reads them: RNA before ADT before ATAC, batches by number.

labels takes a CSV path with a header row (column x, or the only column), a list of CSV paths stacked in that order, a 1-D array or Series, or the name of an .obs column. batch takes the same forms.

With an AnnData, labels and batch can name .obs columns, and obsm= names the embedding. A MuData works the same way, with the joint embedding in its .obsm, and "rna:celltype" names a column of one modality.

A label is missing when it is NaN, None, blank text, or text that a label file reads as missing, such as NA, nan or null.

When the output has cell ids, the ids decide. An AnnData, an .h5ad file, a DataFrame indexed by cell id and a CSV embedding whose first column holds the barcodes have them. A Series indexed by cell id, or a CSV whose first column holds the cell ids, is then aligned by id. Ids of other cells are left out. Where barcodes repeat across a dataset's label files, a cell id is the label file and the barcode, as cty2:AAAC..., the way res.embedding(m) names cells.

A label file without cell ids in a dataset folder takes the cell ids of the matrix next to it. Any other file or Series without cell ids is refused: add the barcodes, or pass .to_numpy() to match by position.

A bare array has no cell ids. A Series or a CSV is then matched by position, with a warning when it holds cell ids. Plain arrays and lists are always matched by position.

A run's embedding.h5 holds the name of each cell (the dataset barcodes), so dataset= matches the label files to its cells whatever method= says. In a run folder written before 0.4.0, the run's record names the method that made the file. For an embedding without cell names, dataset= reads the cell order of method= when it names a package method, also as method="StabMap (this run)", else the folder's order: numbered files ascending, and rna before adt before atac. StabMap puts its reference batch first, which is cty3 on D52. Without category=, a benchmark dataset gives it. On a folder of your own, a package method needs it. A list of label files is stacked in the order given.

Batch metrics

Batch metrics also need one batch per cell. Pass batch=, or dataset= of a multi-batch dataset folder without labels: each of its label files is then a batch. Give the category and the method too, so the files follow the rows of the method's output. A wrong order gives wrong scores without an error.

mtb.data.fetch("D52mini")
mtb.data.write_example("D52mini", "d52")   # d52/my_file.h5ad: donors in obs["batch"]
cross = sc.read_h5ad("d52/my_file.h5ad")
sc.pp.normalize_total(cross)
sc.pp.log1p(cross)
sc.pp.pca(cross)
mtb.evaluate(cross, labels="celltype", batch="batch", obsm="X_pca", metrics="all")
Details

ARI_batch and NMI_batch are 1 minus the absolute ARI and NMI between the batch clusters and the batches, so higher means better mixed batches.

Without batch=, run uses each cell's label file as its batch, so batches inside one file, such as donors, are not scored. Pass batch= to run, or re-score saved outputs with the real batch vector, mtb.load_run("out/").rescore(batch=...). No method is re-run.

Classification metrics

With two or more batches, evaluate also predicts each cell's type from another batch and scores the predictions: OCA, ACA, Spec, Sens and F1. The default adds them unless category="vertical".

mtb.evaluate(cross, labels="celltype", batch="batch", obsm="X_pca",
             metrics="classification")
Details

Each batch in turn is the reference that a small neural network learns from, as in the paper. The scores are the mean over the references, as the paper reports them.

They need the method's embedding. A graph method has none, and run leaves them out for it. Where they cannot be computed, metrics="classification" raises. "all" gives NaN with a warning, and so does the default, except on data in one batch, where it leaves them out.

For vertical data in several batches, ask for them with metrics="classification".


The result

evaluate returns the long table: one row per metric, with the columns metric, value, method, dataset, category, source and scored_with. method=, dataset= and category= fill their columns, and source reads user. A metric that cannot run on your machine is NaN, with a warning.

scores.to_csv("mine.csv", index=False)
scores.pivot(index="method", columns="metric", values="value")

Add your own method concatenates it with the stored results.

What each metric measures

mtb.catalog.metrics() returns this table. The details below give the exact computation.

Metric Family Description
ARI clustering Adjusted Rand Index: how well the Leiden clusters, one per cell type, match the cell-type labels. About 0 = random, 1 = identical, higher = better.
NMI clustering Normalized Mutual Information: how much the Leiden clusters, one per cell type, tell about the cell-type labels. 0-1, higher = better.
ASW clustering Average silhouette width: how well the Leiden clusters, one per cell type, separate from each other on the embedding. Rescaled to 0-1, higher = better.
iASW clustering Isolated-label silhouette: each cell's silhouette (distance to the nearest other Leiden cluster), averaged per cluster and then over the clusters. Every cluster is treated as an isolated label, where scib's default counts only labels found in few batches. 0-1, higher = better.
iF1 clustering Isolated-label F1: how well one Leiden cluster captures each cell type, the best F1 of one cluster (one cluster per cell type), averaged over the cell types. Every cell type is treated as an isolated label, where scib's default counts only types found in few batches. 0-1, higher = better.
cLISI clustering Cell-type LISI: how much each cell's neighbours share its cell type on the embedding, the median over cells, scaled to 0-1, higher = better.
ASW_batch batch Batch silhouette: how well the batches mix within each cell type, 1 minus the absolute batch silhouette within each cell type, averaged over the cell types. 0-1, higher = better.
ARI_batch batch Batch ARI: how little the Leiden clusters, one per batch, match the batches, 1 minus their absolute Adjusted Rand Index. 0-1, higher = better.
NMI_batch batch Batch NMI: how little the Leiden clusters, one per batch, tell about the batches, 1 minus their Normalized Mutual Information. 0-1, higher = better.
GC batch Graph connectivity: how much of each cell type stays in one connected group of neighbours, the share of its cells in the largest connected part of its neighbour graph, averaged over the cell types. 0-1, higher = better.
iLISI batch Integration LISI: how many batches appear among each cell's neighbours on the embedding, the median over cells, scaled to 0-1, higher = better.
kBET batch k-nearest-neighbour batch effect test: how often a neighbourhood holds the overall batch mix, 1 minus the rate at which the test rejects that mix, averaged over the cell types. 0-1, higher = better.
OCA classification Overall classification accuracy: the fraction of query cells predicted correctly. 0-1, higher = better.
ACA classification Average classification accuracy: the fraction of each cell type predicted correctly, averaged over the cell types. 0-1, higher = better.
Spec classification Specificity, averaged over the cell types. 0-1, higher = better.
Sens classification Sensitivity, averaged over the cell types. 0-1, higher = better.
F1 classification F1-score, averaged over the cell types. 0-1, higher = better.
Details

cLISI and iLISI need scib's compiled LISI helper. Where the binary scib ships cannot run, evaluate rebuilds it once if a C++ compiler (g++, c++ or clang++) is on PATH. Otherwise the warning prints the build command.

kBET needs R through rpy2. Otherwise kBET is NaN.

Each metric is a scib function on the embedding. The package needs scib 1.1 or newer, and attrs["scib_version"] records the version.

code       scib function         settings the table above does not give
ARI, NMI   ari, nmi
ASW        silhouette
iASW       isolated_labels_asw   iso_threshold = number of batches + 1
iF1        isolated_labels_f1    iso_threshold = number of batches + 1
cLISI      clisi_graph           90 nearest cells by path on the
                                 15-neighbour graph
ASW_batch  silhouette_batch
ARI_batch  ari
NMI_batch  nmi
GC         graph_connectivity    the 15-neighbour graph
iLISI      ilisi_graph           as cLISI, for the batches
kBET       kBET

cLISI and iLISI take the median m of the per-cell scores and scale it: cLISI = (L - m)/(L - 1) and iLISI = (m - 1)/(B - 1), with L cell types and B batches.


Look at an embedding

res.embedding(method) returns a method's embedding as an AnnData, with the cell types in obs["label"]. Three scanpy lines draw its UMAP:

import scanpy as sc

e = res.embedding("Matilda")          # or mtb.load_run("out/").embedding("Matilda")
sc.pp.neighbors(e, use_rep="X_emb"); sc.tl.umap(e); sc.pl.umap(e, color="label")

When the dataset folder was written from your AnnData, the cells' names are its obs_names, and one line puts the embedding into it: adata.obsm["X_Matilda"] = e[adata.obs_names].obsm["X_emb"].


Are my numbers comparable?

Score a method's output with the dataset's own labels, in the method's cell order: mtb.evaluate(embedding, dataset=..., category=..., method=...).

Scores of your runs can differ a little from the stored ones: some methods, such as sciPENN and VIMCCA, give a slightly different embedding on each run. The Leiden-based and classification scores of the same embedding also depend on the cells' order. Above 4,096 cells, the same embedding also scores a few hundredths differently on another computer, because the neighbours are approximate.

Details

The scored_with column records how the scores were computed: the Leiden backend, where the clusters came from and the multibench version, such as leidenalg/search/0.4.0.


Next: draw it next to the stored results in Load and plot results.