Score an embedding¶
mtb.evaluate scores an embedding against cell-type labels with the
paper's metrics and returns them as a long table. An embedding is a table
with one row per cell, like adata.obsm["X_pca"].
Summary
import scanpy as sc
import multibench as mtb
mtb.data.fetch("D11")
mtb.data.write_example("D11") # my_file.h5ad: cell types in obs["celltype"]
adata = sc.read_h5ad("my_file.h5ad")
sc.pp.normalize_total(adata)
sc.pp.log1p(adata)
sc.pp.pca(adata) # the embedding to score: adata.obsm["X_pca"]
scores = mtb.evaluate(adata, labels="celltype", obsm="X_pca", method="PCA",
category="vertical", metrics="clustering")
print(scores)
Choose the metrics¶
metrics= takes a family ("clustering",
"batch", "classification", "all"), a metric code, or a list of codes and
families. The default computes every metric your inputs allow except kBET.
mtb.evaluate(adata, labels="celltype", obsm="X_pca", metrics="cLISI")
mtb.evaluate(adata, labels="celltype", obsm="X_pca", metrics=["ARI", "NMI"])
The protocol is the paper's. ARI, NMI, ASW, iASW and iF1 score Leiden clusters searched to the number of cell types. ARI_batch and NMI_batch score Leiden clusters searched to the number of batches.
Details
The search bisects the Leiden resolution on [0, 3] for at most 30 steps until the clustering has the number of clusters wanted. When no step gets there, the last step's clusters are scored. The clusters come from scanpy's default 15-neighbour graph of the embedding, or of its first 50 principal components when it is wider, as in the paper. cLISI, iLISI and GC use the 15-neighbour graph of the whole embedding.
A selection without the seven metrics above skips the search, such as
metrics="cLISI".
kBET is computed only when a list names it, and can take hours on large datasets.
Inputs¶
Pass the embedding and one cell-type label per cell, in the embedding's row order. Both can be paths or in-memory objects. The reference lists the accepted forms.
A column is named with or without obs:, so labels="obs:celltype" is the
same. To score a method's output, use evaluate(dataset=, category=,
method=) and no labels=: the embedding.h5 that run writes names its
cells, and evaluate matches the label files to them.
A missing label is not a cell type: evaluate raises when a label is
missing, and so does a missing batch. Score the labelled cells, as
adata[adata.obs["celltype"].notna()].
Details
The labels of a dataset folder, in its cell order:
import pandas as pd
folder = mtb.data.fetch("D52mini")
order = {"rna": 0, "adt": 1, "atac": 2}
files = sorted(folder.glob("*cty*.csv"),
key=lambda f: (order.get(f.stem.split("_")[0], -1), len(f.stem), f.stem))
labels = pd.concat([pd.read_csv(f).iloc[:, 0] for f in files]).to_numpy()
They follow the folder's cells as mtb.io.read reads them: RNA before
ADT before ATAC, batches by number.
labels takes a CSV path with a header row (column x, or the only
column), a list of CSV paths stacked in that order, a 1-D array or
Series, or the name of an .obs column. batch takes the same forms.
With an AnnData, labels and batch can name .obs columns, and
obsm= names the embedding. A MuData works the same way, with the
joint embedding in its .obsm, and "rna:celltype" names a column of
one modality.
A label is missing when it is NaN, None, blank text, or text that a
label file reads as missing, such as NA, nan or null.
When the output has cell ids, the ids decide. An AnnData, an .h5ad
file, a DataFrame indexed by cell id and a CSV embedding whose first
column holds the barcodes have them. A Series indexed by cell id, or a
CSV whose first column holds the cell ids, is then aligned by id. Ids of
other cells are left out. Where barcodes repeat across a dataset's label
files, a cell id is the label file and the barcode, as cty2:AAAC...,
the way res.embedding(m) names cells.
A label file without cell ids in a dataset folder takes the cell ids of
the matrix next to it. Any other file or Series without cell ids is
refused: add the barcodes, or pass .to_numpy() to match by position.
A bare array has no cell ids. A Series or a CSV is then matched by
position, with a warning when it holds cell ids. Plain arrays and lists
are always matched by position.
A run's embedding.h5 holds the name of each cell (the dataset
barcodes), so dataset= matches the label files to its cells whatever
method= says. In a run folder written before 0.4.0, the run's record
names the method that made the file. For an embedding without cell
names, dataset= reads the cell order of method= when it names a
package method, also as method="StabMap (this run)", else the folder's
order: numbered files ascending, and rna before adt before atac.
StabMap puts its reference batch first, which is cty3 on D52. Without
category=, a benchmark dataset gives it. On a folder of your own, a
package method needs it. A list of label files is stacked in the order
given.
Batch metrics¶
Batch metrics also need one batch per cell. Pass batch=, or dataset= of
a multi-batch dataset folder without labels: each of its label files is
then a batch. Give the category and the method too, so the files follow the
rows of the method's output. A wrong order gives wrong scores without an
error.
mtb.data.fetch("D52mini")
mtb.data.write_example("D52mini", "d52") # d52/my_file.h5ad: donors in obs["batch"]
cross = sc.read_h5ad("d52/my_file.h5ad")
sc.pp.normalize_total(cross)
sc.pp.log1p(cross)
sc.pp.pca(cross)
mtb.evaluate(cross, labels="celltype", batch="batch", obsm="X_pca", metrics="all")
Details
ARI_batch and NMI_batch are 1 minus the absolute ARI and NMI between the batch clusters and the batches, so higher means better mixed batches.
Without batch=, run uses each cell's label file as its batch,
so batches inside one file, such as donors, are not scored. Pass
batch= to run, or re-score saved outputs with the real batch
vector, mtb.load_run("out/").rescore(batch=...). No method is re-run.
Classification metrics¶
With two or more batches, evaluate also predicts each cell's type from
another batch and scores the predictions: OCA, ACA, Spec, Sens and F1. The
default adds them unless category="vertical".
Details
Each batch in turn is the reference that a small neural network learns from, as in the paper. The scores are the mean over the references, as the paper reports them.
They need the method's embedding. A graph method has none, and
run leaves them out for it. Where they cannot be computed,
metrics="classification" raises. "all" gives NaN with a warning,
and so does the default, except on data in one batch, where it leaves
them out.
For vertical data in several batches, ask for them with
metrics="classification".
The result¶
evaluate returns the long table: one row per metric, with the columns
metric, value, method, dataset, category, source and
scored_with.
method=, dataset= and category= fill their columns, and source reads
user. A
metric that cannot run on your machine is NaN, with a warning.
scores.to_csv("mine.csv", index=False)
scores.pivot(index="method", columns="metric", values="value")
Add your own method concatenates it with the stored results.
What each metric measures¶
mtb.catalog.metrics()
returns this table. The details below give the exact computation.
| Metric | Family | Description |
|---|---|---|
| ARI | clustering | Adjusted Rand Index: how well the Leiden clusters, one per cell type, match the cell-type labels. About 0 = random, 1 = identical, higher = better. |
| NMI | clustering | Normalized Mutual Information: how much the Leiden clusters, one per cell type, tell about the cell-type labels. 0-1, higher = better. |
| ASW | clustering | Average silhouette width: how well the Leiden clusters, one per cell type, separate from each other on the embedding. Rescaled to 0-1, higher = better. |
| iASW | clustering | Isolated-label silhouette: each cell's silhouette (distance to the nearest other Leiden cluster), averaged per cluster and then over the clusters. Every cluster is treated as an isolated label, where scib's default counts only labels found in few batches. 0-1, higher = better. |
| iF1 | clustering | Isolated-label F1: how well one Leiden cluster captures each cell type, the best F1 of one cluster (one cluster per cell type), averaged over the cell types. Every cell type is treated as an isolated label, where scib's default counts only types found in few batches. 0-1, higher = better. |
| cLISI | clustering | Cell-type LISI: how much each cell's neighbours share its cell type on the embedding, the median over cells, scaled to 0-1, higher = better. |
| ASW_batch | batch | Batch silhouette: how well the batches mix within each cell type, 1 minus the absolute batch silhouette within each cell type, averaged over the cell types. 0-1, higher = better. |
| ARI_batch | batch | Batch ARI: how little the Leiden clusters, one per batch, match the batches, 1 minus their absolute Adjusted Rand Index. 0-1, higher = better. |
| NMI_batch | batch | Batch NMI: how little the Leiden clusters, one per batch, tell about the batches, 1 minus their Normalized Mutual Information. 0-1, higher = better. |
| GC | batch | Graph connectivity: how much of each cell type stays in one connected group of neighbours, the share of its cells in the largest connected part of its neighbour graph, averaged over the cell types. 0-1, higher = better. |
| iLISI | batch | Integration LISI: how many batches appear among each cell's neighbours on the embedding, the median over cells, scaled to 0-1, higher = better. |
| kBET | batch | k-nearest-neighbour batch effect test: how often a neighbourhood holds the overall batch mix, 1 minus the rate at which the test rejects that mix, averaged over the cell types. 0-1, higher = better. |
| OCA | classification | Overall classification accuracy: the fraction of query cells predicted correctly. 0-1, higher = better. |
| ACA | classification | Average classification accuracy: the fraction of each cell type predicted correctly, averaged over the cell types. 0-1, higher = better. |
| Spec | classification | Specificity, averaged over the cell types. 0-1, higher = better. |
| Sens | classification | Sensitivity, averaged over the cell types. 0-1, higher = better. |
| F1 | classification | F1-score, averaged over the cell types. 0-1, higher = better. |
Details
cLISI and iLISI need scib's compiled LISI helper. Where the binary
scib ships cannot run, evaluate rebuilds it once if a C++ compiler
(g++, c++ or clang++) is on PATH. Otherwise the warning prints
the build command.
kBET needs R through rpy2. Otherwise kBET is NaN.
Each metric is a scib function on the embedding. The package needs
scib 1.1 or newer, and attrs["scib_version"] records the version.
code scib function settings the table above does not give
ARI, NMI ari, nmi
ASW silhouette
iASW isolated_labels_asw iso_threshold = number of batches + 1
iF1 isolated_labels_f1 iso_threshold = number of batches + 1
cLISI clisi_graph 90 nearest cells by path on the
15-neighbour graph
ASW_batch silhouette_batch
ARI_batch ari
NMI_batch nmi
GC graph_connectivity the 15-neighbour graph
iLISI ilisi_graph as cLISI, for the batches
kBET kBET
cLISI and iLISI take the median m of the per-cell scores and scale it: cLISI = (L - m)/(L - 1) and iLISI = (m - 1)/(B - 1), with L cell types and B batches.
Look at an embedding¶
res.embedding(method) returns a method's embedding as an AnnData, with the
cell types in obs["label"]. Three scanpy lines draw its UMAP:
import scanpy as sc
e = res.embedding("Matilda") # or mtb.load_run("out/").embedding("Matilda")
sc.pp.neighbors(e, use_rep="X_emb"); sc.tl.umap(e); sc.pl.umap(e, color="label")
When the dataset folder was written from your AnnData, the cells' names are
its obs_names, and one line puts the embedding into it:
adata.obsm["X_Matilda"] = e[adata.obs_names].obsm["X_emb"].
Are my numbers comparable?¶
Score a method's output with the dataset's own labels, in the method's cell
order: mtb.evaluate(embedding, dataset=..., category=..., method=...).
Scores of your runs can differ a little from the stored ones: some methods, such as sciPENN and VIMCCA, give a slightly different embedding on each run. The Leiden-based and classification scores of the same embedding also depend on the cells' order. Above 4,096 cells, the same embedding also scores a few hundredths differently on another computer, because the neighbours are approximate.
Details
The scored_with column records how the scores were computed: the
Leiden backend, where the clusters came from and the multibench
version, such as leidenalg/search/0.4.0.
Next: draw it next to the stored results in Load and plot results.