Skip to content

Metrics (evaluate)

Score an embedding with the paper's metrics. The scores come as the long table that mtb.load_results returns. The API overview summarises the metrics= selector.

Name Summary
mtb.evaluate Score an embedding with the benchmark's metrics.

mtb.evaluate

evaluate(
    embedding,
    labels=None,
    *,
    dataset: str | None = None,
    method: str = "embedding",
    category: str | None = None,
    batch=None,
    metrics=None,
    obsm: str | None = None,
    verbose: bool = True,
) -> DataFrame

Score an embedding with the benchmark's metrics.

The protocol is the paper's: Leiden clusters searched to the number of cell types for ARI, NMI, ASW, iASW and iF1, and to the number of batches for ARI_batch and NMI_batch.

PARAMETERS DESCRIPTION
embedding

Cells x dims: an array, DataFrame, AnnData or MuData (.obsm[obsm]), or a file: .h5 (the benchmark's embedding.h5), .h5ad, .npy, .csv / .tsv. The knn_indices.h5 of a graph method (scMoMaT), with its knn_dists.h5 in the same folder, is scored on that neighbour graph as the paper scores it: ARI, NMI and iF1 on Leiden clusters of the graph, cLISI, GC and iLISI on the graph; ASW, iASW and ASW_batch need an embedding and are NaN.

labels

One cell type per cell: CSV path(s), a 1-D array-like, or an .obs column name ("celltype", "rna:celltype" for a MuData modality). None = the label files of dataset.

default None

dataset

The dataset folder, a name or a path; its name fills the dataset column. With labels=None its label files are read and matched to the cells by the names run writes into embedding.h5; for a file of a run folder without them, in the cell order of the method its run record names; else in the order of method when that is a package method.

type str | None default None

method

Fills the method column: a package method id or your own name.

type str default 'embedding'

category

vertical, diagonal, mosaic or cross; fills the category column. With two or more batches, all but vertical add the classification metrics.

type str | None default None

batch

One batch per cell, in the forms labels takes; needed for the batch metrics unless labels lists two or more files.

default None

metrics

None = every metric the inputs allow except kBET; a family ("clustering", "batch", "classification", "all"), a code ("ARI") or a list of codes and families.

type None, str or list of str default None

obsm

.obsm key of an AnnData, MuData or .h5ad embedding; 'X' means .X. None = X_emb.

type str | None default None

verbose

Print a line when the Leiden search starts on more than 2,000 cells.

type bool default True

RETURNS DESCRIPTION
DataFrame

The long table: metric, value, method, dataset, category, source, scored_with, one row per metric. mtb.plot.bubble draws it, next to mtb.load_results rows or alone.

RAISES DESCRIPTION
ValueError

Missing or misaligned labels or batches, an unknown category or metric, or batch or classification metrics without batch labels.

FileNotFoundError

An embedding, label file or dataset folder that does not exist.

Examples

>>> import multibench as mtb
>>> mtb.evaluate("out/Matilda_D11/embedding.h5", dataset="D11",
...              category="vertical", method="Matilda")
>>> mtb.evaluate(adata, labels="celltype", batch="sample", obsm="X_pca",
...              method="PCA")