Skip to content

Evaluate a run

You have a run output, a joint embedding, a set of cell-type labels, a clustering. Now you want a number. mtb.evaluate takes that output and returns the scIB metrics that the scMultiBench paper reports, in a metric.csv-shaped frame you can read, save, or feed straight into a bubble plot.

TL;DR

import multibench as mtb

metrics = mtb.evaluate(
    result.output,          # embedding (ndarray or .h5 path)
    category="vertical",
    task="clustering",      # ARI, NMI, ASW, iASW, iF1, cLISI
    labels="cty.csv",       # cell-type labels (required)
    # clustering= is optional - omitted, an optimal-resolution Leiden
    # clustering is derived from the embedding automatically
)
print(metrics)              # index = metric, column "Value"

What evaluate does

mtb.evaluate(output, category, task="clustering", labels=None,
             clustering=None, batch=None, metric_set="scib",
             slow_metrics=False) -> pd.DataFrame

Under the hood it assembles a small AnnData carrying your embedding in obsm["X_emb"], with celltype, cluster, and batch as observation columns, builds a neighbour graph on the embedding, and runs the scIB metric functions over it. The result is a DataFrame indexed by metric name with a single "Value" column, the same shape as the metric.csv files the benchmark ships.

task Metrics computed
"clustering" ARI, NMI, ASW, iASW, iF1, cLISI
"batch" ASW_batch, GC (graph connectivity), iLISI (+ kBET with slow_metrics=True)
"all" both groups above

The category argument (vertical, diagonal, mosaic, cross) is accepted for API stability and future per-category metric dispatch, but is not yet used in v1, pass the category your run belongs to and it will keep working when dispatch lands.

Required inputs

evaluate is honest about what each metric group needs, and raises early if something is missing.

labels is required; clustering is optional — clustering quality is measured by comparing the predicted clusters against ground-truth cell types.

metrics = mtb.evaluate(
    result.output,
    category="vertical",
    task="clustering",
    labels="cty.csv",        # single-column CSV, one header line, one label per cell
    # clustering="cluster.h5" # optional: pass a precomputed clustering
                             # (h5 dataset /obs/cluster_leiden) to skip
                             # the internal Leiden derivation
)

Batch correction is scored against a batch label vector, so task="batch" (and task="all") additionally require batch.

metrics = mtb.evaluate(
    result.output,
    category="diagonal",
    task="batch",
    labels="cty.csv",
    clustering="cluster.h5",
    batch="batch.csv",       # required for batch / all
)

Every input can be passed either as a path or as an in-memory numpy.ndarray. Paths are read with the benchmark's own conventions:

  • output / clustering → an HDF5 file (embedding from dataset data, oriented as cells × dims; clustering from /obs/cluster_leiden).
  • labels / batch → a single-column CSV with one header line (typically x), one value per cell factorised to integer codes.

Missing labels raises

Calling evaluate with task="clustering" and no labels raises ValueError (omitting clustering is fine - it is derived from the embedding). Calling task="batch" or task="all" without batch raises ValueError too. These are programmer errors, not missing optional metrics, see graceful degradation below for the latter.

Graceful degradation

A few scIB metrics depend on external machinery that is not always present. Rather than failing the whole evaluation when one optional metric can't run, evaluate computes each metric defensively: if it raises, the metric is recorded as NaN with a warning, and every other metric still returns.

Metric Needs If unavailable
cLISI, iLISI a compiled LISI binary NaN + warning
kBET rpy2 (R bridge) NaN + warning

So on a machine without the LISI binary, a task="clustering" call still gives you real ARI, NMI, ASW, iASW, and iF1, only cLISI comes back NaN.

Reading the result

>>> metrics
       Value
ARI    0.812
NMI    0.847
ASW    0.591
iASW   0.604
iF1    0.733
cLISI    NaN     # LISI binary not available on this host

It's a plain DataFrame, so you can save it as the benchmark's metric.csv directly:

metrics.to_csv("metric.csv")

Closing the loop: tidy → bubble plot

The wide "Value" frame is the right shape for one run. To compare across methods and datasets, and to draw the scIB-style bubble table from the paper, reshape it to the long frame that plot.bubble and load_results use, with mtb.eval.to_long:

mtb.eval.to_long(value_df, method, dataset, category) -> pd.DataFrame
# columns: metric, value, method, dataset, category

to_long also maps each metric name to its canonical form, so the result lines up with mtb.load_results and can be concatenated with published tables.

import pandas as pd
import multibench as mtb

# one run -> one long frame
long_df = mtb.eval.to_long(
    metrics,
    method="SCALEX",
    dataset="D35",
    category="diagonal",
)

# stack several runs, then draw the bubble table
all_runs = pd.concat([long_df, other_long_df])

fig = mtb.plot.bubble(
    all_runs,
    metrics=["ARI", "NMI", "ASW"],
    aggregate="dataset",   # "summary" averages ranks across datasets
    title="Diagonal integration",
    save="bubble.pdf",
)

plot.bubble returns a Matplotlib Figure (circle radius tracks rank, fill tracks value, with an "overall" column). See the plotting tutorial for the full set of options, and load_results if you want to drop your run alongside the published benchmark tables.

v1 supports metric_set="scib" only

Passing any other metric_set, or a task outside {clustering, batch, all}, raises NotImplementedError. Additional metric sets are planned but not yet wired.

Next: turn these numbers into a figure in Plot a bubble table.