Load & plot results

scMultiBench ships the published benchmark metric tables from Liu, Ding et al. (Nature Methods 2025) alongside the code that drew the figures. This tutorial covers the two functions you need to turn those tables into a publication-ready panel: mtb.load_results reads the on-disk metric CSVs into one tidy long frame, and mtb.plot.bubble renders the scIB-style bubble table the paper uses.

TL;DR

import multibench as mtb

df = mtb.load_results(category="vertical", dataset="D12")
fig = mtb.plot.bubble(df, metrics=["ARI", "NMI", "ASW", "cLISI"],
                      title="Vertical integration, D12", save="d12.pdf")

The same panel from the shell:

terminal

multibench plot bubble --category vertical --dataset D12 \
    --metrics ARI,NMI,ASW,cLISI --out d12.pdf

---

Step 1, Load a tidy results frame

mtb.load_results walks the benchmark result tree and returns a single long-format pandas.DataFrame with exactly five columns, metric, value, method, dataset, category. One row per (method, dataset, metric) cell. This is the shape every downstream function (plot.bubble, your own groupby/pivot) expects.

import multibench as mtb

df = mtb.load_results(
    category="vertical",      # vertical | diagonal | mosaic | cross
    task="clustering",        # which task's metric set to read
    metric_set="scib",        # only "scib" is wired in v1
    dataset="D12",            # None -> every dataset in the category
    method=None,              # None -> every method present
    metric=None,              # None -> every metric; or ["ARI", "NMI"]
    clustering="default",     # default | louvain | kmeans
)
df.head()
#   metric     value method dataset  category
# 0    ARI  0.372622  MOFA2     D12  vertical
# 1    NMI  0.620118  MOFA2     D12  vertical
# 2    ASW  0.481119  MOFA2     D12  vertical
# 3   iASW  0.445618  MOFA2     D12  vertical
# 4    iF1  0.652439  MOFA2     D12  vertical

Category tokens

scMultiBench groups its 40 integration methods into four integration categories. Pass the clean token; the loader maps it to the (space-named) folder on disk for you.

Token	Integration setting	Modalities
vertical	Paired, the same cells profiled in every modality	RNA+ADT, RNA+ATAC, RNA+ADT+ATAC
diagonal	Unpaired, separate batches bridged by shared features	RNA batch + ATAC batch
mosaic	Overlapping modalities joined through a bridge	mixed
cross	Several fully-matched datasets integrated together	matched

Clustering variants

scIB clustering metrics depend on the upstream clustering. The benchmark stores three variants per (dataset, method); pick one with clustering=:

Value	Reads	Meaning
"default"	metric.csv	the paper's default clustering (Leiden), with corrected ASW/iASW/iF1 coalesced in
"louvain"	metric_louvain.csv	Louvain re-clustering
"kmeans"	metric_kmeans.csv	k-means re-clustering

Only the default variant ships with the bundled tables

The vendored result tree carries the default metric.csv for every (dataset, method); the metric_louvain.csv / metric_kmeans.csv companions are not part of this distribution. clustering="louvain" (or "kmeans") therefore raises FileNotFoundError unless you point result_path= at a tree that contains those files. Keep clustering="default" for the examples below.

Metric names are canonicalised

Both metric= and the returned metric column are passed through catalog.canonical_metric, so you can filter with the clean scIB names, ARI, NMI, ASW, iASW, iF1, cLISI (clustering) and ASW_batch, GC, iLISI, kBET (batch), regardless of how the underlying CSV spelled them. Likewise method= accepts any alias of a method id.

These are the published tables, not a live run

(The four category tutorials draw their figures from a different source: the package's own re-run sweeps shipped under notebooks/results/.) load_results reads the metric CSVs already present under config.DEFAULT.result_path. To score your own run instead, compute metrics with mtb.evaluate and reshape the wide frame with mtb.eval.to_long(value_df, method, dataset, category), it emits the same five-column long frame, so it drops straight into plot.bubble. (Note that cLISI/iLISI need a compiled LISI binary and kBET needs rpy2; where those are unavailable evaluate returns NaN for that metric rather than failing.) Point at a different tree with result_path=.

---

Step 2, Render the scIB bubble table

mtb.plot.bubble takes the long frame and draws the scIB-style bubble table — the grid of circles the paper uses to rank methods. It returns a matplotlib.Figure.

fig = mtb.plot.bubble(
    df,
    metrics=["ARI", "NMI", "ASW", "cLISI"],   # column order, left to right
    methods=None,                              # None -> every method in df
    order=None,                                # None -> sort by overall (best first)
    aggregate="dataset",                       # "dataset" | "summary"
    cmap="Blues",                              # any matplotlib colormap name
    title="Vertical integration, D12",
    save="d12_bubble.pdf",                     # also returns the Figure
)

How to read it:

• Each row is a method, each column a metric. Methods are sorted by their overall score, best at the top.
• Circle fill ~ value — darker = better, on a per-metric min–max scale.
• Circle radius ~ rank — bigger = higher-ranked for that metric (ties take the maximum rank, matching the original R ties.method="max").
• An "overall" column is prepended on the left: the min–max of each method's mean rank across the shown metrics, the single number the row ordering is built on.

Individual vs. summary panels

The aggregate argument decides what a row's numbers mean.

aggregate=\"dataset\" (individual)aggregate=\"summary\" (across datasets)

# one dataset -> raw metric values are plotted directly
df = mtb.load_results(category="vertical", dataset="D12")
fig = mtb.plot.bubble(df, metrics=["ARI", "NMI", "ASW", "cLISI"])

The default. Fill encodes the actual metric value for that dataset. If the frame happens to hold several datasets, values are averaged per (method, metric) first.

# all vertical datasets -> rank within each, then average ranks
df = mtb.load_results(category="vertical")          # no dataset= -> all
fig = mtb.plot.bubble(df, metrics=["ARI", "NMI", "ASW", "cLISI"],
                      aggregate="summary",
                      title="Vertical integration, summary")

Ranks methods within each dataset per metric, then averages those ranks across datasets — the robust cross-dataset leaderboard, not a raw mean of incommensurable scores.

Pinning the row order

Pass order=[...] (a list of method ids) to override the sorted-by-overall layout, handy when you want a fixed method order shared across several figures. Any methods not in df are silently dropped from the list, and methods=[...] subsets the frame before anything is ranked.

---

A concrete example, a vertical panel

Reproduce one vertical-integration panel end to end:

reproduce_d12.py

import multibench as mtb

# 1. published scib clustering table for one dataset
df = mtb.load_results(category="vertical", dataset="D12", clustering="default")

# 2. bubble table, four clustering metrics, saved to PDF
fig = mtb.plot.bubble(
    df,
    metrics=["ARI", "NMI", "ASW", "cLISI"],
    aggregate="dataset",
    cmap="Blues",
    title="Vertical integration, D12",
    save="d12_vertical.pdf",
)

The returned fig is a plain matplotlib.Figure, so you can keep tweaking it (fig.axes[0]...), embed it in a multi-panel layout, or re-save at another DPI.

CLI equivalent

The same figure without writing any Python:

terminal

multibench plot bubble \
    --category vertical \
    --dataset D12 \
    --metrics ARI,NMI,ASW,cLISI \
    --out d12_vertical.pdf

multibench plot accepts --aggregate dataset|summary and writes whatever format --out's extension implies (.pdf, .png, …). It calls load_results and plot.bubble under the hood, so the result is identical to the script above.

Visual fidelity

The bubble table is a matplotlib reproduction of scIB's original R tables (helpers.R / scIB_knit_table.R). The numeric encoding, min–max fills, max-rank radii, the overall column, is ported verbatim, so rankings and shadings match; the rendering is visually faithful within tolerance rather than a pixel-exact clone of the R figure.

---

Next: see the API reference for the full load_results, plot.bubble, evaluate, and eval.to_long signatures.