Skip to content
Open In Colab

Quickstart

Open in Colab

scMultiBench wraps the 40-method single-cell multimodal integration benchmark behind one clean Python surface, imported as multibench and aliased mtb by convention. This page walks the full loop in five steps: discover the methods that fit your data, plot the published benchmark, run a method on your own data, evaluate that run, and plot your own result alongside the rest.

TL;DR

import multibench as mtb

# 1. what can integrate paired RNA + ATAC, unpaired (diagonal)?
mtb.find_methods(category="diagonal", modalities=["rna", "atac"])

# 2. plot the published vertical benchmark for dataset D12
df = mtb.load_results(category="vertical", dataset="D12")
mtb.plot.bubble(df, metrics=["ARI", "NMI", "ASW"], save="fig.pdf")

# 3. run a method on your own data (needs the method's conda env)
res = mtb.run(method="SCALEX", category="diagonal",
              inputs={"rna": "data/D35/rna1.h5",
                      "atac_gas": "data/D35/atac_gas1.h5"},
              out_dir="out/", convert=True)

# 4. score it with scIB metrics
metrics = mtb.evaluate(res.output, category="diagonal", task="clustering",
                       labels="cty.csv")   # clustering derived automatically

# 5. reshape and plot it next to the benchmark
long = mtb.eval.to_long(metrics, method="SCALEX", dataset="my_data",
                        category="diagonal")
mtb.plot.bubble(long, metrics=["ARI", "NMI", "ASW"])

Step 1 — Discover methods for your data

The benchmark spans four integration categories — vertical (paired, same cells), diagonal (unpaired, bridged by features), mosaic (overlapping modalities with a bridge), and cross (fully-matched datasets). Start by filtering the catalog down to the methods that consume your modalities.

import multibench as mtb

mtb.find_methods(category="diagonal", modalities=["rna", "atac"])
# -> ['SCALEX', 'sciCAN', 'Portal', 'uniPort', 'MultiMAP', ...]

find_methods accepts any combination of category, task, needs_labels, atac, and modalities — every filter is ANDed, so you only get methods that satisfy all of them. Once you have a candidate, inspect its full spec:

mtb.method_info("SCALEX")
# {'id': 'SCALEX', 'language': 'python', 'categories': ['diagonal'],
#  'tasks': ['clustering', 'batch'], 'env': 'scmb_torch_v2',
#  'atac': 'gene_activity', 'needs_labels': False,
#  'status': 'verified', 'variants': ['tools_scripts/SCALEX/main_SCALEX.py']}

Step 2 — Plot the published results

Before running anything, you can reproduce the paper's scIB-style bubble tables straight from the shipped metric tables. load_results returns a tidy long frame (metric, value, method, dataset, category); plot.bubble turns it into a figure where each circle's radius tracks the method's rank and its fill tracks the metric value, with an "overall" summary column.

df = mtb.load_results(category="vertical", dataset="D12")
fig = mtb.plot.bubble(df, metrics=["ARI", "NMI", "ASW"], save="fig.pdf")

load_results also takes task, method, metric, and clustering ("default", "louvain", or "kmeans"). To rank methods across several datasets instead of within one, pass aggregate="summary" to plot.bubble, which averages each method's per-dataset ranks.

Step 3 — Run a method on your data

mtb.run builds the method's real command line, executes it in the method's own conda environment, and loads the typed output back into Python. inputs is a dict mapping each modality role to a file path; non-canonical inputs are auto-converted to the canonical scMultiBench .h5 first (pass convert=False to skip this). The method scripts themselves are never modified.

res = mtb.run(
    method="SCALEX",
    category="diagonal",
    inputs={"rna": "data/D35/rna1.h5", "atac_gas": "data/D35/atac_gas1.h5"},
    out_dir="out/",
    convert=True,
)

res.output      # the loaded primary output (e.g. the joint embedding ndarray)
res.cmd         # the exact argv that was executed
res.extra       # dict of any additional outputs
res.stdout      # captured stdout / stderr

The return value is a RunResult with fields method, out_dir, cmd, output, extra, stdout, and stderr.

Each method runs in its own conda env

run orchestrates per-tool environments via the default cmd_template="conda run -n {env} {cmd}", so the target env (for SCALEX, scmb_torch_v2) must already exist on your machine. The env name comes from method_info(...)["env"]. To run inside the current interpreter instead, pass cmd_template="{cmd}". Point repo_path= at your checkout of the scMultiBench method scripts. The relative data/D35/... paths above resolve from the repository root once the benchmark data has been downloaded into data/ (see Get the data); use absolute paths from any other working directory.

Step 4 — Evaluate your run

mtb.evaluate computes scIB metrics on a run output and returns a metric.csv-shaped frame (index = metric name, single "Value" column). Clustering metrics need both the ground-truth cell-type labels and a clustering assignment; for task="batch" or task="all" you also pass batch labels.

metrics = mtb.evaluate(
    res.output,
    category="diagonal",
    task="clustering",
    labels="cty.csv",
    # clustering= optional - derived from the embedding when omitted
)
# index: ARI, NMI, ASW, iASW, iF1, cLISI ...   column: Value

Step 5 — Plot your own result

To compare your run against the benchmark, reshape evaluate's wide frame into the same long format load_results produces, then hand it to plot.bubble, optionally concatenated with the published frame from Step 2.

import pandas as pd

mine = mtb.eval.to_long(metrics, method="SCALEX", dataset="my_data",
                        category="diagonal")

combined = pd.concat([df, mine], ignore_index=True)
mtb.plot.bubble(combined, metrics=["ARI", "NMI", "ASW"],
                title="SCALEX on my data vs. D12", save="compare.pdf")

The same loop is available from the command line:

terminal
multibench plot bubble --category vertical --dataset D12 \
    --metrics ARI,NMI,ASW --out fig.pdf

multibench also exposes list, find, run, evaluate, and env (doctor / install) subcommands that mirror the Python API.

Next steps

  • End-to-end walkthrough — one executed notebook covering discovery, running a method, scIB evaluation, and reproducing the published bubble tables.
  • Run a method — drive any of the 40+ methods through mtb.run, each in its own conda environment.
  • Installation — set up scMultiBench and the per-method conda environments.
  • API Reference — every function, signature, and parameter.