Installation¶

multibench is the Python API that wraps the scMultiBench benchmark (Liu, Ding et al., Nature Methods 2025). It is tested on Python 3.9+ on Linux and macOS - the API itself; the per-method conda environments are Linux-only. The package itself is lightweight, a handful of pure-Python data dependencies, so installing it takes under a minute with a warm cache.

TL;DR

conda create -n multibench python=3.11 -y && conda activate multibench
pip install multibench-sc
python -c "import multibench as mtb; print(mtb.__version__)"

Two layers, two install scopes

The multibench API only orchestrates the benchmark, it has light dependencies. The integration methods it runs each live in their own conda env (they have mutually incompatible deps). You install multibench once; you install a method's env only when you want to run that method. See Method environments below.

Step 1, Create an isolated environment¶

Conda (recommended)Mamba (faster solver)venv

terminal

conda create -n multibench python=3.11 -y
conda activate multibench

terminal

mamba create -n multibench python=3.11 -y
mamba activate multibench

terminal

python3.11 -m venv .venv
source .venv/bin/activate

Why isolate?

Even though multibench's own footprint is small, a dedicated env keeps it cleanly separated from the per-method environments that the benchmark orchestrates via conda run. Conda is recommended because that orchestration relies on named conda envs.

Step 2, Install multibench¶

Two ways in - both give you the same import multibench:

PyPI (your own data)Repository (tutorials)

terminal

pip install multibench-sc

The full API in a 184 KB package: discovery, running methods (environments build from the shipped lockfiles; the method scripts fetch themselves on first run), evaluation and figures. Datasets and clones go to ~/.cache/multibench/.

terminal

git clone https://github.com/DSichang/scMultiBench.git
cd scMultiBench
pip install -e . --config-settings editable_mode=compat

What the tutorials use: also carries the stored benchmark tables their figure sections reproduce, and the notebooks themselves.

What multibench is

multibench is the Python API for the scMultiBench benchmark (imported throughout the docs as mtb). For the editable from source install, the compat editable mode writes a plain path entry so import multibench resolves from any working directory (not only the repo root). Plain pip install -e . works too, but the compat flag avoids an empty-namespace import surprise when you later call the API from a script outside the clone.

This pulls in only the core data-handling stack:

Package	Min version	Used for
`numpy`	1.24	array math, embeddings
`pandas`	2.0	tidy metric frames, catalog tables
`h5py`	3.8	canonical scMultiBench `.h5` matrix I/O
`anndata`	0.9	data container / format adapters
`matplotlib`	3.7	scIB-style bubble plots
`pyyaml`	6.0	method registry / config parsing

What is not installed here

No deep-learning frameworks, no per-method packages (torch, scvi-tools, Seurat, etc.). Those belong to the individual method environments, not to multibench. Keeping the core thin is deliberate, you can load published results, plot, and resolve inputs without any heavy stack present.

Step 3, Verify the install¶

terminal

python -c "import multibench as mtb; print(mtb.__version__)"

Expected output: the installed version string (e.g. 0.1.0). If a version prints, the public API is reachable. A quick smoke test of the read-only surface:

import multibench as mtb

print(mtb.list_tasks())                 # available tasks
print(mtb.list_methods(category="vertical"))   # methods declared for a category

Method environments¶

This is the part that makes multibench different from an ordinary library. The benchmark wraps 40 integration methods across four categories (vertical, diagonal, mosaic, cross). Each method ships its own toolchain, totalVI needs scvi-tools, Seurat_v5 / Seurat_WNN / iNMF / Conos need R, scBridge / Portal / uniPort each pin their own PyTorch line, and these conflict with one another. There is no single env that can satisfy them all.

So multibench does not import a method's code. Instead, mtb.run(...):

Resolves the method's command from the registry,
Converts your inputs to the canonical scMultiBench .h5 (unless convert=False),
Executes the command inside that method's own conda env via a cmd_template, default "conda run -n {env} {cmd}",
Loads the typed output back into a RunResult.

import multibench as mtb

res = mtb.run(
    "SCALEX", category="diagonal", task="clustering",
    inputs={"rna": "data/D35/rna1.h5", "atac_gas": "data/D35/atac_gas1.h5"},
    out_dir="runs/scalex_d35",
)
res.output    # the joint embedding (ndarray)
res.cmd       # the exact command that was executed (list[str])

Create the method's env first

mtb.run(...) will fail at the conda run -n <env> step if that env does not exist. Create the per-tool environment from the recipe shipped alongside that method in the scMultiBench repository before running it. The method scripts themselves are never modified by multibench.

Custom orchestration

Not using conda envs by name? Override how the command is launched with cmd_template (e.g. a container wrapper), or point repo_path at a local checkout of the method scripts. The default is "conda run -n {env} {cmd}".

Every method in the registry is wired and run-verified: its command has been cross-checked against the real entrypoint and executed end-to-end on a reference dataset.

Build the environments¶

The multibench package itself is ~2 MB; disk goes to the conda environments of the methods you choose to run. The repository commits one lockfile per environment, and the CLI builds them reproducibly (Linux; method environments are not supported on macOS). Install only what you need:

terminal

multibench env doctor                             # which envs exist / are missing
multibench env install --methods Matilda --run    # one method: its env only (2-14 GB)
multibench env install --category vertical --run  # one category (45-101 GB, see below)
multibench env install --run                      # the whole benchmark (29 envs, ~167 GB)

Add --packed to any env install to use prebuilt archives (download + unpack in minutes instead of a 10-30 minute solve-and-build). All 29 environments are covered: smaller archives ship as release assets, larger ones from Zenodo (10.5281/zenodo.21928885, 10.5281/zenodo.21928889); if a download fails, the install falls back to the lockfile build automatically.

Measured per-category footprints: vertical ~101 GB (18 envs), diagonal ~58 GB (9), mosaic ~45 GB (7), cross ~71 GB (11) - categories share environments, so combinations cost less than their sum. env doctor is safe to run any time; env install skips environments that already exist, so it resumes after interruption.

Get the data¶

The benchmark datasets are not part of the git clone, and for the tutorials you do not need to fetch anything by hand: each tutorial auto-downloads its own reference data (11-290 MB) from the repository's release assets into data/ on first run. Manual equivalent, e.g. for D11:

terminal

wget -qO- https://github.com/DSichang/scMultiBench/releases/download/data-v1/D11.tar.gz | tar xz -C data/

Available there: D11 (vertical, 11 MB), D28 (diagonal, 137 MB), D45 (mosaic, 290 MB), D46 (mosaic, 97 MB), D52 (cross, 179 MB). The full 65-dataset collection is linked from the scMultiBench README; unpack any of it under data/ at the repository root - mtb.config.DEFAULT.data_path defaults to exactly that folder, and every data_path= argument in the API accepts an alternative location.

The evaluation stack¶

mtb.evaluate(...) computes scIB metrics on a run output. Since v0.2, scib and scanpy install with the package - no extra step. (pip install -e ".[eval]" remains accepted for compatibility; it is now a no-op.)

Troubleshooting¶

Stuck on install?

conda run -n <env> fails with "EnvironmentNotFound" when calling mtb.run: The method's per-tool conda env does not exist. Create it from that method's recipe in the scMultiBench repo first, or pass a custom cmd_template= to mtb.run(...) that launches the command another way.
ImportError: No module named scib when calling mtb.evaluate: Install the eval extra: pip install scib. The core multibench install deliberately omits it.
cLISI / iLISI / kBET come back as NaN: This is expected when the LISI binary or rpy2 is unavailable, it is a warning, not an error. Install the LISI binary / rpy2 to populate those rows; everything else still computes.
conda solver hangs for >2 minutes: Switch to mamba: conda install -n base -c conda-forge mamba then re-create the env with mamba. Or conda config --set solver libmamba to use libmamba globally.
unsupported input format from mtb.io.to_canonical: The adapter accepts .h5ad, .csv / .tsv, .loom, and in-memory AnnData. Convert other formats to one of these first, or pass an already-canonical scMultiBench .h5 (it is detected and passed through unchanged).
Still broken?: Open an issue, paste the output of python -c "import multibench as mtb; print(mtb.__version__)" or your pip freeze.

Next: the Quickstart, or jump straight to running a method with the Run a method tutorial.