Skip to content

API reference

The scMultiBench API (imported as multibench, aliased throughout the docs as mtb) exposes a small, flat, free-function API organised into a handful of namespaces:

mtb.run returns a RunResult; its output flows into mtb.evaluate, whose wide frame is reshaped by mtb.eval.to_long into the same tidy long frame that mtb.load_results returns and mtb.plot.bubble consumes.

Method environments

Running a method shells out to the conda environment it shares with other compatible tools via conda run -n <env> <cmd>, where <env> is mtb.env.group_for(method) — the same env mtb.env.plan provisions (overridable through cmd_template). The original method scripts are never modified — multibench only builds the command, converts inputs to the canonical .h5, and loads the typed output.

Top-level functions

mtb.run

mtb.run(method: str, category: str, task: str = "clustering", *, inputs: dict, out_dir: str, params: dict | None = None, convert: bool = True, cmd_template: str | None = None, repo_path: Path | None = None) -> RunResult

Build a method variant's command, wrap it (default "conda run -n {env} {cmd}"), run it in out_dir, and load the typed output. inputs is a {modality_role: path} dict; non-canonical modality inputs are auto-converted to the canonical scMultiBench .h5 unless convert=False. Auxiliary roles (data_dir, source_data, target_data, source_cty, target_cty, out_dir) are passed through verbatim. Returns a RunResult.

Parameters

  • method (str) — registry method id (see list_methods).
  • category (str) — one of vertical, diagonal, mosaic, cross; drives variant selection together with the supplied modalities.
  • task (str) — reserved for future per-task dispatch; not used for variant selection in v1.
  • inputs (dict) — {modality_role: path}, e.g. {"rna1": ..., "atac_gas1": ...}.
  • out_dir (str) — working directory; created if missing.
  • params (dict | None) — extra method hyperparameters merged into the command.
  • convert (bool) — convert non-aux inputs to the canonical .h5 (default True).
  • cmd_template (str | None) — command wrapper; None uses the method's env via conda run -n {env} {cmd}.
  • repo_path (Path | None) — path to the reference scMultiBench repo; defaults to config.DEFAULT.repo_path.

mtb.evaluate

mtb.evaluate(output, category: str, task: str = "clustering", labels=None, clustering=None, batch=None, metric_set: str = "scib") -> pd.DataFrame

Compute scIB metrics on a run output, returning a metric.csv-shaped frame (index = metric, single Value column). v1 supports metric_set="scib" with task in {clustering, batch, all}. Clustering metrics require both labels (cell types) and clustering; batch/all additionally require batch. This is re-exported from mtb.eval.evaluate.

Parameters

  • output — a run embedding (np.ndarray) or a path/handle readable as one.
  • category (str) — reserved for future per-category metric dispatch; accepted but unused in v1.
  • task (str) — clustering, batch, or all.
  • labels — cell-type labels (np.ndarray or readable source).
  • clustering — predicted cluster assignments.
  • batch — batch labels (required for batch/all).
  • metric_set (str) — only "scib" is wired in v1.

Graceful degradation

Metrics whose external dependency is unavailable return NaN with a warning rather than failing the call: cLISI / iLISI need a compiled LISI binary, and kBET needs rpy2. All other scIB metrics (ARI, NMI, ASW, iASW, iF1, ASW_batch, GC) compute in pure Python.

mtb.load_results

mtb.load_results(category: str, task: str = "clustering", metric_set: str = "scib", dataset: str | None = None, method: str | None = None, metric: list[str] | str | None = None, clustering: str = "default", result_path: Path | str | None = None) -> pd.DataFrame

Return a tidy long frame (metric, value, method, dataset, category) of the published benchmark metric tables, with metric codes and method ids canonicalised. Optionally filter by dataset, method, and metric. When the default-clustering correction file metric_asw_iasw_if1.csv is present, its corrected ASW / iASW / iF1 values override those in metric.csv.

Published-metric availability

Only metric_set="scib" is wired in v1. Published scib metric tables exist for the vertical, cross, and diagonal categories; mosaic metrics are not published, so load_results(category="mosaic") raises FileNotFoundError (the methods still run, there is just no published table to load).

Parameters

  • category (str) — vertical, diagonal, mosaic, or cross.
  • task (str) — task token (v1 results are clustering/batch under scib).
  • metric_set (str) — only "scib" is wired in v1.
  • dataset (str | None) — restrict to a single dataset id (e.g. "D12").
  • method (str | None) — restrict to one method (any known spelling).
  • metric (list[str] | str | None) — restrict to one or more metric codes.
  • clustering (str) — default, louvain, or kmeans.
  • result_path (Path | str | None) — override config.DEFAULT.result_path.

mtb.available_datasets

mtb.available_datasets(category: str, metric_set: str = "scib", result_path: Path | str | None = None) -> list[str]

Return the dataset ids that have published metric_set results for a category — exactly the datasets load_results can load. Returns [] when the category has no published metrics (e.g. mosaic). Use this to discover what is loadable before calling load_results.

mtb.list_methods

mtb.list_methods(category: str | None = None, task: str | None = None, runnable: bool | None = None) -> list[str]

Return the registry method ids, optionally filtered to those declaring a given category and/or task. Every registry method is wired to run; runnable remains as a filter for forward compatibility (None, the default, returns all).

mtb.list_tasks

mtb.list_tasks() -> list[str]

Return the sorted set of tasks declared across all method specs.

mtb.find_methods

mtb.find_methods(category: str | None = None, task: str | None = None, needs_labels: bool | None = None, atac: str | None = None, modalities: list[str] | set[str] | None = None, runnable: bool | None = None) -> list[str]

Return method ids matching every supplied filter. modalities keeps only methods that consume all of the requested base modality types (e.g. ["rna", "atac"]); needs_labels and atac filter by the method's declared requirements.

Parameters

  • category (str | None) — integration category the method must declare.
  • task (str | None) — task the method must declare.
  • needs_labels (bool | None) — whether the method requires cell-type labels.
  • atac (str | None) — required ATAC representation — exact match, one of "peak" or "gene_activity" (not a boolean).
  • modalities (list[str] | set[str] | None) — modality types the method must consume, e.g. ["rna", "atac"]. Derived from a method's variants.
  • runnable (bool | None) — filter on whether the method has a runnable variant; every registry method currently does.

mtb.method_info

mtb.method_info(method: str, files_dir: Path | str | None = None) -> dict

Return a flat dict combining the registry spec (id, language, categories, tasks, env, atac, needs_labels, status, setup_hint, variants) with optional catalog metadata (deep_learning, output) when files_dir is given. env is the conda environment mtb.run actually executes the method in — the resolved group_for env (a shared group env, or the method's own scmb_<method> env).

mtb.inputs_for

mtb.inputs_for(dataset: str, method: str, category: str, modalities: list[str] | set[str] | None = None, data_path: Path | str | None = None, check: bool = False) -> dict

Resolve a catalog dataset id + method to concrete {modality_role: path} input paths. The dataset tree is flat (<data_path>/<dataset>/<file>); each role is resolved to the file actually present in that directory — the role token or a known alias (atac_peak → peak.h5, atac_gas → atac.h5), falling back to <role>.h5 when no candidate exists. When modalities is omitted and the method has exactly one variant for the category, that variant is used; if several match, a ValueError lists the modality-sets and asks you to disambiguate with modalities=. Pass check=True to raise FileNotFoundError (listing what is present) if any resolved file is missing, rather than returning a best-effort path that only fails later inside run. Use labels_for to get the matching cell-type label files for evaluate.

Parameters

  • dataset (str) — catalog dataset id (e.g. "D35").
  • method (str) — registry method id.
  • category (str) — integration category.
  • modalities (list[str] | set[str] | None) — exact modality-set to select a variant.
  • data_path (Path | str | None) — override config.DEFAULT.data_path.
  • check (bool) — if True, raise FileNotFoundError when a resolved input file does not exist.

mtb.labels_for

mtb.labels_for(dataset: str, data_path: Path | str | None = None) -> dict

Return {name: path} of the cell-type label CSVs for a dataset — the *cty*.csv files in the (flat) dataset directory, under their dataset-specific names (cty.csv, rna_cty.csv, cty1.csv, …), excluding tool-specific *_scjoint* reformats. Use these as the labels= argument to mtb.evaluate. Raises FileNotFoundError if the dataset directory is absent.

Parameters

  • dataset (str) — catalog dataset id.
  • data_path (Path | str | None) — override config.DEFAULT.data_path.

Benchmark workflow

The four category tutorials are built on these functions.

mtb.scan

mtb.scan(dataset: str, category: str | None = None,
         data_path: Path | str | None = None) -> pd.DataFrame

Report every method that can run on dataset, and why the rest cannot: one row per (method, category, modalities) with runnable and, when it is not, a reason. modalities is a +-joined string here (e.g. "rna+adt"); run_all / inputs_for take it as a list, so split on "+".

mtb.run_all

mtb.run_all(dataset: str, category: str, *, out_dir, modalities=None,
            methods=None, params=None, data_path=None, evaluate=True,
            dry_run=False, verbose=True, timeout=None,
            skip_existing=False) -> BatchResult

Run every applicable method (or the methods= subset) on dataset end to end - inputs, the method's own conda env, output loading, metrics - and collect everything in a BatchResult. dataset is the directory NAME of your data; data_path is the folder that contains it (defaults to the configured data root). A method that fails becomes a row in .failures, not an exception; timeout= bounds each method's whole step; skip_existing=True resumes an interrupted sweep.

BatchResult: .summary (one row per method), .long (tidy metric/value frame - the input to mtb.plot.bubble), .failures, .plot() (bubble figure of every scored method) and .save(out_dir) which writes summary.csv, long.csv and failures.csv.

mtb.params_for

mtb.params_for(method: str, category: str | None = None, modalities=None) -> dict

The tunable hyperparameters a method's variant exposes (with defaults), plus the fixed arguments the wrapper controls. Multi-variant methods need modalities= to pick the variant.

mtb.describe_layout

mtb.describe_layout(category: str | None = None) -> str

Human-readable description of the on-disk data layout each category expects.

mtb.sweep

mtb.sweep(dataset: str, category: str, method: str, param: str, values, *,
          out_dir, modalities=None, data_path=None, timeout=None,
          verbose=True) -> pd.DataFrame

Run ONE method repeatedly over a range of one hyperparameter, one output directory per value, and return the collected metric rows.

mtb.plot.*

mtb.plot.bubble

mtb.plot.bubble(long_df, *, metrics=None, methods=None, order=None, aggregate="dataset", cmap="Blues", title=None, save=None) -> matplotlib.figure.Figure

Render an scIB-style bubble table from a tidy long frame: each circle's radius encodes the metric rank and its fill encodes the (min-max normalised) value, with methods sorted by an overall column shown first. aggregate="dataset" averages values across datasets; aggregate="summary" averages ranks across datasets. If save is given, the figure is written there (bbox_inches="tight"). Returns the matplotlib.Figure.

Parameters

  • long_df (pd.DataFrame) — long frame with metric, value, method, dataset columns.
  • metrics (list[str] | None) — metric columns to include, in order.
  • methods (list[str] | None) — restrict to these methods.
  • order (list[str] | None) — explicit row order (overrides the overall sort).
  • aggregate (str) — "dataset" (mean value) or "summary" (mean rank).
  • cmap (str) — matplotlib colormap for the fill (default "Blues").
  • title (str | None) — axes title.
  • save (str | None) — output path; e.g. "fig.pdf".

Callable namespace

mtb.plot.bubble is callable for the common path and exposes the underlying builders — mtb.plot.bubble.build_table(...) and mtb.plot.bubble.render(...) reach the module functions directly. The same callables are also exported on the namespace itself — mtb.plot.plot_bubble, mtb.plot.build_table, mtb.plot.render — for discoverability.

mtb.eval.*

mtb.eval.evaluate

mtb.eval.evaluate(output, category: str, task: str = "clustering", labels=None, clustering=None, batch=None, metric_set: str = "scib") -> pd.DataFrame

The evaluation entry point, also re-exported as mtb.evaluate. See that entry for the full parameter list and the LISI/kBET graceful-NaN behaviour.

mtb.eval.to_long

mtb.eval.to_long(value_df, method, dataset, category) -> pd.DataFrame

Reshape an evaluate wide frame (index = metric, column Value) into the long frame (metric, value, method, dataset, category) that load_results and plot.bubble consume, canonicalising metric codes along the way. This is the bridge from the evaluate (wide) output to the load_results/plot.bubble (long) shape, and is also re-exported at the top level as mtb.to_long.

Parameters

  • value_df (pd.DataFrame) — an evaluate() result (index = metric, Value column).
  • method (str) — method id to stamp onto every row.
  • dataset (str) — dataset id to stamp onto every row.
  • category (str) — integration category to stamp onto every row.

mtb.io.*

The input-format adapter to and from the canonical scMultiBench .h5 (an HDF5 file with matrix/data of shape features × cells, plus matrix/features and matrix/barcodes).

mtb.io.to_canonical

mtb.io.to_canonical(src, out: Path | str | None = None, modality: str | None = None, convert: bool = True) -> Path

Convert src to a canonical .h5 and return its path. Accepts .h5ad, .csv / .tsv, .loom, or an in-memory AnnData. An already-canonical path is returned as-is when no conversion is needed; otherwise out is required.

Parameters

  • src — source data (path or AnnData).
  • out (Path | str | None) — destination .h5; required unless src is already canonical.
  • modality (str | None) — optional modality hint.
  • convert (bool) — force conversion even for a canonical source (default True).

mtb.io.read_canonical

mtb.io.read_canonical(path: Path | str) -> AnnData

Read a canonical .h5 back into an AnnData (cells × genes), restoring var_names from matrix/features and obs_names from matrix/barcodes.

mtb.catalog.*

Typed views of the paper's metadata CSVs, plus the canonicalisers used throughout the API.

mtb.catalog.methods

mtb.catalog.methods(files_dir: Path | str | None = None) -> pd.DataFrame

The methods table with normalised columns (method, canonical_id, language, deep_learning, atac, output, needs_labels) and list-valued categories / tasks. Defaults to config.DEFAULT.files_path.

mtb.catalog.datasets

mtb.catalog.datasets(files_dir: Path | str | None = None) -> pd.DataFrame

The datasets table with a derived simulated flag (true for ids beginning SD).

mtb.catalog.metrics

mtb.catalog.metrics(files_dir: Path | str | None = None) -> pd.DataFrame

The metric-details table (whitespace-normalised column names).

mtb.catalog.canonical_id

mtb.catalog.canonical_id(name: str) -> str

Return the canonical registry method id for any known display or result-dir spelling (e.g. "seurat v5" → "Seurat_v5", "mofa+" → "MOFA2").

mtb.catalog.canonical_metric

mtb.catalog.canonical_metric(code: str) -> str | None

Canonicalise a raw metric short-code (e.g. "ari" → "ARI", "kbet" → "kBET"); returns None for blank or empty codes.

mtb.config.*

mtb.config.Config

@dataclass
class Config:
    result_path: Path   # published benchmark metric tables
    files_path: Path    # metadata CSVs (method/dataset/metric)
    repo_path: Path     # reference scMultiBench repo (method entrypoints)
    data_path: Path     # resolved input datasets

Resolved filesystem paths. Override any field to point at custom locations. A module-level default instance is exposed as mtb.config.DEFAULT; callers may replace its fields in place.

mtb.config.category_folder

mtb.config.category_folder(token: str) -> str

Map a clean category token to the benchmark's space-named result folder (e.g. "vertical" → "vertical integration"). Raises ValueError for an unknown token.

mtb.config.metric_set_dir

mtb.config.metric_set_dir(token: str) -> str

Map a metric-set token to its top-level result directory (e.g. "scib" → "scib_metric"). Raises ValueError for an unknown token.

mtb.env.*

Helpers for the conda environments that mtb.run orchestrates. They describe how to provision an env, they do not create it for you — the recipes are emitted for you to apply.

Two provisioning routes

There are two ways to provision, and run uses the shared group env by default. (1) Group (what run uses): plan / groups build the shared env named by group_for (e.g. scmb_torch_v2 serves several torch methods). (2) Per-method: recipe / the env recipe/env yml CLI emit an isolated scmb_<method> env from one method's own deps; to run in it, pass cmd_template="conda run -n scmb_<method> {cmd}" to run. Prefer the group route unless you need an isolated per-method env.

mtb.env.recipe

mtb.env.recipe(method: str) -> dict

Return the conda environment recipe for a method — the env name, language, channels, and the package/pin list needed to build the env that method runs in. The recipe mirrors what ships alongside the method in the reference scMultiBench repo.

mtb.env.group_for

mtb.env.group_for(method: str) -> str

Return the conda env a method runs and provisions in: its shared group env (e.g. SCALEX and Cobolt map to "scmb_torch_v2"; the R methods to "scmb_r") when the method belongs to a group, otherwise its own default_env_name (scmb_<method>). This is the {env} the default cmd_template fills, and the same value as mtb.method_info(...)["env"]. Because plan / groups build exactly these names, the env you provision is the env run executes in.

mtb.env.default_env_name

mtb.env.default_env_name(method: str) -> str

Return the method's own (singleton) env name — scmb_<method> (e.g. "scmb_scalex"). This is only the env used when a method is not a member of any shared group; the env run actually executes in is always group_for (which returns the shared group env for grouped methods and falls back to this name otherwise).

mtb.env.groups

mtb.env.groups() -> dict[str, list[str]]

Return the mapping of shared env name → the methods that share it. Several methods reuse a single environment, so provisioning one env can unlock several methods; this view shows which.

mtb.env.plan

mtb.env.plan(methods: list[str] | str | None = None) -> list[dict]

Return an ordered provisioning plan — the deduplicated set of envs needed to run the requested methods (or every verified method when None), each entry pairing an env name with its recipe so shared envs are built only once.

RunResult

mtb.run returns a RunResult dataclass. Its fields:

  • method (str) — the method id that was run.
  • out_dir (Path) — the working directory the command ran in.
  • cmd (list[str]) — the fully wrapped argv that was executed (incl. the conda run prefix).
  • output — the loaded primary output (e.g. the joint embedding ndarray).
  • extra (dict) — {file: loaded_output} for any declared extra outputs.
  • stdout (str) — captured standard output.
  • stderr (str) — captured standard error.

A failed run (non-zero exit) raises RuntimeError with the tail of stderr/stdout rather than returning a RunResult.

Versioning

The package version is available as multibench.__version__.

import multibench as mtb
mtb.__version__   # e.g. "0.1.0"

See also