Skip to content

API reference

The package is imported as import multibench as mtb. Its functions form four groups, methods, run, metrics and figures, plus a setup group. Each name links to its reference entry, with the parameters and examples.

1. Methods

call returns
mtb.find_methods(category, modalities=, needs_labels=) The method ids that fit.
mtb.method_info(method) Its language, env, GPU use, labels, ATAC form, variants and runtime.
mtb.params_for(method, category, modalities) What params= can set, with each default.
import multibench as mtb

mtb.find_methods("vertical", modalities=["rna", "adt"], needs_labels=False)
mtb.params_for("Matilda", "vertical", ["rna", "adt"])["tunable"]

2. Run (Linux or Windows)

call returns
mtb.scan(dataset, category, methods=, params=) The plan, one row per method: can it run here and why not. command=True adds its command.
mtb.run(dataset, category, out_dir, methods=, params=) A RunResult, also saved in out_dir.
mtb.load_run(out_dir) The RunResult saved in out_dir.
mtb.RunResult summary, long, embedding(method), plot(), rescore(), save() and records, one dict per method with its status, parameters, label order and provenance.
mtb.data.fetch("D11")
mtb.scan("D11", "vertical")
res = mtb.run("D11", "vertical", out_dir="out/",
              methods=["Matilda", "totalVI"], params={"Matilda": {"epochs": 10}})
res.summary
emb = res.embedding("Matilda")      # obsm["X_emb"]; obs: each cell's label (and batch)

A dataset is a folder: its path, or its name under mtb.config.data_path. One method is methods=["Matilda"].

3. Metrics

call returns
mtb.evaluate(embedding, labels, method=, dataset=, category=) The scores, as the long table.
mtb.load_results(category, dataset=) The stored results, as the long table.
mtb.rank(category, long_df=) The overall score and row order of mtb.plot.bubble, best first.
mtb.catalog.metrics() The metric codes and their families.
import pandas as pd

# your run of Matilda on D11, next to the stored results of D11
mine = mtb.evaluate("out/Matilda_D11/embedding.h5", dataset="D11",
                    category="vertical", method="Matilda (this run)")
both = pd.concat([mtb.load_results("vertical"), mine])
mtb.rank("vertical", long_df=both)

4. Figures

call returns
mtb.plot.bubble(long_df, metrics=, methods=) The paper's bubble table.
mtb.plot.bar(long_df, metrics=, methods=) One bar per method.
mtb.plot.bubble(both, metrics="clustering", save="vertical.pdf")

Setup

call does
mtb.io.export_dataset(adata, folder, ...) Writes an AnnData or MuData as a dataset folder.
mtb.io.read(path) Reads a dataset file or an embedding.h5 as an AnnData.
mtb.data.fetch(*datasets) Downloads demo datasets and returns each dataset's folder.
mtb.data.write_example(dataset) Writes a tutorial's example .h5ad files.
mtb.catalog.datasets() The datasets to fetch or with stored results, with their category.
mtb.env.install(methods, dry_run=) Installs the methods' envs. dry_run=True shows each env's state and size.
mtb.config The settings: data_path, repo_path, envs_dir and leiden_flavor. print(mtb.config) shows them.

The long table and metrics=

evaluate, load_results and RunResult.long return the same columns: metric, value, method, dataset, category and source ("stored" or "user") and scored_with. The figures and rank read any such table. Next to stored results, name your rows apart, such as mine.assign(method=mine.method + " (this run)"), and score them with dataset= of the stored dataset.

metrics= takes a family, a code, or a list of codes and families:

family codes
"clustering" ARI, NMI, ASW, iASW, iF1, cLISI
"batch" ASW_batch, ARI_batch, NMI_batch, GC, iLISI, kBET
"classification" OCA, ACA, Spec, Sens, F1

"all" is the three families, and kBET runs only when named. Higher is better for every metric.