API reference
The package is imported as import multibench as mtb. Its functions form four
groups, methods, run, metrics and figures, plus a setup group. Each name links
to its reference entry, with the parameters and examples.
1. Methods
| call |
returns |
mtb.find_methods(category, modalities=, needs_labels=) |
The method ids that fit. |
mtb.method_info(method) |
Its language, env, GPU use, labels, ATAC form, variants and runtime. |
mtb.params_for(method, category, modalities) |
What params= can set, with each default. |
import multibench as mtb
mtb.find_methods("vertical", modalities=["rna", "adt"], needs_labels=False)
mtb.params_for("Matilda", "vertical", ["rna", "adt"])["tunable"]
2. Run (Linux or Windows)
| call |
returns |
mtb.scan(dataset, category, methods=, params=) |
The plan, one row per method: can it run here and why not. command=True adds its command. |
mtb.run(dataset, category, out_dir, methods=, params=) |
A RunResult, also saved in out_dir. |
mtb.load_run(out_dir) |
The RunResult saved in out_dir. |
mtb.RunResult |
summary, long, embedding(method), plot(), rescore(), save() and records, one dict per method with its status, parameters, label order and provenance. |
mtb.data.fetch("D11")
mtb.scan("D11", "vertical")
res = mtb.run("D11", "vertical", out_dir="out/",
methods=["Matilda", "totalVI"], params={"Matilda": {"epochs": 10}})
res.summary
emb = res.embedding("Matilda") # obsm["X_emb"]; obs: each cell's label (and batch)
A dataset is a folder: its path, or its name under
mtb.config.data_path. One method is methods=["Matilda"].
3. Metrics
import pandas as pd
# your run of Matilda on D11, next to the stored results of D11
mine = mtb.evaluate("out/Matilda_D11/embedding.h5", dataset="D11",
category="vertical", method="Matilda (this run)")
both = pd.concat([mtb.load_results("vertical"), mine])
mtb.rank("vertical", long_df=both)
mtb.plot.bubble(both, metrics="clustering", save="vertical.pdf")
Setup
The long table and metrics=
evaluate, load_results and RunResult.long return the same columns:
metric, value, method, dataset, category and source ("stored" or
"user") and scored_with. The figures and rank read any such table.
Next to stored results, name your rows apart, such as
mine.assign(method=mine.method + " (this run)"), and score them with dataset=
of the stored dataset.
metrics= takes a family, a code, or a list of codes and families:
| family |
codes |
"clustering" |
ARI, NMI, ASW, iASW, iF1, cLISI |
"batch" |
ASW_batch, ARI_batch, NMI_batch, GC, iLISI, kBET |
"classification" |
OCA, ACA, Spec, Sens, F1 |
"all" is the three families, and kBET runs only when named. Higher is better for
every metric.