Skip to content

Quickstart

Five steps take you from finding methods to plotting your run next to the stored results. Only step 3 runs a method. It needs Linux, Windows or macOS with Apple silicon, and a method environment. The other steps also work on an Intel Mac.

For your own data:

Words used on this site
embedding
A table with one row per cell and tens to hundreds of numbers per cell, like adata.obsm["X_pca"].
category
How your cells were measured, for example vertical for CITE-seq from one experiment.
variant
One version of a method for one data type, such as RNA + ADT.
role
The name of one input file, such as rna or adt.
dataset folder
One folder that holds the input files of a dataset, such as rna.h5 and cty.csv.
raw counts
Integer counts per cell and feature, before any normalization.
demo dataset
A dataset that mtb.data.fetch downloads: a benchmark dataset such as D11, or a smaller mini copy of one such as D22mini.
stored results
The scores that ship with the package: its own runs of the methods, one dataset per category.
long table
A table with one row per method, dataset and metric.
cell order
The order of the rows in a method's output. For diagonal data most methods put the RNA cells first, then the ATAC cells. run writes each cell's barcode into embedding.h5, and evaluate(dataset=) matches the label files by them.
import multibench as mtb, pandas as pd

# 1. methods for RNA + ADT measured in several batches (cross)
mtb.find_methods(category="cross", modalities=["rna", "adt"])

# 2. plot the stored results of dataset D52
df = mtb.load_results("cross", dataset="D52")
mtb.plot.bubble(df, metrics=["ARI", "NMI", "cLISI"], save="fig.pdf")

# 3. run and score a method on D52 (Linux, Windows or macOS)
mtb.data.fetch("D52")
mtb.env.install(["StabMap"])
res = mtb.run("D52", "cross", methods=["StabMap"], out_dir="out/")

# 4. score an embedding of your own (adata: your AnnData)
scores = mtb.evaluate(adata, labels="celltype", obsm="X_pca", method="PCA")

# 5. plot your run of step 3 next to the stored results
mine = res.long.assign(method=res.long.method + " (this run)")
mtb.plot.bubble(pd.concat([df, mine]), metrics=["ARI", "NMI", "cLISI"])
  1. Discover methods covers filtering the methods by category, modality and labels, and ranking them on the stored results.
  2. Load and plot results covers loading the stored results and drawing the paper's bubble table.
  3. Run a method covers running a method on a demo dataset or on your own data.
  4. Score an embedding covers what each metric measures and what evaluate takes.
  5. Add your own method covers putting your scores next to the stored ones, and Are my numbers comparable? what to expect.

Next steps