End-to-end walkthrough¶
This notebook takes one method through each step: find it, resolve its inputs, check its parameters, install its environment, run it, score it and plot the scores. The method is Matilda and the dataset is D11: 2,864 blood cells measured with CITE-seq, which gives RNA and surface protein (ADT) for each cell.
find_methods → inputs_for → params_for → env.install → run → evaluate → plot
The category tutorials run several methods at once with run_all. Methods run on Linux. On macOS or Windows, open this notebook in Colab.
1. Install¶
This cell installs multibench-sc. It keeps the numpy and pandas that are already installed, so Colab needs no restart.
Details
On Colab, choose a GPU runtime before you run the notebook: Runtime -> Change runtime type -> T4 GPU. On a CPU runtime, an environment that has a smaller CPU build gets that build, and training methods run slower.
import numpy, pandas
%pip install -q multibench-sc numpy=={numpy.__version__} pandas=={pandas.__version__}
Note: you may need to restart the kernel to use updated packages.
2. Find a method¶
find_methods lists the methods that fit a category and a set of modalities. method_info describes one of them.
import pandas as pd
import multibench as mtb
mtb.find_methods(category="vertical", modalities=["rna", "adt"])
['Multigrate', 'totalVI', 'scMM', 'scMDC', 'scMoMaT', 'moETM', 'Concerto', 'Matilda', 'VIMCCA', 'sciPENN', 'scMSI', 'UINMF', 'MOFA2', 'Seurat_WNN']
info = mtb.method_info("Matilda")
{k: info[k] for k in ("id", "language", "env", "gpu", "needs_labels", "reference")}
{'id': 'Matilda',
'language': 'python',
'env': 'matilda',
'gpu': 'used when present',
'needs_labels': True,
'reference': {'doi': '10.1093/nar/gkad157',
'title': 'Multi-task learning from multimodal single-cell omics with Matilda',
'authors': 'Liu C, Huang H, Yang P',
'journal': 'Nucleic Acids Research',
'year': 2023}}
3. Resolve the inputs¶
mtb.data.fetch downloads D11 (11 MB) once. inputs_for returns the files Matilda reads from it. labels_for returns the cell-type labels used for scoring.
Details
check=True raises an error now if a file is missing, instead of when the method runs.
A dataset folder of your own works the same way. mtb.io.export_dataset writes one from an AnnData, as the category tutorials show.
mtb.data.fetch("D11")
inputs = mtb.inputs_for("D11", "vertical", "Matilda", modalities=["rna", "adt"],
check=True)
inputs
downloading D11 (11 MB) from https://github.com/DSichang/scMultiBench/releases/download/data-v1/D11.tar.gz ...
{'rna': '/tmp/mbtut3/home/.cache/multibench/data/D11/rna.h5',
'adt': '/tmp/mbtut3/home/.cache/multibench/data/D11/adt.h5',
'cty': '/tmp/mbtut3/home/.cache/multibench/data/D11/cty.csv'}
labels = mtb.labels_for("D11")
labels
{'cty': '/tmp/mbtut3/home/.cache/multibench/data/D11/cty.csv'}
4. Check the parameters¶
params_for lists what a run passes to Matilda. defaults are passed on every run, and tunable are the options the method's script accepts. Section 6 changes one with params=.
Details
The values under tunable are the script's own defaults. Many methods take no parameters from the command line, and their tunable is empty.
The params_for reference lists the other keys.
p = mtb.params_for("Matilda", "vertical", ["rna", "adt"])
print("defaults:", p["defaults"])
print("tunable:", list(p["tunable"]))
defaults: {'epochs': 30, 'seed': 1, 'device': 'auto'}
tunable: ['augmentation', 'batch_size', 'device', 'epochs', 'hidden_adt', 'hidden_atac', 'hidden_rna', 'lr', 'seed', 'z_dim']
5. Install the environment¶
Matilda runs in its own environment. mtb.env.install downloads it, with no conda needed. The download is 0.6 GB on a computer without a GPU and 3.0 GB on one with a GPU.
Details
state is PACKED for an environment downloaded now and have for one that was already there. Without dry_run=False, mtb.env.install downloads nothing and returns the plan with its sizes.
From a terminal, multibench env install --methods Matilda --packed --run does the same.
envs = mtb.env.install(["Matilda"], dry_run=False)
pd.DataFrame(envs)[["env", "methods", "state"]]
[env] downloading matilda (gpu build) 3.0 GB ...
[env] 0.3 GB of 3.0 GB (20 MB/s)
[env] 0.6 GB of 3.0 GB (20 MB/s)
[env] 0.9 GB of 3.0 GB (18 MB/s)
[env] 1.2 GB of 3.0 GB (18 MB/s)
[env] 1.5 GB of 3.0 GB (18 MB/s)
[env] 1.8 GB of 3.0 GB (18 MB/s)
[env] 2.1 GB of 3.0 GB (18 MB/s)
[env] 2.4 GB of 3.0 GB (18 MB/s)
[env] 2.7 GB of 3.0 GB (18 MB/s)
[env] 3.0 GB of 3.0 GB (18 MB/s)
[env] unpacking matilda -> /tmp/mbtut3/home/.cache/multibench/envs/matilda ...
| env | methods | state | |
|---|---|---|---|
| 0 | matilda | [Matilda] | PACKED |
6. Run¶
run runs Matilda on the inputs and loads the embedding it writes. An embedding is a table of numbers with one row per cell. params={"epochs": 10} shortens training for this demo.
Details
res.obs_names holds the cell barcodes of the rows of res.output. When they match adata.obs_names, adata.obsm["X_Matilda"] = res.output adds the result to your AnnData.
mtb.run(..., dry_run=True) prints the command without running it. It works on any computer.
res = mtb.run("Matilda", "vertical", inputs=inputs, out_dir="out/Matilda_D11",
params={"epochs": 10})
emb = res.output
emb.shape
Fetching the method scripts from PYangLab/scMultiBench into /tmp/mbtut3/home/.cache/multibench/scMultiBench_ref ...
(2864, 100)
7. Score¶
evaluate scores the embedding against the cell-type labels. The first line selects the Leiden backend of the stored scores, so that section 8 compares like with like.
For a dataset with several label files, pass mtb.labels_for(dataset, category, method). It returns the files in the order the method stacks its cells, and a wrong order gives wrong scores without an error. evaluate needs an embedding: a method whose output is a graph, such as Seurat_WNN, has none to score.
Details
metrics="clustering" computes ARI, NMI, ASW, iASW, iF1 and cLISI. D11 has a single batch, so the batch metrics do not apply.
The default Leiden backend is igraph, which is faster. The two backends can move ARI by up to about 0.1.
mtb.config.DEFAULT.leiden_flavor = "leidenalg"
scores = mtb.evaluate(emb, labels=labels, category="vertical", metrics="clustering")
scores
Clustering 2,864 cells at 10 resolutions for ARI, NMI and iF1, with the leidenalg Leiden backend. This takes from seconds to a few minutes. To skip it, pass metrics= without ARI, NMI and iF1.
| Value | |
|---|---|
| ARI | 0.919230 |
| NMI | 0.912757 |
| ASW | 0.653349 |
| iASW | 0.643189 |
| iF1 | 0.887337 |
| cLISI | 0.998403 |
8. Plot¶
to_long turns the scores into a long table, with one row per metric. The cell adds this run to the package's stored D11 scores and plots them together as Matilda (this run).
Details
Circle size shows the rank within a column, and bigger is better. The fill compares a value with the other rows in the same column.
The stored scores hold only the demo datasets. Rows from your own dataset go in a figure of their own.
stored = mtb.load_results("vertical", dataset="D11", source="rerun")
mine = mtb.to_long(scores, method="Matilda (this run)", dataset="D11",
category="vertical")
mtb.plot.bubble(pd.concat([stored, mine], ignore_index=True))
Summary¶
| step | API |
|---|---|
| find a method | find_methods, method_info |
| resolve the data | inputs_for, labels_for |
| check the parameters | params_for |
| install the environment | mtb.env.install |
| run | run(..., params={...}) |
| score | evaluate |
| plot | to_long, plot.bubble |
The category tutorials (vertical, diagonal, mosaic, cross) run and score several methods in one call, on the demo data and on your own.