Skip to content

Run a method

mtb.run runs the methods you name on a dataset folder, each in its own environment, scores each output and saves the results. methods=["SCALEX"] runs one method.

Methods run on Linux, and on Windows and macOS (Apple silicon) for the methods with a Windows or macOS environment. SCALEX, the example of this page, runs on Linux and macOS. On an Intel Mac, mtb.scan(dataset, category, command=True) returns the plan with each command. Run the call itself on a Linux machine.

Summary

import multibench as mtb

mtb.data.fetch("D28")                  # 137 MB; skipped when already present
res = mtb.run("D28", "diagonal", methods=["SCALEX"], out_dir="runs/")
res.summary                            # status, run time and every score
emb = res.embedding("SCALEX")          # AnnData: obsm["X_emb"], 11014 x 10
Details

run finds the version of the method that fits the category and the files of the folder. It converts the files if needed, runs the method script in <out_dir>/<method>_<dataset>/ and scores its output.

mtb.scan creates no files: with command=True its command column holds each command, with <out_dir> for the output folder.

Some commands read a file that run writes first under <out_dir>/<method>_<dataset>/inputs/, such as a converted AnnData, the renamed peak file of GLUE and Seurat_v3, or the links to the data files of scMM, scMVP and scJoint. The caveat column of scan says so. Call mtb.run in the job for those methods, not the command alone.


Install the method's environment

Install the method's environment before its first run. mtb.env.install(["SCALEX"], dry_run=True) lists what is installed and what is missing, with the download sizes.

mtb.method_info("SCALEX")["env"]                # 'scmb_torch'
mtb.env.install(["SCALEX"])
Details

A missing environment marks the method not runnable in mtb.scan, and run skips it. On an Intel Mac the reason says that the environment runs only on Linux.

GLUE reads the GENCODE v43 human annotation, which its first run downloads, and MIRA a logger.py, which the package puts next to its script. GLUE runs its human script only, so mouse data does not run through the package. mtb.method_info(m)["setup_hint"] holds these notes, and the caveat column of scan lists them.


What run returns

res is a RunResult. res.summary has one row per method: its status, run time and scores, and its reason column says why a method failed or was skipped. res.embedding(method) returns the method's embedding as an AnnData, in the method's cell order: obsm["X_emb"]. Its obs holds each cell's label, and its batch when the run had more than one batch: the label file of each cell, or the batch= of run. The cell names are the barcodes of the dataset files. Where two files share a barcode, each name starts with its label file, such as cty2:AAACCTGAG....

res.summary[res.summary.status != "scored"]   # empty when every method scored
emb = res.embedding("SCALEX")
emb.obs.head()                     # label, and batch, per cell
res.long                           # the scores, as the long table
res = mtb.load_run("runs/")        # the same result, later

run also writes the cell names into each embedding.h5, as the dataset barcodes, so mtb.evaluate and mtb.io.read find each cell of the file by its name.

Details

res.records holds one dict per method, with the keys mtb.RunResult lists. Five methods put the cells in another order than RNA before ATAC and cty1, cty2, .... uniPort and Seurat_v5 put the ATAC cells first. StabMap puts its reference batch first, in cross and in the mosaic patterns that start with an RNA-only batch. SMILE and MultiVI in mosaic use their own batch order. embedding(method) and mtb.evaluate(..., dataset=, method=) follow that order.

method_info(m)["supports"] gives each variant's output_kind, embedding or graph. run scores scMoMaT on the neighbour graph it writes, as the paper does: ARI, NMI, iF1, cLISI, GC and iLISI. ASW, iASW, ASW_batch and the classification metrics need an embedding, so scMoMaT has none. MIRA and Seurat_WNN get the status ran, and the reason says that they write a graph.

res.rescore(batch=..., labels=..., metrics=...).save() scores the saved outputs again and writes the new scores. metrics= replaces those scores and keeps the others. No method runs again.


Set a method's parameters

Every method takes parameters. mtb.params_for(method, category, modalities) lists them with their defaults, which are the paper's values. A default of None leaves the setting to the method's library. run(params={method: {name: value}}) sets them. For several values of one parameter, loop over run, one out_dir per value, and name each run's rows:

import pandas as pd

p = mtb.params_for("scJoint", "diagonal")
p["tunable"]["epochs_stage1"]          # {'default': 20, 'type': 'int'}

runs = []
for v in (10, 30, 50):
    r = mtb.run("D28", "diagonal", methods=["scJoint"], out_dir=f"runs/e{v}",
                params={"scJoint": {"epochs_stage1": v}})
    runs.append(r.long.assign(method=f"scJoint epochs={v}"))
mtb.plot.bubble(pd.concat(runs))

Inputs

A dataset folder holds one file per role: a role is the name of one input, such as rna, atac_gas or cty, the label file. Dataset folder layout lists the file names, and mtb.scan(dataset, category) says which methods can run on a folder, and why not the rest.

mtb.method_info("SCALEX")["supports"][0]["modalities"]   # ['rna', 'atac_gas']
mtb.scan("D28", "diagonal")
Details

Each entry of supports is one variant: the version of a method for one category and one set of roles. For a method with several variants, run runs the one with the most of the folder's modalities. modalities=["rna", "adt"] picks another.

scan checks that each file is there and that no matrix is transposed. It also checks that each label file has as many rows as its matrix, and that files that must hold the same cells do. Its reason column names what is wrong.

Dataset folder layout

Put the files of one dataset in one folder, <data_path>/<DATASET>/, for example data/MYDATA/. The folder name is the dataset name. mtb.io.export_dataset writes the folder from an AnnData or MuData.

Category Files
vertical rna.h5 with adt.h5, atac.h5 or both, from the same cells, and cty.csv
diagonal rna.h5 and rna_cty.csv for the RNA cells, atac_peak.h5, atac_gas.h5 or both, and atac_cty.csv for the ATAC cells
mosaic one file per batch and modality, numbered by batch (rna1.h5, rna2.h5, atac2.h5 ...), and one label file per batch (cty1.csv, cty2.csv ...)
cross as mosaic, with atac_peak1.h5, atac_peak2.h5 ... for ATAC

Each .h5 file holds matrix/data (features x cells, the transpose of AnnData.X), matrix/features and matrix/barcodes. R's rhdf5 shows the dimensions reversed, so write a cells x features matrix from R. A label file is a one-column CSV: a header line, then one label per cell, in the order of the cells.

Vertical, mosaic and cross methods read ATAC as peaks. In diagonal, atac_peak.h5 holds peaks and atac_gas.h5 gene-activity scores of the same cells, and each method reads one of them or both:

ATAC files Diagonal methods
both MultiMAP, Seurat_v3
atac_peak.h5 Seurat_v5, GLUE
atac_gas.h5 SCALEX, scBridge, VIPCCA, Portal, uniPort, sciCAN, scJoint, iNMF, online_iNMF, Conos

For several RNA and several ATAC batches in diagonal, scJoint, GLUE, iNMF and online_iNMF read exactly 3 of each. They are numbered rna1.h5, atac_gas1.h5 or atac_peak1.h5, rna_cty1.csv and atac_cty1.csv, then 2 and 3. For another number of batches, see Another number of donors.

Mosaic and cross methods read exactly the batches of one of these patterns. Number yours to match one:

Category Batch 1 Batch 2 Batch 3 Methods
mosaic rna rna+adt adt Multigrate, scMoMaT, StabMap
mosaic rna rna+atac atac Multigrate, MultiVI, Cobolt, scMoMaT, SMILE, StabMap
mosaic rna+adt rna+atac rna Multigrate, scMoMaT, StabMap, UINMF
mosaic rna+adt rna+atac adt Multigrate, scMoMaT, StabMap
cross rna+adt rna+adt rna+adt Multigrate, totalVI, scMM, scMDC, scMoMaT, Concerto, sciPENN, StabMap, UINMF, MOFA2
cross rna+atac rna+atac rna+atac Multigrate, scMM, scMDC, scMoMaT, UnitedNet, StabMap, UINMF, MOFA2
cross adt+atac adt+atac Multigrate, scMoMaT, StabMap, UINMF, MOFA2
cross rna+adt+atac rna+adt+atac Multigrate, scMoMaT, StabMap, UINMF, MOFA2
cross rna+adt rna+adt UINMF
cross rna+atac rna+atac UINMF

Your own data as a dataset folder

mtb.io.export_dataset writes an AnnData or MuData as a dataset folder, which scan and run read.

For a Cell Ranger .h5 file, labels in a CSV file, data without labels or another number of donors, see Recipes for your own data.

Give raw counts for RNA, ADT and peaks. The methods normalise the data themselves. The folder stores each matrix dense, so drop genes or peaks you do not need first.

With ATAC, set atac_kind= to "peak" or "gene_activity". Most methods read one of the two, and method_info(m)["atac"] names it. MultiMAP and Seurat_v3 read both. scan and run skip a method whose file holds the other one, also when methods= names it.

For CITE-seq, section 5 of the RNA + ADT tutorial writes the folder. Other layouts:

import mudata as md      # pip install mudata

mdata = md.read_h5mu("my_multiome.h5mu")
# labels in mdata.obs; use rna:celltype for mdata["rna"].obs
mtb.io.export_dataset(mdata, "data/MYMULTIOME", rna="rna", atac="atac",
                      atac_kind="peak", labels="obs:celltype",
                      category="vertical")
mtb.scan("data/MYMULTIOME", "vertical", modalities=["rna", "atac"])
# on Linux, Windows or macOS: the RNA + ATAC methods; batch = each cell's sample
res = mtb.run("data/MYMULTIOME", "vertical", out_dir="out/",
              modalities=["rna", "atac_peak"], batch=mdata.obs["sample"])

Keep all samples in one folder and pass each cell's sample as batch=. atac_peak keeps the methods that read RNA and ATAC peaks.

import anndata as ad

rna = ad.read_h5ad("lung_rna.h5ad")
atac = ad.read_h5ad("lung_atac.h5ad")     # other cells than rna
mtb.io.export_dataset(rna, "data/LUNG", atac=atac, atac_kind="peak",
                      labels="obs:cell_type", category="diagonal")
peak_methods = mtb.find_methods("diagonal", modalities=["atac_peak"])
mtb.scan("data/LUNG", "diagonal", methods=peak_methods)

scan checks the four methods that read peaks. With peaks alone, only GLUE passes the file check. To add gene activity, see My ATAC is peaks only.

A mosaic or cross dataset can arrive as one file per batch. Here cite, multiome and rna_only hold one batch each.

kw = dict(labels="obs:cell_type", category="mosaic")
mtb.io.export_dataset(cite, "data/LAB", adt="obsm:protein",
                      batch_index=1, **kw)
mtb.io.export_dataset(multiome, "data/LAB", rna="rna", atac="atac",
                      atac_kind="peak", batch_index=2, **kw)
mtb.io.export_dataset(rna_only, "data/LAB", batch_index=3, **kw)
mtb.scan("data/LAB", "mosaic")

batch_index=N writes the whole object as batch N. Number the batches to match a pattern the methods read, as Dataset folder layout lists them. For cross, write each donor file the same way with category="cross". One file with a donor column is split by batch=:

mtb.io.export_dataset(adata, "data/MYCITE3", adt="obsm:protein",
                      labels="obs:celltype", batch="obs:donor",
                      category="cross")

The cross methods for RNA + ADT read batches 1-3. Dataset folder layout lists every batch pattern.

Details: files and labels

mtb.io.read reads a file of the folder back as an AnnData.

A label file is a one-column CSV: a header line x, then one label per cell, for example pd.Series(labels, name="x").to_csv(path, index=False).

A 10x Multiome AnnData read with gex_only=False holds genes and peaks in one X. Select each part with a feature filter: rna="X[feature_types=Gene Expression]" and atac="X[feature_types=Peaks]". Without a filter on such data, export_dataset raises ValueError. The recipe A 10x .h5 file reads the file and splits it.

Running an export again raises FileExistsError for the files already in the folder. overwrite=True replaces them. A call that fails writes nothing. An export also raises FileExistsError when the folder holds files of other data that it does not write: files with other cells, and after batch= files of other batch numbers. With batch_index=, a call checks its own batch number. The message names the first file. overwrite=True does not remove them. Write data with fewer batches into a new folder.

mtb.scan checks the folder for each method. Its caveat column flags values that are not whole numbers. When an expected file is missing, scan names the file it found instead.

Details: ATAC

The ATAC file name depends on the category. Vertical reads atac.h5. method_info(m)["atac"] names the form a method reads. Diagonal reads atac_peak.h5 for peaks and atac_gas.h5 for gene activity. Mosaic reads atac<i>.h5. The older name atac.h5 for gene activity still works.

GLUE and Seurat_v3 read peak names such as chr1:100-200. scan and run also skip these two methods when the peak names cannot be rewritten in that form, such as peak_0. With such names, scan cannot tell peaks from gene activity. The other methods run, with a caveat that names the file. scBridge reads the whole folder, and scan does not check its ATAC file. For a method skipped for its ATAC file, export the ATAC form it reads.

Recipes for your own data

Each task tutorial has a section "Your own data". It reads .h5ad files that hold the cell types in obs, writes a dataset folder and runs the methods on it. These recipes cover four other starting points. Run them in the tutorial notebook of your task. There mtb is imported, and METHODS is the list of methods from its section 2. Its check cell and run cell use NAME, the folder name that a recipe sets. The recipes write to mydata/ and call the cell-type column celltype, as the tutorials do.

A 10x .h5 file

Cell Ranger writes filtered_feature_bc_matrix.h5. One file holds one sample of cells measured in two modalities, which is the data of a vertical task. scan and run do not read this file. Read it with scanpy first.

import scanpy as sc

adata = sc.read_10x_h5("filtered_feature_bc_matrix.h5",   # <- your file
                       gex_only=False)
print(adata.var["feature_types"].value_counts())

gex_only=False keeps the peaks or the antibody counts. Without it, scanpy returns the genes alone. The printout lists the feature types of the file. A 10x Multiome file holds Gene Expression and Peaks, and a CITE-seq file holds Gene Expression and Antibody Capture.

scanpy warns that variable names are not unique, here and in the next steps. Do not call var_names_make_unique on adata: on the whole object it also renames an antibody that has the name of a gene. The split below calls it on the genes only.

The file holds no cell types. Add them to adata now, before the split, with the blocks of Labels in a CSV file, on adata whatever the tutorial, then come back here for the split. Without cell types, leave the labels= line out of the export below, as in No labels.

Split adata into one AnnData per modality and write the folder. For 10x Multiome, in the vertical RNA + ATAC tutorial:

kind = adata.var["feature_types"]
rna = adata[:, kind == "Gene Expression"].copy()
rna.var_names_make_unique()     # the gene names only
atac = adata[:, kind == "Peaks"].copy()

NAME = "MY10X"   # <- a name for your data
mtb.io.export_dataset(rna, "mydata/" + NAME,
                      rna="X",
                      atac=atac,
                      atac_kind="peak",
                      labels="obs:celltype",
                      overwrite=True)

For CITE-seq, in the vertical RNA + ADT tutorial:

kind = adata.var["feature_types"]
rna = adata[:, kind == "Gene Expression"].copy()
rna.var_names_make_unique()     # the gene names only
adt = adata[:, kind == "Antibody Capture"].copy()

NAME = "MY10X"   # <- a name for your data
mtb.io.export_dataset(rna, "mydata/" + NAME,
                      rna="X",
                      adt=adt,
                      labels="obs:celltype",
                      overwrite=True)

When export_dataset says that a matrix holds 0 features, compare the type name in the split with the feature types printed above. A type that is not printed is not in adata: read the file again with gex_only=False.

Then run the check cell and the run cell of that tutorial. They use NAME, so they work on this folder:

mtb.scan("mydata/" + NAME, "vertical", methods=METHODS)

For 10x Multiome the check lists atac.h5 and atac_peak.h5: one matrix under two names, one copy on disk.

Each matrix is stored dense and read dense: features × cells × 8 bytes of memory. 150,000 peaks and 10,000 cells take 12 GB. Keep the genes and peaks you need before the export. export_dataset raises ValueError when a modality would hold two feature types, for example rna="X" on the object before the split.

Labels in a CSV file

export_dataset reads the cell types from obs, not from a file. Read the CSV with pandas and add the column. The barcodes in the CSV must be unique: a duplicated barcode stops the block with the pandas error cannot reindex on an axis with duplicate labels. For diagonal data, do the recipe once for rna and once for atac.

In the code, adata is the object that holds your cells. In a tutorial that reads one object, it is adata. In the RNA + ATAC tutorials it is rna, and the name adata does not exist there: write rna for adata in every line of this recipe. With another name, such as adt or rna1, write the object whose cells the CSV lists.

import pandas as pd

ann = pd.read_csv("annotation.csv", index_col=0)               # <- your file, barcodes first
lab = ann["cluster_annotation"].astype(str).str.strip()        # <- your column
adata.obs["celltype"] = lab.where(~lab.isin(["", "NA", "nan"]))
print(adata.obs["celltype"].notna().sum(), "of", adata.n_obs, "cells have a label")

pandas matches the rows by barcode, so the CSV may be in any order. The third line turns a blank label and the text NA into a missing value, because a label file reads them as missing. When export_dataset names another such text, add it to that list.

The last line counts the cells that got a label. When it prints 0, or fewer cells than you expect, stop here. Some barcodes of the CSV are then spelled differently from adata.obs_names, often by an ending such as -1. Compare ann.index[:3] with adata.obs_names[:3], and do not go on until the number is right.

When the barcodes of the CSV lack the -1 ending that the cell names have, put the next line into the block above, after the line that reads the CSV, and run the block again. The line that reads the CSV comes first, so that a second run does not add the ending twice.

ann.index = ann.index + "-1"

A cell without a label cannot be scored, and export_dataset refuses it. When some cells have no label, keep the labelled cells.

adata = adata[adata.obs["celltype"].notna()].copy()

A second object of the same cells, such as atac, needs the same cells: atac = atac[adata.obs_names].copy(). Then go on with the export cell and labels="obs:celltype".

No labels

Cell types are used for the scores. A few methods also read them as an input. The check names those methods: their reason says cty.csv is missing. The label file that it names depends on the layout. For diagonal data it is rna_cty.csv, for some methods together with atac_cty.csv: the reason then holds rna_cty.csv is missing. or rna_cty.csv, atac_cty.csv are missing. Without cell types, leave labels= out of the export and pass evaluate=False to run.

The two blocks below are those of the vertical RNA + ADT tutorial. In another tutorial, do the same by removing the labels= line of that tutorial's export cell, and pass evaluate=False in its run cell.

NAME = "MYDATA"   # <- a name for your data
mtb.io.export_dataset(adata, "mydata/" + NAME,
                      rna="X",              # <- where the RNA counts are
                      adt="obsm:protein",   # <- where the proteins (ADT) are
                      overwrite=True)
mtb.scan("mydata/" + NAME, "vertical", methods=METHODS)
mine = mtb.run("mydata/" + NAME, "vertical", methods=METHODS,
               out_dir="out/" + NAME, evaluate=False)
mine.summary[["method", "status"]]

The status ran means that the method ran and nothing was scored. skipped marks a method that needs cell types as an input. Each method that ran has a folder in out/MYDATA/. For most methods the embedding is the file embedding.h5 in it. Without evaluate=False the methods run as well, and the status ran says that nothing was scored.

Diagonal data with cell types for the RNA cells only is the same case. run scores a diagonal folder only when both sides have labels. Leave labels= out and pass evaluate=False. No score compares methods on such data: scores of the RNA cells alone do not change with where a method puts the ATAC cells.

When you have cell types later, add them to the folder and score the saved embeddings. No method runs again. Here adata is the object whose cells the folder holds. In the vertical RNA + ATAC tutorial it is rna.

mtb.io.export_dataset(adata, "mydata/" + NAME, rna=None, labels="obs:celltype",
                      overwrite=True)
mine = mtb.load_run("out/" + NAME).rescore()
mine.save()                        # the new scores into out/MYDATA
mine.summary

export_dataset writes the labels in the cell order of the files already in the folder, whatever the row order of adata is by then. It refuses labels for other cells.

This line is for a folder without batch numbers that holds one set of cells, as the vertical tutorials write it. For cross, mosaic and diagonal data it does not apply: run the export cell of your tutorial again with its labels= line, and then the rescore line above. The files keep their cell order. Diagonal data needs the column in both files.

With cell types for some of the cells only, keep the labelled cells before the export, as in the recipe above. evaluate raises when a label is missing: score an embedding on its labelled cells, as mtb.evaluate(emb[emb.obs["label"].notna()], labels="label").

Another number of donors

A batch is a donor, a sample or an experiment. A cross method reads a fixed number of batches: three for RNA + ADT and for RNA + ATAC, and two for ADT + ATAC and for RNA + ADT + ATAC. Dataset folder layout lists them. With another number of donors, treat the data as a vertical task: keep all cells in one folder and give run the donor of each cell.

Do this in the vertical tutorial of your modalities. Its METHODS are vertical methods, and its section 2 installs their environments. ADT + ATAC has no vertical task: for that data, keep two donors, as shown further down, with two names in the list.

NAME = "MYDONORS"   # <- a name for your data
mtb.io.export_dataset(adata, "mydata/" + NAME,
                      rna="X",                # <- where the RNA counts are
                      adt="obsm:protein",     # <- where the proteins (ADT) are
                      labels="obs:celltype",  # <- the cell-type column
                      overwrite=True)         # no batch= here
mine = mtb.run("mydata/" + NAME, "vertical", methods=METHODS,
               out_dir="out/" + NAME,
               batch=adata.obs["donor"])  # <- your donor column
mine.summary

The methods do not read batch=. The summary then has the batch metrics ASW_batch, ARI_batch, NMI_batch, GC and iLISI, which say how well the donors mix. batch= takes the column itself, not its name. run reads the folder, and the folder holds no obs columns. The values are matched to the cells by barcode. Every cell needs a donor: run stops when one is missing.

For diagonal data with another number of donors than 3, do the same in the diagonal RNA + ATAC tutorial. Export rna and atac with all their cells, without batch_index=, and give run the donor of every RNA and ATAC cell:

import pandas as pd

mine = mtb.run("mydata/" + NAME, "diagonal", methods=METHODS,
               out_dir="out/" + NAME,
               batch=pd.concat([rna.obs["donor"], atac.obs["donor"]]))  # <- your column

To use the cross tutorial instead, keep as many donors as its methods read. First print the donor names of your file. This block changes nothing.

print(adata.obs["donor"].value_counts())   # the names in your file

Write as many of the printed names as the methods read into the list of the next block, in place of A, B and C. Then run it.

adata = adata[adata.obs["donor"].isin(["A", "B", "C"])].copy()   # <- the donors to keep

When none of the names is in the file, no cell is kept, and the export stops: The data holds 0 cells, so there is nothing to write. When one name is wrong, the cells of the other donors are kept, and the check names the files of the batch that is not there.

When the list holds a name that is not in the file, adata ends with fewer cells or with none, and a corrected list does not bring them back. Run the read cell of the tutorial again to get the data back, and then this block.

In a cross folder with more batches than a method reads, the method reads the first ones, and the check says so: StabMap reads batches 1-3 of 5. Batches 4-5 are not used. With fewer, it names the files that are not there: rna3.h5, adt3.h5 are missing. The batches are numbered in the sorted order of the donor names, read as text, so 10 comes before 2.


GPU or CPU?

Most methods also run without a GPU, only slower. Six need an NVIDIA GPU. On a host without one, scan and run mark them not runnable.

[m for m in mtb.find_methods() if mtb.method_info(m)["gpu"] == "required"]
# ['scBridge', 'moETM', 'SMILE', 'sciCAN', 'UnitedNet', 'iPOLNG']
mtb.method_info("scJoint")["cpu_params"]
# {'use_cuda': ''}
Details

scJoint and scMDC use CUDA unless told otherwise. On a host without a GPU, run adds the flags in method_info(m)["cpu_params"] and prints one line to stderr. A key you pass in params wins.

Six torch environments, env_sciPENN, matilda, scmb_scjoint, scmb_torch, scmb_scmm2 and scmb_multigrate2, also come as smaller CPU-only builds. mtb.env.install picks them on a host without a GPU, and flavor="cpu" or "gpu" overrides that. scMM never uses a GPU, so scmb_scmm2 comes as its CPU build on every host. Both builds install under the environment's own name, and the flavor column of mtb.env.install(..., dry_run=True) reports the installed one.


Next: score an embedding of your own, then draw the bubble table.