Discover methods¶
These calls show which methods fit your data and your task. They need no environments, GPU or data.
The four integration categories¶
Pick the category that matches your experimental design and pass it as
category= first. A method written for one design cannot take data from
another.
| Category | Design | Typical modalities |
|---|---|---|
| vertical | Several modalities measured in the same cells | RNA+ADT, RNA+ATAC, RNA+ADT+ATAC |
| diagonal | Modalities measured in different cells, bridged by shared features | RNA batch + ATAC batch |
| mosaic | Several batches with different modalities, linked by a batch that measures two of them | mixed, partially paired |
| cross | Several batches, each with every modality | RNA+ADT, RNA+ATAC, ADT+ATAC, RNA+ADT+ATAC |
Several samples of 10x Multiome (RNA+ATAC) belong in vertical: one folder
with all cells. To score batch mixing, pass each cell's sample as batch=
to run or evaluate.
Filter by what your data has¶
find_methods
lists the methods, all of them without a filter. Its filters are what your data
has: the category, the modalities you measured, how your ATAC is encoded,
and whether you have cell-type labels.
import multibench as mtb
mtb.find_methods() # all methods
mtb.find_methods(category="vertical") # paired-design methods only
Each method reads ATAC either as peaks or as gene activity. Give each
method the representation it reads: atac_peak or atac_gas in
modalities= keeps the methods that read yours.
# unpaired RNA and ATAC batches, no cell-type labels
mtb.find_methods(category="diagonal", modalities=["rna", "atac"],
needs_labels=False)
# ['SCALEX', 'Seurat_v5', 'VIPCCA', 'Portal', ...]
mtb.find_methods(category="diagonal", modalities=["atac_peak"]) # ATAC as peaks
Details
A method matches when one of its variants meets all the filters
together. A variant is the version of a method for one category and
one set of inputs. None, the default, turns a filter off.
modalities= keeps the variants that take every listed type, so
["rna"] is broad and ["rna", "atac"] stricter. "atac" matches
any ATAC input, and protein means adt.
The other ATAC tokens select by what the method reads. "gas",
"gene_activity" and "atac_gas" keep the methods that read gene
activity. "peak" and "atac_peak" keep those that read peaks,
including moETM, scMM and iPOLNG, whose input is named atac_gas.
Naming both representations keeps MultiMAP and Seurat_v3, which read
both files.
mtb.scan and run use the same tokens. They keep a row only
when its modalities are exactly the listed ones, so scMoMaT's
rna+adt+atac row is not kept for ["rna", "atac"]. A representation
token keeps every row whose method reads it, so "atac_peak" also
keeps MultiMAP and Seurat_v3.
needs_labels is judged per variant. method_info(m)["needs_labels"]
is True when any variant needs labels.
An unknown token raises ValueError listing the valid values. A bare
string such as modalities="rna" raises TypeError.
My ATAC is peaks only¶
Most diagonal methods read gene activity. With a peak matrix alone, check which methods can run:
- GLUE runs on peaks. It reads peak names such as
chr1:100-200, and a run rewriteschr1_100_200orchr1-100-200into that form. It also needs the GENCODE annotation file. See Run a method. - Seurat_v5 needs a third dataset: a paired bridge with RNA and ATAC from the same cells. The package passes your RNA and ATAC files as that bridge, so it cannot run on unpaired data here.
- MultiMAP and Seurat_v3 also need gene activity for the same ATAC cells.
- The other ten need gene activity instead of peaks, for the ATAC cells
of
atac_peak.h5. Make it outside the package, for example with SignacGeneActivity, read it into an AnnDatagas, and write it into the folder withmtb.io.export_dataset(gas, "data/X", rna=None, atac="X", atac_kind="gene_activity", category="diagonal").
mtb.scan then shows which methods have their files.
Details
mtb.scan compares the cells of atac_gas.h5 with those of
atac_peak.h5. Other cells, or the same cells in another order, fail
the file check. Barcodes that differ only in a suffix such as -1 or
-2 count as the same cell.
export_dataset writes the rows in the cell order of the folder's
atac_peak.h5, which atac_cty.csv follows, and refuses other cells.
Inspect a single method¶
method_info
returns what the package knows about one method.
info = mtb.method_info("SCALEX")
info["env"], info["atac"], info["needs_labels"]
# ('scmb_torch', 'gene_activity', False)
info["supports"] # one entry per variant: category, modalities, labels, ...
info["runtime"] # {'measured_sec': 301, 'tier': 'medium', 'worst_sec': 2233, ...}
Details
measured_sec is the package's own run on the full stored dataset, and
tier (fast to very_slow) follows it. worst_sec, the longest of
runtime["observed"], sets run(timeout=...). On a CPU, training
methods take much longer than these times.
info["setup_hint"] names what a method needs before its first run,
such as GLUE's annotation file. setup_hint is empty for most methods.
mtb.params_for(method, category, modalities) lists what run(params=...) can change
and what the script fixes.
Unknown method ids raise KeyError. Every function takes a method id in
any case, and the paper's spellings such as MOFA+.
The keys of method_info are listed in its
reference entry.
gpu says how a method uses an NVIDIA GPU: required, used when
present, not used or unknown. cpu_params holds the flags that
turn CUDA off, which a run adds on a computer without a GPU.
Rank from stored results¶
rank
ranks a category's methods by their stored results, in the order of the
figures.
mtb.rank("vertical", modalities=["rna", "adt"])
# method grand_score needs_labels runtime_tier worst_sec env
mtb.rank("diagonal", modalities=["atac_peak"]) # only methods that read peaks
mtb.rank("vertical", methods=mtb.find_methods("vertical", needs_labels=False))
grand_score compares a category's methods with each other: it is a rank,
not a quality score. How it evaluates says
how it is computed.
The stored results cover one dataset per category: for vertical, D11 (RNA + ADT). Read the
ranking as a hint. The full benchmark ranking is in the
interactive explorer and in
Fig. 6 of the paper.
Methods with needs_labels=True train on your cell types, so compare them
only with each other.
Details
The Seurat_v5 script reads its bridge as two extra files. Its stored diagonal scores come from runs with a separate paired bridge dataset. They do not mean that it runs on your unpaired folder.
To rank your own method alongside, pass
long_df=pd.concat([mtb.load_results(category), mine]).
Methods that fit the category but have no stored results are listed
last, with grand_score NaN.
modalities= naming a modality that no ranked dataset measured raises.
modalities= follows the find_methods rule above. It only shortens
the list, so a filtered list can start below 1 and end above 0.
metrics= picks the metric family. The default ranks on every metric
present, as the figures do. needs_labels follows the category's variants.
The full benchmark catalog¶
mtb.catalog returns the dataset and metric tables as DataFrames.
mtb.catalog.datasets() # the fetchable datasets: category, has_results
mtb.catalog.metrics() # each metric with its family and what it measures
Details
The rows are the demo datasets, which mtb.data.fetch downloads.
has_results marks the datasets with stored results.
Next: run a chosen method in Run a method, score its output in Score an embedding, and compare methods in Load and plot results.