Skip to content

Discover methods

These calls show which methods fit your data and your task. They need no environments, GPU or data.


The four integration categories

Pick the category that matches your experimental design and pass it as category= first. A method written for one design cannot take data from another.

Category Design Typical modalities
vertical Several modalities measured in the same cells RNA+ADT, RNA+ATAC, RNA+ADT+ATAC
diagonal Modalities measured in different cells, bridged by shared features RNA batch + ATAC batch
mosaic Several batches with different modalities, linked by a batch that measures two of them mixed, partially paired
cross Several batches, each with every modality RNA+ADT, RNA+ATAC, ADT+ATAC, RNA+ADT+ATAC

Several samples of 10x Multiome (RNA+ATAC) belong in vertical: one folder with all cells. To score batch mixing, pass each cell's sample as batch= to run or evaluate.


Filter by what your data has

find_methods lists the methods, all of them without a filter. Its filters are what your data has: the category, the modalities you measured, how your ATAC is encoded, and whether you have cell-type labels.

import multibench as mtb

mtb.find_methods()                       # all methods
mtb.find_methods(category="vertical")    # paired-design methods only

Each method reads ATAC either as peaks or as gene activity. Give each method the representation it reads: atac_peak or atac_gas in modalities= keeps the methods that read yours.

# unpaired RNA and ATAC batches, no cell-type labels
mtb.find_methods(category="diagonal", modalities=["rna", "atac"],
                 needs_labels=False)
# ['SCALEX', 'Seurat_v5', 'VIPCCA', 'Portal', ...]

mtb.find_methods(category="diagonal", modalities=["atac_peak"])   # ATAC as peaks
Details

A method matches when one of its variants meets all the filters together. A variant is the version of a method for one category and one set of inputs. None, the default, turns a filter off.

modalities= keeps the variants that take every listed type, so ["rna"] is broad and ["rna", "atac"] stricter. "atac" matches any ATAC input, and protein means adt.

The other ATAC tokens select by what the method reads. "gas", "gene_activity" and "atac_gas" keep the methods that read gene activity. "peak" and "atac_peak" keep those that read peaks, including moETM, scMM and iPOLNG, whose input is named atac_gas. Naming both representations keeps MultiMAP and Seurat_v3, which read both files.

mtb.scan and run use the same tokens. They keep a row only when its modalities are exactly the listed ones, so scMoMaT's rna+adt+atac row is not kept for ["rna", "atac"]. A representation token keeps every row whose method reads it, so "atac_peak" also keeps MultiMAP and Seurat_v3.

needs_labels is judged per variant. method_info(m)["needs_labels"] is True when any variant needs labels.

An unknown token raises ValueError listing the valid values. A bare string such as modalities="rna" raises TypeError.

My ATAC is peaks only

Most diagonal methods read gene activity. With a peak matrix alone, check which methods can run:

  • GLUE runs on peaks. It reads peak names such as chr1:100-200, and a run rewrites chr1_100_200 or chr1-100-200 into that form. It also needs the GENCODE annotation file. See Run a method.
  • Seurat_v5 needs a third dataset: a paired bridge with RNA and ATAC from the same cells. The package passes your RNA and ATAC files as that bridge, so it cannot run on unpaired data here.
  • MultiMAP and Seurat_v3 also need gene activity for the same ATAC cells.
  • The other ten need gene activity instead of peaks, for the ATAC cells of atac_peak.h5. Make it outside the package, for example with Signac GeneActivity, read it into an AnnData gas, and write it into the folder with mtb.io.export_dataset(gas, "data/X", rna=None, atac="X", atac_kind="gene_activity", category="diagonal").

mtb.scan then shows which methods have their files.

Details

mtb.scan compares the cells of atac_gas.h5 with those of atac_peak.h5. Other cells, or the same cells in another order, fail the file check. Barcodes that differ only in a suffix such as -1 or -2 count as the same cell.

export_dataset writes the rows in the cell order of the folder's atac_peak.h5, which atac_cty.csv follows, and refuses other cells.


Inspect a single method

method_info returns what the package knows about one method.

info = mtb.method_info("SCALEX")
info["env"], info["atac"], info["needs_labels"]
# ('scmb_torch', 'gene_activity', False)
info["supports"]    # one entry per variant: category, modalities, labels, ...
info["runtime"]     # {'measured_sec': 301, 'tier': 'medium', 'worst_sec': 2233, ...}
Details

measured_sec is the package's own run on the full stored dataset, and tier (fast to very_slow) follows it. worst_sec, the longest of runtime["observed"], sets run(timeout=...). On a CPU, training methods take much longer than these times.

info["setup_hint"] names what a method needs before its first run, such as GLUE's annotation file. setup_hint is empty for most methods. mtb.params_for(method, category, modalities) lists what run(params=...) can change and what the script fixes.

Unknown method ids raise KeyError. Every function takes a method id in any case, and the paper's spellings such as MOFA+.

The keys of method_info are listed in its reference entry. gpu says how a method uses an NVIDIA GPU: required, used when present, not used or unknown. cpu_params holds the flags that turn CUDA off, which a run adds on a computer without a GPU.


Rank from stored results

rank ranks a category's methods by their stored results, in the order of the figures.

mtb.rank("vertical", modalities=["rna", "adt"])
#   method  grand_score  needs_labels  runtime_tier  worst_sec  env
mtb.rank("diagonal", modalities=["atac_peak"])   # only methods that read peaks
mtb.rank("vertical", methods=mtb.find_methods("vertical", needs_labels=False))

grand_score compares a category's methods with each other: it is a rank, not a quality score. How it evaluates says how it is computed.

The stored results cover one dataset per category: for vertical, D11 (RNA + ADT). Read the ranking as a hint. The full benchmark ranking is in the interactive explorer and in Fig. 6 of the paper. Methods with needs_labels=True train on your cell types, so compare them only with each other.

Details

The Seurat_v5 script reads its bridge as two extra files. Its stored diagonal scores come from runs with a separate paired bridge dataset. They do not mean that it runs on your unpaired folder.

To rank your own method alongside, pass long_df=pd.concat([mtb.load_results(category), mine]).

Methods that fit the category but have no stored results are listed last, with grand_score NaN. modalities= naming a modality that no ranked dataset measured raises.

modalities= follows the find_methods rule above. It only shortens the list, so a filtered list can start below 1 and end above 0.

metrics= picks the metric family. The default ranks on every metric present, as the figures do. needs_labels follows the category's variants.


The full benchmark catalog

mtb.catalog returns the dataset and metric tables as DataFrames.

mtb.catalog.datasets()    # the fetchable datasets: category, has_results
mtb.catalog.metrics()     # each metric with its family and what it measures
Details

The rows are the demo datasets, which mtb.data.fetch downloads. has_results marks the datasets with stored results.


Next: run a chosen method in Run a method, score its output in Score an embedding, and compare methods in Load and plot results.