Vertical integration: RNA + ADT¶
Vertical integration combines modalities measured in the same cells. This tutorial is for RNA and surface protein, as in CITE-seq. It runs Matilda and sciPENN on D11, a CITE-seq dataset of 2,864 cells, and then on data in your own format.
Matilda and sciPENN run on Linux, Windows and macOS (Apple silicon). Or open in Colab: sections 1 to 4 take about 7 minutes there.
1. Install¶
pip may then print a note about restarting the kernel: no restart is needed, because the pins keep the numpy and pandas already installed.
import numpy, pandas
pins = f"numpy=={numpy.__version__} pandas=={pandas.__version__}"
%pip install -q "multibench-sc>=0.7.4" {pins}
Note: you may need to restart the kernel to use updated packages.
2. Download the data and the environments¶
mtb.data.fetch downloads D11 (11 MB) once. mtb.env.install downloads the environments for Matilda and sciPENN. On Linux the download is 1.1 GB on a computer without a GPU and 5.1 GB on one with a GPU. On Windows it is 0.3 GB and 2.8 GB. On macOS it is 0.7 GB.
import multibench as mtb
mtb.data.fetch("D11")
METHODS = ["Matilda", "sciPENN"]
envs = mtb.env.install(METHODS)
downloading D11 (11 MB) ... [env] downloading env_sciPENN (gpu build) 2.1 GB ... [env] 2.1 GB of 2.1 GB (32 MB/s) [env] unpacking env_sciPENN ... [env] downloading matilda (gpu build) 3.0 GB ... [env] 3.0 GB of 3.0 GB (22 MB/s) [env] unpacking matilda ...
3. Run the methods¶
run runs each method on D11 and scores its output with the scIB metrics. The clustering metrics score how well the embedding's clusters and neighbourhoods match the cell types, and how separate the clusters are. Higher is better for every metric.
Details
params={"Method": {"key": value}} sets a method's parameters, and mtb.params_for lists them.
mtb.load_run("out/D11") reloads these results later.
res = mtb.run("D11", "vertical", methods=METHODS, out_dir="out/D11")
res.summary
[run] Matilda (vertical/D11) ... [run] -> scored (27.7s) ARI 0.866 [run] sciPENN (vertical/D11) ... [run] -> scored (16.2s) ARI 0.476
| method | status | ARI | NMI | ASW | iASW | iF1 | cLISI | run_sec | caveat | reason | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Matilda | scored | 0.8657 | 0.8932 | 0.6662 | 0.6807 | 0.8247 | 0.9989 | 27.7 | ||
| 1 | sciPENN | scored | 0.4761 | 0.6388 | 0.6297 | 0.6489 | 0.5709 | 0.9798 | 16.2 |
4. Plot¶
res.plot() draws the scores as a bubble table. Circle size ranks the rows within each column and colour shows the value scaled per column, so a small gap can look large. res.summary has the values.
Details
Metrics are grouped by family: blue for clustering. Each family starts with an Overall bar: each row's mean rank over the family's metrics, scaled so that the lowest gets the shortest bar.
A column with fewer than two different values is drawn grey. A figure of one method is grey everywhere.
res.plot()
5. Your own data¶
Your data needs raw counts for both modalities and a cell type for each cell, in one .h5ad file.
Recipes cover a 10x .h5 file, labels in a CSV file, no labels and several donors or samples.
Details: export
The example keeps the protein names in adata.uns["protein_names"]. A DataFrame in adata.obsm["protein"] brings its own names in its columns.
The next cell writes a small example file, my_file.h5ad, from 60% of the cells of D11. Skip it for your own data.
mtb.data.write_example("D11")
wrote my_file.h5ad: 1,718 cells. RNA counts in X, protein counts in obsm["protein"], cell types in obs["celltype"].
Read your file. For your own data, change the file name marked # <-. On Colab, upload the file first: click the folder icon on the left, then the upload icon.
import anndata as ad
adata = ad.read_h5ad("my_file.h5ad") # <- your file
adata
AnnData object with n_obs × n_vars = 1718 × 9822
obs: 'celltype'
uns: 'protein_names'
obsm: 'protein'
layers: None (.X)
Write the dataset folder. Change each name in quotes marked # <- to one the printout above lists for your file, with its prefix, as "obs:celltype" or "obsm:protein".
NAME = "MYCITE" # <- a new name for your own data
mtb.io.export_dataset(adata, "mydata/" + NAME,
rna="X", # <- where the RNA counts are:
# "X" or "layer:counts"
adt="obsm:protein", # <- where the proteins (ADT) are
labels="obs:celltype", # <- the cell-type column
overwrite=True)
mtb.scan("mydata/" + NAME, "vertical", methods=METHODS)
[scan] 2 of 2 methods have their input files in MYCITE. 2 of 2 can run on this computer. MYCITE holds RNA and ADT. [scan] Cells per file: rna.h5 1,718, adt.h5 1,718.
| method | modalities | runnable | reason | env | caveat | |
|---|---|---|---|---|---|---|
| 0 | Matilda | rna+adt | True | matilda | ||
| 1 | sciPENN | rna+adt | True | env_sciPENN |
mine = mtb.run("mydata/" + NAME, "vertical", methods=METHODS,
out_dir="out/" + NAME)
mine.summary
[run] Matilda (vertical/MYCITE) ... [run] -> scored (17.8s) ARI 0.892 [run] sciPENN (vertical/MYCITE) ... [run] -> scored (14.3s) ARI 0.470
| method | status | ARI | NMI | ASW | iASW | iF1 | cLISI | run_sec | caveat | reason | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Matilda | scored | 0.8920 | 0.9181 | 0.6511 | 0.6760 | 0.8868 | 0.9993 | 17.8 | ||
| 1 | sciPENN | scored | 0.4696 | 0.6133 | 0.6359 | 0.6852 | 0.6026 | 0.9689 | 14.3 |
mine.plot()
6. Stored results¶
The package ships stored results for 13 methods on D11. load_results reads them and mtb.plot.bubble draws them, without running anything.
Details
Scores of your runs differ a little from the stored ones: some methods, such as sciPENN, give a slightly different embedding on each run.
stored = mtb.load_results("vertical", dataset="D11")
mtb.plot.bubble(stored)
7. Every method of this task¶
14 methods run this task. This notebook runs them all. Its 13 environments download 24.5 GB on a computer without a GPU and 31.9 GB on one with a GPU.
Troubleshooting¶
res.summary[res.summary.status != "scored"] lists each method that failed or was skipped. Its reason column ends with the method's error output.
Details
| symptom | fix |
|---|---|
| a method fails on a small dataset | mtb.scan names what the method needs from the data's size in its caveat column |
Next steps¶
- Run a method, Score an embedding, Load and plot results and Discover methods.
- Look at an embedding draws a method's embedding as a UMAP.
- The interactive explorer has the full benchmark's rankings, with no install needed.