Installation¶
pip install multibench-sc installs the API on Python 3.9 or newer. Methods
run on Linux, such as Google Colab, and on
Windows and macOS (Apple silicon) for the methods
listed there. On an Intel Mac you can use the stored results, check your
data with scan, score embeddings and plot.
Quick setup¶
One block creates an environment, installs multibench with JupyterLab and
adds a Jupyter kernel named multibench. Steps 1 to 3 below explain its
lines.
The last line prints the installed version. jupyter lab then opens
JupyterLab (on Windows, & "$HOME\multibench-env\Scripts\jupyter-lab.exe").
In a notebook, choose the kernel multibench. To run a method, install its
environment next: see Method environments.
python3 and py start a Python that is already installed. The tutorials
are run with Python 3.12. On Windows, py is the launcher that the
python.org installer adds, and py -3.12 asks for that version. If
PowerShell does not know py, write python in the first line.
With an AI assistant¶
An AI coding assistant that runs commands on your computer can do the setup for you. Give it this prompt:
Set up the Python package multibench on this computer and show me that it works.
Facts
- PyPI name: multibench-sc. Import: import multibench as mtb. It needs
Python 3.9 or newer.
- Docs: https://dsichang.github.io/scMultiBench/installation/
- multibench runs single-cell integration methods. Methods run on Linux and,
for the methods the docs list, on Windows and on a Mac with Apple silicon.
On an Intel Mac no method runs; stored results, data checks, scoring and
plots work.
- Each method runs in its own environment. mtb.env.install(["<method>"])
downloads it as a prebuilt archive: no conda, compiler or administrator
rights needed. print(mtb.config) shows where it goes (envs_dir: conda's
envs folder when conda is on PATH, else ~/.cache/multibench/envs).
Rules
- Make one new environment named multibench. With conda installed:
conda create -n multibench python=3.12. Without conda: a venv in
~/multibench-env from a Python that is already installed, 3.12 or newer
if there is one, else 3.9 or newer. If there is none, stop and ask me to
install Python.
- Install nothing else on the computer: no Python, no package manager,
nothing into the base or system Python. No sudo, no administrator rights.
If a conda environment named multibench, the folder ~/multibench-env or a
Jupyter kernel named multibench already exists, stop and ask me.
- In every command, call the environment's python by its full path instead
of relying on activation. On Windows that is Scripts\python.exe in a venv
and python.exe in a conda environment's own folder.
- Ask me before you install a method environment, and tell me its size:
mtb.env.install([...], dry_run=True) shows it. Ask me before you fetch a
dataset larger than 100 MB.
- Steps 2 and 5 take several minutes each. Wait for them, and do not start
them again.
- When a command fails, show me the error and what you would try next. Do
not skip a check to get past a failure.
Steps
1. Tell me the operating system, on a Mac whether it has Apple silicon or
an Intel processor, and whether conda is installed.
2. Create the environment and install into it:
python -m pip install multibench-sc jupyterlab
3. Add the Jupyter kernel: python -m ipykernel install --user --name multibench
4. Check, on any system:
python -c "import multibench as mtb; print(mtb.__version__); print(mtb.config); print(len(mtb.load_results('cross', dataset='D52')), 'stored result rows')"
It prints the version, the folders multibench uses and a row count above 0.
5. On Linux, Windows or a Mac with Apple silicon, after my yes, run one
method on a 2 MB demo dataset.
Save these lines as multibench_check.py and run the file with the
environment's python:
import multibench as mtb
mtb.data.fetch("D52mini")
mtb.env.install(["StabMap"])
res = mtb.run("D52mini", "cross", methods=["StabMap"], out_dir="multibench_check")
print(res.summary.T.to_string())
The status must be "scored". On Windows and macOS, cLISI and iLISI are
empty without a C++ compiler: that is expected. On an Intel Mac, skip this
step and say why.
Then report: the environment's location, the multibench version, the kernel
name, the outcome of step 5, and the commands that remove the environment,
the kernel, the folders that print(mtb.config) names and the files of step 5.
Step 1: Create an isolated environment¶
Any fresh Python 3.9+ environment works. Conda is optional: the method environments install without it.
Step 2: Install multibench¶
Dependencies
The install brings numpy, pandas, h5py, anndata, matplotlib,
pyyaml, scanpy and scib. PyTorch, R and the method packages are not
installed here. Each method's environment holds them.
Developer install
Clone the repository only to edit the notebooks or run the test suite:
Step 3: Verify the install¶
These calls need no method environment:
Dataset folder layout shows how to lay out your own dataset.
Method environments¶
Each method runs in its own environment. Install only the ones you
need. mtb.env.install downloads prebuilt archives, with no conda needed.
import multibench as mtb
# the environments a category needs: installed or not, with their sizes
mtb.env.install(category="vertical", dry_run=True)
# one method
mtb.env.install(["StabMap"])
# one category
mtb.env.install(category="vertical")
Details
mtb.config.envs_diris where they go:$MULTIBENCH_ENVS_DIRwhen set, else the writable envs directory of the conda or mamba onPATH, else~/.cache/multibench/envs.print(mtb.config)shows the folder in use.- A failed archive download falls back to a build from the lockfile, which needs conda or mamba and takes longer.
flavor="cpu"or"gpu"picks the build.flavor="gpu"installs the GPU build from a host without a GPU, such as a cluster login node; by default such a host gets the CPU build.- The totals of the dry run: categories share environments, so two categories cost less than the sum of their totals.
mtb.env.installskips environments that already exist, so a rerun resumes an interrupted install. With no methods and nocategory, it installs all 24. Check the total with the dry run first (reference).
Details: method scripts
mtb.env.install downloads the method scripts with the environments,
and a run downloads them if they are missing. They go to
mtb.config.repo_path, which is
~/.cache/multibench/scMultiBench_scripts/<id> after pip install.
The variable MULTIBENCH_SCRIPTS_REF instead fetches a given commit or
tag of PYangLab/scMultiBench with git.
On a host without network, run mtb.env.install on a connected machine
that sees the same home folder, or copy mtb.config.repo_path from one
and set MULTIBENCH_REPO_PATH to the copy.
Google Colab¶
The tutorials open in Colab from their badge and run on its default CPU runtime. Each notebook installs the environments of the methods it runs and runs them, with no conda needed.
Details
On a GPU runtime, an environment that has a GPU build gets that build: a larger download, and faster training.
Windows¶
mtb.env.install and mtb.run also work on Windows, without conda, WSL or
administrator rights, for these methods (tested on Windows 11):
| Windows environment | Methods |
|---|---|
| A CPU build and a GPU build | Matilda, sciPENN, Cobolt, Multigrate, MultiVI, totalVI, scMDC, scMoMaT, MultiMAP, uniPort, Portal |
| A GPU build only: the method needs an NVIDIA GPU, as on Linux | moETM, SMILE, iPOLNG, scBridge, sciCAN, UnitedNet |
| One build, on the CPU | StabMap, iNMF, online_iNMF, Conos, Seurat_v5, Seurat_WNN, MOFA2, scMSI, scMVP, VIMCCA, VIPCCA |
UINMF, SCALEX, Concerto, scMM, GLUE, scJoint, Seurat_v3 and MIRA have no
Windows environment. Asked for on Windows, they are refused: "<method> has
no Windows environment. Run it on a Linux machine, such as Colab."
Details
- Name the methods to install.
mtb.env.install(category=...)and the call without methods are refused on Windows, because every category has a method without a Windows environment. - Keep the path of
mtb.config.envs_dirshort. Windows limits a file path to 259 characters, andmtb.env.installrefuses a folder that leaves too little room. - Nothing is built on Windows:
build=Trueis refused, and a download that fails raises an error. - PyTorch needs the Microsoft Visual C++ Redistributable. Install it if a method stops because a DLL is missing.
- A user name with a space can stop the R methods (StabMap, Seurat, iNMF
and Conos). Set
TMPDIRto a folder without a space before you start Python. cLISIandiLISIneed a C++ compiler on Windows, such as theg++of Rtools. Without one they areNaN, with a warning (Troubleshooting).
macOS¶
mtb.env.install and mtb.run also work on a Mac with Apple silicon,
without conda, for these methods: Matilda, sciPENN, Cobolt, Multigrate,
MultiVI, totalVI, scMDC, scMoMaT, MultiMAP, uniPort, Portal, StabMap, iNMF,
online_iNMF, Seurat_v5, Seurat_WNN, UINMF, SCALEX, GLUE, scJoint, Seurat_v3
and MIRA. Each environment has one build, on the CPU.
moETM, SMILE, iPOLNG, scBridge, sciCAN, UnitedNet, Conos, MOFA2, scMSI,
scMVP, VIMCCA, VIPCCA, Concerto and scMM have no macOS environment. Asked
for on macOS, they are refused: "<method> has no macOS environment. Run it
on a Linux machine, such as Colab."
No method runs on an Intel Mac.
Details
- Name the methods to install.
mtb.env.install(category=...)and the call without methods are refused on macOS, because every category has a method without a macOS environment. - Nothing is built on macOS:
build=Trueis refused. cLISIandiLISIneed a C++ compiler on macOS, such as the one in the Xcode command-line tools. Without one they areNaN, with a warning (Troubleshooting).
Get the data¶
Each tutorial downloads its dataset on first run. Datasets go to
mtb.config.data_path: ~/.cache/multibench/data after
pip install, <repo>/data in a developer install.
import multibench as mtb
mtb.catalog.datasets().dataset.tolist() # ['D11', 'D28', 'D45', 'D46', 'D52', 'D22mini', ...]
mtb.data.fetch("D11") # no-op when already present
Details
The tutorials run on one dataset per integration task. D11 is a
benchmark dataset; each mini is a smaller copy of one, with a subset of
its cells, genes and peaks:
| Task | Dataset | Size |
|---|---|---|
| vertical, RNA + ADT | D11 |
11 MB |
| vertical, RNA + ATAC | D27mini_vertical |
24 MB |
| vertical, RNA + ADT + ATAC | D22mini |
8 MB |
| diagonal, [RNA, ATAC] | D27mini |
15 MB |
| diagonal, [multiple RNA, multiple ATAC] | D37mini |
12 MB |
| mosaic, [RNA, RNA + ADT, ADT] | D38mini |
2 MB |
| mosaic, [RNA, RNA + ATAC, ATAC] | D45mini |
11 MB |
| mosaic, Mixed, with shared modality | D46mini |
4 MB |
| mosaic, Mixed, without shared modality | D49mini |
5 MB |
| cross, Multiple RNA + ADT | D52mini |
2 MB |
| cross, Multiple RNA + ATAC | D56mini |
10 MB |
| cross, Multiple ADT + ATAC | D58mini |
7 MB |
| cross, Multiple RNA + ADT + ATAC | D59mini |
8 MB |
The benchmark datasets D28 (diagonal, 137 MB), D45 (mosaic, 290 MB),
D46 (mosaic, 97 MB) and D52 (cross, 179 MB) are also fetchable.
MULTIBENCH_DATA_PATH sets the data path. Without it, and with
XDG_CACHE_HOME set, the data path after pip install is
$XDG_CACHE_HOME/multibench/data. print(mtb.config) shows the
path in use.
To download a dataset by hand, for example D11:
DATA=~/.cache/multibench/data # the data_path that print(mtb.config) shows
URL=https://github.com/DSichang/scMultiBench/releases/download/data-v1
mkdir -p "$DATA"
wget -qO- "$URL/D11.tar.gz" | tar xz -C "$DATA"
The full collection of processed datasets is linked from the
scMultiBench README.
Unpack a dataset under the data path, or anywhere else and pass its
folder path as dataset.
Troubleshooting¶
OSError: Environment <env> of <method> is not installed
The method's environment is missing. Run the install command the message
names, for example mtb.env.install(["<method>"]).
mtb.scan() names it per method.
Environment <env> was not installed by this version of multibench
A conda environment has the name of a method's environment, but this
version of multibench did not install it, so it is not used: an
environment you made yourself, or one that multibench 0.3.9 or older
built from a recipe, or 0.3.0 or older unpacked from an archive. On
Linux, rename or remove the folder the message names, then run
mtb.env.install(["<method>"]). On Windows and macOS, run it as it is:
when that folder is in conda's envs folder, 0.7.3 leaves it alone,
unpacks into ~/.cache/multibench/envs and prints
[env] <folder>/<env> is another environment, so <env> goes to <folder>.
OSError: Methods run only on Linux ...
The message names this computer's system. Methods do not run on an
Intel Mac or on a
system other than Linux, Windows and macOS. Run the call on a Linux
machine, such as Colab.
Make the commands there: mtb.scan(..., command=True) writes this
computer's paths into them.
mtb.env.install raises the same error.
OSError: <method> has no Windows environment ..., or no macOS environment
The method has no environment for this system. Windows and macOS list the methods that run there. Run the others on a Linux machine, such as Colab.
OSError naming envs_dir, with MULTIBENCH_RUN_MODE=prefix
Prefix mode is forced, but the environment is not under envs_dir.
Install it (mtb.env.install(["<method>"])),
point MULTIBENCH_ENVS_DIR at the folder that holds it, or unset
MULTIBENCH_RUN_MODE.
RuntimeError: Conda is not installed on this computer. build=True needs it.
Only a lockfile build needs conda. Leave out build=True.
ImportError: No module named scib from mtb.evaluate
multibench was installed with --no-deps or into another interpreter.
Run pip install multibench-sc in the environment you import from.
TypingError: Failed in nopython mode pipeline ..., naming knn_dists
Scoring builds a neighbour graph with scanpy, and umap-learn 0.6.0
fails there with scanpy 1.12.5 on 8,192 cells or more. mtb.evaluate
raises the error, and a run ends with the status ran and the error in
reason. Install another umap-learn release:
Then restart Python and call evaluate again, or score a finished run
with mtb.load_run(out_dir).rescore().
scanpy ... cannot read the neighbours pynndescent ... found
Scoring builds a neighbour graph with scanpy, and scanpy 1.12.0 and
earlier cannot read what pynndescent 0.7 returns on 8,192 cells or more.
multibench-sc 0.7.2 installs the pynndescent line before it. Earlier
releases can meet the failure, and report it as
AttributeError: 'csr_array' object has no attribute 'getnnz'. Install
the earlier pynndescent line:
Then restart Python and call evaluate again, or score a finished run
with mtb.load_run(out_dir).rescore().
cLISI / iLISI are NaN, with a warning naming knn_graph.o
Both need scib's compiled LISI helper, which ships as a Linux x86-64
binary. On other platforms evaluate() rebuilds it automatically when a
C++ compiler (g++, c++ or clang++) is on PATH.
Without a compiler, install one (Xcode command-line tools on macOS, the
g++ of Rtools on Windows) and rerun evaluate in a new Python session,
or run the g++ command the warning prints. The other metrics are
computed either way.
unsupported input format
mtb.io.export_dataset takes an in-memory AnnData or MuData. Read
other formats into an AnnData first.
If the problem remains,
open an issue with the
output of python -c "import multibench as mtb; print(mtb.__version__)" and
pip freeze.
Next: the Quickstart, or run a method with Run a method.