Skip to content

Installation

pip install multibench-sc installs the API on Python 3.9 or newer. Methods run on Linux, such as Google Colab, and on Windows and macOS (Apple silicon) for the methods listed there. On an Intel Mac you can use the stored results, check your data with scan, score embeddings and plot.


Quick setup

One block creates an environment, installs multibench with JupyterLab and adds a Jupyter kernel named multibench. Steps 1 to 3 below explain its lines.

terminal
conda create -n multibench python=3.12 -y
conda activate multibench
pip install multibench-sc jupyterlab
python -m ipykernel install --user --name multibench
python -c "import multibench as mtb; print(mtb.__version__)"
terminal
python3 -m venv ~/multibench-env
source ~/multibench-env/bin/activate
pip install multibench-sc jupyterlab
python -m ipykernel install --user --name multibench
python -c "import multibench as mtb; print(mtb.__version__)"
PowerShell
py -m venv "$HOME\multibench-env"
$py = "$HOME\multibench-env\Scripts\python.exe"
& $py -m pip install multibench-sc jupyterlab
& $py -m ipykernel install --user --name multibench
& $py -c "import multibench as mtb; print(mtb.__version__)"

The last line prints the installed version. jupyter lab then opens JupyterLab (on Windows, & "$HOME\multibench-env\Scripts\jupyter-lab.exe"). In a notebook, choose the kernel multibench. To run a method, install its environment next: see Method environments.

python3 and py start a Python that is already installed. The tutorials are run with Python 3.12. On Windows, py is the launcher that the python.org installer adds, and py -3.12 asks for that version. If PowerShell does not know py, write python in the first line.

With an AI assistant

An AI coding assistant that runs commands on your computer can do the setup for you. Give it this prompt:

prompt
Set up the Python package multibench on this computer and show me that it works.

Facts
- PyPI name: multibench-sc. Import: import multibench as mtb. It needs
  Python 3.9 or newer.
- Docs: https://dsichang.github.io/scMultiBench/installation/
- multibench runs single-cell integration methods. Methods run on Linux and,
  for the methods the docs list, on Windows and on a Mac with Apple silicon.
  On an Intel Mac no method runs; stored results, data checks, scoring and
  plots work.
- Each method runs in its own environment. mtb.env.install(["<method>"])
  downloads it as a prebuilt archive: no conda, compiler or administrator
  rights needed. print(mtb.config) shows where it goes (envs_dir: conda's
  envs folder when conda is on PATH, else ~/.cache/multibench/envs).

Rules
- Make one new environment named multibench. With conda installed:
  conda create -n multibench python=3.12. Without conda: a venv in
  ~/multibench-env from a Python that is already installed, 3.12 or newer
  if there is one, else 3.9 or newer. If there is none, stop and ask me to
  install Python.
- Install nothing else on the computer: no Python, no package manager,
  nothing into the base or system Python. No sudo, no administrator rights.
  If a conda environment named multibench, the folder ~/multibench-env or a
  Jupyter kernel named multibench already exists, stop and ask me.
- In every command, call the environment's python by its full path instead
  of relying on activation. On Windows that is Scripts\python.exe in a venv
  and python.exe in a conda environment's own folder.
- Ask me before you install a method environment, and tell me its size:
  mtb.env.install([...], dry_run=True) shows it. Ask me before you fetch a
  dataset larger than 100 MB.
- Steps 2 and 5 take several minutes each. Wait for them, and do not start
  them again.
- When a command fails, show me the error and what you would try next. Do
  not skip a check to get past a failure.

Steps
1. Tell me the operating system, on a Mac whether it has Apple silicon or
   an Intel processor, and whether conda is installed.
2. Create the environment and install into it:
   python -m pip install multibench-sc jupyterlab
3. Add the Jupyter kernel: python -m ipykernel install --user --name multibench
4. Check, on any system:
   python -c "import multibench as mtb; print(mtb.__version__); print(mtb.config); print(len(mtb.load_results('cross', dataset='D52')), 'stored result rows')"
   It prints the version, the folders multibench uses and a row count above 0.
5. On Linux, Windows or a Mac with Apple silicon, after my yes, run one
   method on a 2 MB demo dataset.
   Save these lines as multibench_check.py and run the file with the
   environment's python:
     import multibench as mtb
     mtb.data.fetch("D52mini")
     mtb.env.install(["StabMap"])
     res = mtb.run("D52mini", "cross", methods=["StabMap"], out_dir="multibench_check")
     print(res.summary.T.to_string())
   The status must be "scored". On Windows and macOS, cLISI and iLISI are
   empty without a C++ compiler: that is expected. On an Intel Mac, skip this
   step and say why.

Then report: the environment's location, the multibench version, the kernel
name, the outcome of step 5, and the commands that remove the environment,
the kernel, the folders that print(mtb.config) names and the files of step 5.

Step 1: Create an isolated environment

Any fresh Python 3.9+ environment works. Conda is optional: the method environments install without it.

terminal
conda create -n multibench python=3.12 -y
conda activate multibench
terminal
python3 -m venv .venv
source .venv/bin/activate

On Windows, activate it with .venv\Scripts\activate.


Step 2: Install multibench

terminal
pip install multibench-sc
Dependencies

The install brings numpy, pandas, h5py, anndata, matplotlib, pyyaml, scanpy and scib. PyTorch, R and the method packages are not installed here. Each method's environment holds them.

Developer install

Clone the repository only to edit the notebooks or run the test suite:

terminal
git clone https://github.com/DSichang/scMultiBench.git
cd scMultiBench
pip install -e ".[dev]" --config-settings editable_mode=compat

Step 3: Verify the install

terminal
python -c "import multibench as mtb; print(mtb.__version__)"

These calls need no method environment:

import multibench as mtb

print(mtb.find_methods(category="vertical"))

Dataset folder layout shows how to lay out your own dataset.


Method environments

Each method runs in its own environment. Install only the ones you need. mtb.env.install downloads prebuilt archives, with no conda needed.

import multibench as mtb

# the environments a category needs: installed or not, with their sizes
mtb.env.install(category="vertical", dry_run=True)
# one method
mtb.env.install(["StabMap"])
# one category
mtb.env.install(category="vertical")
Details
  • mtb.config.envs_dir is where they go: $MULTIBENCH_ENVS_DIR when set, else the writable envs directory of the conda or mamba on PATH, else ~/.cache/multibench/envs. print(mtb.config) shows the folder in use.
  • A failed archive download falls back to a build from the lockfile, which needs conda or mamba and takes longer.
  • flavor="cpu" or "gpu" picks the build. flavor="gpu" installs the GPU build from a host without a GPU, such as a cluster login node; by default such a host gets the CPU build.
  • The totals of the dry run: categories share environments, so two categories cost less than the sum of their totals.
  • mtb.env.install skips environments that already exist, so a rerun resumes an interrupted install. With no methods and no category, it installs all 24. Check the total with the dry run first (reference).
Details: method scripts

mtb.env.install downloads the method scripts with the environments, and a run downloads them if they are missing. They go to mtb.config.repo_path, which is ~/.cache/multibench/scMultiBench_scripts/<id> after pip install. The variable MULTIBENCH_SCRIPTS_REF instead fetches a given commit or tag of PYangLab/scMultiBench with git.

On a host without network, run mtb.env.install on a connected machine that sees the same home folder, or copy mtb.config.repo_path from one and set MULTIBENCH_REPO_PATH to the copy.

Google Colab

The tutorials open in Colab from their badge and run on its default CPU runtime. Each notebook installs the environments of the methods it runs and runs them, with no conda needed.

Details

On a GPU runtime, an environment that has a GPU build gets that build: a larger download, and faster training.

Windows

mtb.env.install and mtb.run also work on Windows, without conda, WSL or administrator rights, for these methods (tested on Windows 11):

Windows environment Methods
A CPU build and a GPU build Matilda, sciPENN, Cobolt, Multigrate, MultiVI, totalVI, scMDC, scMoMaT, MultiMAP, uniPort, Portal
A GPU build only: the method needs an NVIDIA GPU, as on Linux moETM, SMILE, iPOLNG, scBridge, sciCAN, UnitedNet
One build, on the CPU StabMap, iNMF, online_iNMF, Conos, Seurat_v5, Seurat_WNN, MOFA2, scMSI, scMVP, VIMCCA, VIPCCA

UINMF, SCALEX, Concerto, scMM, GLUE, scJoint, Seurat_v3 and MIRA have no Windows environment. Asked for on Windows, they are refused: "<method> has no Windows environment. Run it on a Linux machine, such as Colab."

Details
  • Name the methods to install. mtb.env.install(category=...) and the call without methods are refused on Windows, because every category has a method without a Windows environment.
  • Keep the path of mtb.config.envs_dir short. Windows limits a file path to 259 characters, and mtb.env.install refuses a folder that leaves too little room.
  • Nothing is built on Windows: build=True is refused, and a download that fails raises an error.
  • PyTorch needs the Microsoft Visual C++ Redistributable. Install it if a method stops because a DLL is missing.
  • A user name with a space can stop the R methods (StabMap, Seurat, iNMF and Conos). Set TMPDIR to a folder without a space before you start Python.
  • cLISI and iLISI need a C++ compiler on Windows, such as the g++ of Rtools. Without one they are NaN, with a warning (Troubleshooting).

macOS

mtb.env.install and mtb.run also work on a Mac with Apple silicon, without conda, for these methods: Matilda, sciPENN, Cobolt, Multigrate, MultiVI, totalVI, scMDC, scMoMaT, MultiMAP, uniPort, Portal, StabMap, iNMF, online_iNMF, Seurat_v5, Seurat_WNN, UINMF, SCALEX, GLUE, scJoint, Seurat_v3 and MIRA. Each environment has one build, on the CPU.

moETM, SMILE, iPOLNG, scBridge, sciCAN, UnitedNet, Conos, MOFA2, scMSI, scMVP, VIMCCA, VIPCCA, Concerto and scMM have no macOS environment. Asked for on macOS, they are refused: "<method> has no macOS environment. Run it on a Linux machine, such as Colab."

No method runs on an Intel Mac.

Details
  • Name the methods to install. mtb.env.install(category=...) and the call without methods are refused on macOS, because every category has a method without a macOS environment.
  • Nothing is built on macOS: build=True is refused.
  • cLISI and iLISI need a C++ compiler on macOS, such as the one in the Xcode command-line tools. Without one they are NaN, with a warning (Troubleshooting).

Get the data

Each tutorial downloads its dataset on first run. Datasets go to mtb.config.data_path: ~/.cache/multibench/data after pip install, <repo>/data in a developer install.

import multibench as mtb

mtb.catalog.datasets().dataset.tolist()   # ['D11', 'D28', 'D45', 'D46', 'D52', 'D22mini', ...]
mtb.data.fetch("D11")                     # no-op when already present
Details

The tutorials run on one dataset per integration task. D11 is a benchmark dataset; each mini is a smaller copy of one, with a subset of its cells, genes and peaks:

Task Dataset Size
vertical, RNA + ADT D11 11 MB
vertical, RNA + ATAC D27mini_vertical 24 MB
vertical, RNA + ADT + ATAC D22mini 8 MB
diagonal, [RNA, ATAC] D27mini 15 MB
diagonal, [multiple RNA, multiple ATAC] D37mini 12 MB
mosaic, [RNA, RNA + ADT, ADT] D38mini 2 MB
mosaic, [RNA, RNA + ATAC, ATAC] D45mini 11 MB
mosaic, Mixed, with shared modality D46mini 4 MB
mosaic, Mixed, without shared modality D49mini 5 MB
cross, Multiple RNA + ADT D52mini 2 MB
cross, Multiple RNA + ATAC D56mini 10 MB
cross, Multiple ADT + ATAC D58mini 7 MB
cross, Multiple RNA + ADT + ATAC D59mini 8 MB

The benchmark datasets D28 (diagonal, 137 MB), D45 (mosaic, 290 MB), D46 (mosaic, 97 MB) and D52 (cross, 179 MB) are also fetchable.

MULTIBENCH_DATA_PATH sets the data path. Without it, and with XDG_CACHE_HOME set, the data path after pip install is $XDG_CACHE_HOME/multibench/data. print(mtb.config) shows the path in use.

To download a dataset by hand, for example D11:

terminal
DATA=~/.cache/multibench/data   # the data_path that print(mtb.config) shows
URL=https://github.com/DSichang/scMultiBench/releases/download/data-v1
mkdir -p "$DATA"
wget -qO- "$URL/D11.tar.gz" | tar xz -C "$DATA"

The full collection of processed datasets is linked from the scMultiBench README. Unpack a dataset under the data path, or anywhere else and pass its folder path as dataset.


Troubleshooting

OSError: Environment <env> of <method> is not installed

The method's environment is missing. Run the install command the message names, for example mtb.env.install(["<method>"]). mtb.scan() names it per method.

Environment <env> was not installed by this version of multibench

A conda environment has the name of a method's environment, but this version of multibench did not install it, so it is not used: an environment you made yourself, or one that multibench 0.3.9 or older built from a recipe, or 0.3.0 or older unpacked from an archive. On Linux, rename or remove the folder the message names, then run mtb.env.install(["<method>"]). On Windows and macOS, run it as it is: when that folder is in conda's envs folder, 0.7.3 leaves it alone, unpacks into ~/.cache/multibench/envs and prints [env] <folder>/<env> is another environment, so <env> goes to <folder>.

OSError: Methods run only on Linux ...

The message names this computer's system. Methods do not run on an Intel Mac or on a system other than Linux, Windows and macOS. Run the call on a Linux machine, such as Colab. Make the commands there: mtb.scan(..., command=True) writes this computer's paths into them. mtb.env.install raises the same error.

OSError: <method> has no Windows environment ..., or no macOS environment

The method has no environment for this system. Windows and macOS list the methods that run there. Run the others on a Linux machine, such as Colab.

OSError naming envs_dir, with MULTIBENCH_RUN_MODE=prefix

Prefix mode is forced, but the environment is not under envs_dir. Install it (mtb.env.install(["<method>"])), point MULTIBENCH_ENVS_DIR at the folder that holds it, or unset MULTIBENCH_RUN_MODE.

RuntimeError: Conda is not installed on this computer. build=True needs it.

Only a lockfile build needs conda. Leave out build=True.

ImportError: No module named scib from mtb.evaluate

multibench was installed with --no-deps or into another interpreter. Run pip install multibench-sc in the environment you import from.

TypingError: Failed in nopython mode pipeline ..., naming knn_dists

Scoring builds a neighbour graph with scanpy, and umap-learn 0.6.0 fails there with scanpy 1.12.5 on 8,192 cells or more. mtb.evaluate raises the error, and a run ends with the status ran and the error in reason. Install another umap-learn release:

terminal
pip install "umap-learn!=0.6.0"

Then restart Python and call evaluate again, or score a finished run with mtb.load_run(out_dir).rescore().

scanpy ... cannot read the neighbours pynndescent ... found

Scoring builds a neighbour graph with scanpy, and scanpy 1.12.0 and earlier cannot read what pynndescent 0.7 returns on 8,192 cells or more. multibench-sc 0.7.2 installs the pynndescent line before it. Earlier releases can meet the failure, and report it as AttributeError: 'csr_array' object has no attribute 'getnnz'. Install the earlier pynndescent line:

terminal
pip install "pynndescent<0.7"

Then restart Python and call evaluate again, or score a finished run with mtb.load_run(out_dir).rescore().

cLISI / iLISI are NaN, with a warning naming knn_graph.o

Both need scib's compiled LISI helper, which ships as a Linux x86-64 binary. On other platforms evaluate() rebuilds it automatically when a C++ compiler (g++, c++ or clang++) is on PATH.

Without a compiler, install one (Xcode command-line tools on macOS, the g++ of Rtools on Windows) and rerun evaluate in a new Python session, or run the g++ command the warning prints. The other metrics are computed either way.

unsupported input format

mtb.io.export_dataset takes an in-memory AnnData or MuData. Read other formats into an AnnData first.

If the problem remains, open an issue with the output of python -c "import multibench as mtb; print(mtb.__version__)" and pip freeze.


Next: the Quickstart, or run a method with Run a method.