Skip to content

Run

Run the methods of a category on a dataset folder and get their embeddings. Methods run on Linux, and on Windows and macOS (Apple silicon) for the methods with a Windows or macOS environment. On an Intel Mac, mtb.scan shows the plan and its commands.

Name Summary
mtb.scan Check which methods can run on a dataset, and how run would run them.
mtb.run Run the runnable methods of a category on a dataset folder, and score them.
mtb.load_run Reload a saved run result.
mtb.RunResult The result of mtb.run: its summary table, long table and figure.

mtb.scan

scan(
    dataset: str,
    category: str | None = None,
    *,
    methods: list[str] | None = None,
    modalities: list[str] | str | None = None,
    params: dict | None = None,
    assume_gpu: bool = False,
    verbose: bool = True,
    command: bool = False,
) -> DataFrame

Check which methods can run on a dataset, and how run would run them.

Nothing is executed and no file is written. Call it first on a new dataset: it prints what the folder holds and why each method that cannot run cannot.

PARAMETERS DESCRIPTION
dataset

Dataset folder: its path, or its name under mtb.config.data_path.

type str

category

Integration category to scan; None = the benchmark's category of a dataset it scored (D11 is vertical), else all four.

type str | None default None

methods

Method ids to include, as a list; None = every method.

type list[str] | None default None

modalities

Modality tokens of one combination, e.g. ["rna", "adt"]; None = the combinations whose modalities the folder holds; "all" = every combination.

type list[str] | str | None default None

params

Parameters per method, as run takes them; they are checked and appear in command when it is asked for.

type dict | None default None

assume_gpu

True = skip this host's GPU test, to check a job for a GPU node from a login node.

type bool default False

verbose

Print what the folder holds, the cells per file and each reason.

type bool default True

command

True = add the command column: the line run would execute, with <out_dir> for the output folder.

type bool default False

RETURNS DESCRIPTION
DataFrame

One row per method (per method and category when category is None), runnable methods first: method, modalities (of the variant run would run, +-joined), runnable, reason (why it cannot run), env, caveat (what to know before running it), and command with command=True.

RAISES DESCRIPTION
FileNotFoundError

The dataset folder does not exist; the message lists the folders present.

ValueError

Unknown category or modality token, a method in methods without a variant there, or a benchmark dataset given another category than its own.

KeyError

Unknown method id, or a params key the method does not accept.

Examples

>>> import multibench as mtb
>>> mtb.scan("D11", "vertical")
>>> mtb.scan("data/MYCITE", "vertical", methods=["Matilda"],
...          params={"Matilda": {"epochs": 10}}, command=True).command[0]
See Also

mtb.run : runs the runnable methods.

mtb.run

run(
    dataset: str,
    category: str,
    out_dir,
    *,
    methods=None,
    modalities=None,
    params: dict | None = None,
    batch=None,
    evaluate: bool = True,
    timeout: float | None = None,
    skip_existing: bool = False,
    verbose: bool = True,
) -> "RunResult"

Run the runnable methods of a category on a dataset folder, and score them.

Only the methods mtb.scan marks runnable run. A method's failure is recorded, never raised, and the results are saved under out_dir after each method. methods=["Matilda"] runs one method. mtb.scan shows the plan first, with each command.

PARAMETERS DESCRIPTION
dataset

Dataset folder: its path, or its name under mtb.config.data_path.

type str

category

Integration category: vertical, diagonal, mosaic or cross.

type str

out_dir

Output root, one <out_dir>/<method>_<dataset>/ per method. An embedding.h5 there also holds its cells' names (barcodes).

type path

methods

Method ids to run, as a list; None = every runnable method.

type list[str] | None default None

modalities

Modality tokens of one combination, e.g. ["rna", "adt"]; None = every combination the folder holds; "all" = every combination.

type list[str] | str | None default None

params

Parameters per method, {"Matilda": {"epochs": 3}}; mtb.params_for lists the keys.

type dict | None default None

batch

One batch per cell; a Series or a barcode-indexed CSV is aligned by barcode. None = each cell's label file.

type array - like | Series | path | None default None

evaluate

Score each embedding; False only runs (status ran), and a reused saved record keeps its scores.

type bool default True

timeout

Wall-clock limit per method, in seconds; None = none.

type float | None default None

skip_existing

Reuse an output already in out_dir instead of running the method again, to resume an interrupted run.

type bool default False

verbose

Print [run] ... progress lines.

type bool default True

RETURNS DESCRIPTION
RunResult

Read summary first, embedding(method) for an embedding.

RAISES DESCRIPTION
FileNotFoundError

The dataset folder does not exist; the message lists the folders present.

ValueError

Nothing in the selection can run, an out_dir that holds the result of another dataset or category, or a batch that does not fit the cells; checked before anything runs.

KeyError

Unknown method id, or a params key the method does not accept; checked before anything runs.

Examples

>>> import multibench as mtb
>>> res = mtb.run("D11", "vertical", out_dir="out/")
>>> res = mtb.run("D11", "vertical", out_dir="out/", methods=["Matilda"],
...               params={"Matilda": {"epochs": 10}})

mtb.load_run

load_run(out_dir, *, methods=None) -> 'RunResult'

Reload a saved run result.

PARAMETERS DESCRIPTION
out_dir

A run out_dir: the folder holding batch_result.json.

type path - like

methods

Methods whose records to keep; None = every record.

type list[str] | None default None

RETURNS DESCRIPTION
RunResult

The reloaded result; save() with no argument writes back to out_dir.

RAISES DESCRIPTION
FileNotFoundError

out_dir holds no batch_result.json.

KeyError

A name in methods has no record; the message lists the methods that do.

WARNS DESCRIPTION
UserWarning

The scores were made by multibench 0.3.x; res.rescore() scores them as this version does.

Examples

>>> import multibench as mtb
>>> res = mtb.load_run("out/")
>>> res.summary
>>> mtb.load_run("out/", methods=["Matilda", "scMM"]).plot()
Notes

A copied run folder can be re-scored: each record's output folder is looked for next to batch_result.json when its recorded path is gone, and the dataset folder under the recorded data root, then mtb.config.data_path.

mtb.RunResult

RunResult(records, dataset, category, out_dir=None)

The result of mtb.run: its summary table, long table and figure.

mtb.run and mtb.load_run build it. It keeps one record per method, which rescore and plot read.

PARAMETERS DESCRIPTION
records

One record per method run, as run builds them.

type list[dict]

dataset

Dataset folder name the run used.

type str

category

Integration category of the run.

type str

out_dir

Where the run wrote its outputs; save() defaults to it.

type path | None default None

ATTRIBUTES DESCRIPTION
records

One dict per method, as saved in batch_result.json: method, status (as in summary), out_dir (its output folder), params_used, run_sec, reused (True when skip_existing reused a saved output), metrics, labels_used (the label files in the cell order scored), error or note (the reason of summary) and the run's provenance (multibench_version, scripts_commit, env_flavor, hostname).

type list[dict]

dataset

Dataset folder name.

type str

category

Integration category.

type str

out_dir

The run's output root, or None for an in-memory result.

type path | None

Examples

>>> import multibench as mtb
>>> res = mtb.run("D11", "vertical", out_dir="out/", timeout=3600)
>>> res.summary[["method", "status", "ARI", "NMI"]]
>>> res.summary[res.summary.status != "scored"]   # what did not finish, and why
>>> fig = res.plot(metrics=["ARI", "NMI", "ASW"])
>>> res.rescore().save()                      # nothing re-run
Notes

len(res) is the number of method records.

See Also

mtb.run : produces one.

mtb.load_run : reloads one from save()'s folder.

mtb.plot.bubble : the figure plot draws from long.

summary property

summary: DataFrame

One row per method: status, timing, label matching and every metric.

RETURNS DESCRIPTION
DataFrame

One row per method, sorted by method:

method                  method id
status                  scored, ran, failed, skipped or timeout
ARI, NMI, ASW, ...      one column per metric
run_sec                 seconds of the method run
caveat                  scan's caveat for the method, or ""
reason                  why the method has no scores, or ""

Examples

>>> res = mtb.load_run("out/")
>>> res.summary[["method", "status", "ARI"]]
See Also

mtb.load_run : reads a saved folder; its batch_result.json holds every record, with the label orders tried.

long property

long: DataFrame

The scores as a long table, for plotting.

This is what plot and mtb.plot.bubble consume.

RETURNS DESCRIPTION
DataFrame

Columns metric, value, method, dataset, category and source. When the scores carry scored_with, so does the table. Empty, with the first six columns, when no method produced metrics.

Examples

>>> res = mtb.load_run("out/")
>>> mtb.plot.bubble(res.long, metrics=["ARI", "NMI"])
>>> res.long.pivot_table(index="method", columns="metric", values="value")
Notes

Source of the rows. Each record contributes the unrounded frame run attached (or long.csv via mtb.load_run) when present. Otherwise the record contributes its metrics dict.

See Also

RunResult.plot : draws the bubble figure from this frame.

plot

plot(**kw)

Bubble figure of every method that produced metrics.

Rows are methods, best first. Circle size shows the rank within a column (bigger is better). Fill compares the values in a column: the lightest is the lowest in this figure, not zero.

PARAMETERS DESCRIPTION
**kw

Keyword arguments of mtb.plot.bubble: metrics=, methods=, title= and save=.

default {}

RETURNS DESCRIPTION
Figure

Save it with fig.savefig("out.png").

RAISES DESCRIPTION
ValueError

Nothing was scored (summary says why), or an unknown name in methods=.

Examples

>>> res = mtb.load_run("out/")
>>> fig = res.plot()
>>> fig = res.plot(metrics=["ARI", "NMI", "ASW"], title="D11 vertical")
>>> fig.savefig("D11_vertical.png", dpi=200)
Notes

Keywords. metrics= sets the column order and methods= a subset.

Reading the figure. Size and colour are relative to the methods in this figure, so with few methods a small gap fills the whole colour scale. Check the values in summary.

See Also

mtb.plot.bubble : the underlying function and its full keyword list.

RunResult.long : the frame handed to it.

rescore

rescore(
    *, batch=None, labels=None, metrics=None, verbose: bool = True
) -> "RunResult"

Score the saved outputs again with new labels, batch or metrics.

No method is re-run. Each record's embedding is read back from its out_dir. The Leiden clusters depend on the cell order and the machine, so the same embedding can score a few hundredths differently.

PARAMETERS DESCRIPTION
batch

Batch ids in the order of the dataset's label files, or a barcode-indexed Series or CSV. Every cell needs a batch. None = the batch run was given, else each cell's label file.

type array - like | Series | path | None default None

labels

One cell-type label per cell; a Series or a barcode-indexed CSV is aligned by barcode. None = the dataset's label files, in the order the record was scored with (ranked as run ranks them when it has none).

type array - like | Series | path | None default None

metrics

Metric family ("clustering", "batch", "classification", "all") or metric codes, which replace those scores and keep the others (with new batch or labels, only these are kept); None = every metric the batch structure allows, as in mtb.evaluate.

type str | list[str] | None default None

verbose

Print one line per method, and one before each label-order ranking.

type bool default True

RETURNS DESCRIPTION
RunResult

A new result; this one is untouched. Call .save(out_dir) on it to persist it.

RAISES DESCRIPTION
FileNotFoundError

The output of no method is found; the message gives the first method's error. A method that cannot be scored otherwise gets the status ran, and reason says why.

ValueError

labels or batch holds ids that are not cells of the dataset, or a cell has no batch or no label (also in the label files).

WARNS DESCRIPTION
UserWarning

A Series or barcode-indexed CSV cannot be aligned and is matched by position.

Examples

>>> mtb.load_run("out/").rescore().save()
See Also

mtb.evaluate : the scoring function applied per record. RunResult.save : persist the re-scored result.

embedding

embedding(method: str)

The embedding of one method, as an AnnData.

PARAMETERS DESCRIPTION
method

A method of this result.

type str

RETURNS DESCRIPTION
AnnData

obsm["X_emb"] in the method's cell order; obs_names are the cells' barcodes, each prefixed by its label file (cty2:AAAC...) when some barcode is in two files; obs holds each cell's label, and its batch when the run had more than one batch.

RAISES DESCRIPTION
KeyError

No record for method.

ValueError

The method did not run, or it writes a graph instead of an embedding.

FileNotFoundError

Its output folder is gone.

Examples

>>> res = mtb.run("D11", "vertical", methods=["Matilda"], out_dir="out/")
>>> adata = res.embedding("Matilda")
>>> sc.pp.neighbors(adata, use_rep="X_emb")

save

save(out_dir=None) -> 'Path'

Write this result to disk so it outlives the process.

PARAMETERS DESCRIPTION
out_dir

Target folder, created if missing; None = the result's own out_dir, else the current directory.

type path | None default None

RETURNS DESCRIPTION
Path

The folder written.

RAISES DESCRIPTION
ValueError

The folder holds a saved result of another dataset or category.

Examples

>>> res = mtb.load_run("out/")
>>> res.rescore(metrics="clustering").save("out/clustering_only")
>>> mtb.load_run("out/clustering_only").summary
Notes

Files written.

  • summary.csv - the summary frame.
  • long.csv - the long frame, only when some method produced metrics.
  • batch_result.json - dataset, category and the per-method records.
  • batch_<hash>.csv - a batch vector the records were scored with.

Saving into a folder that has a result. When the folder already holds batch_result.json for the same dataset and category, the records are merged. A method in this result replaces its earlier record, unless its status is skipped and the earlier one's is not; the other earlier records are kept, every file in the list above is rewritten from the merged set, and a line [save] Merged with 1 earlier record in <folder> (StabMap). names them. This result object is not changed. Jobs running at the same time should use one out_dir each; combine their long tables with pandas.concat to plot them together.

See Also

mtb.load_run : reads the folder back.