Run¶
Run the methods of a category on a dataset folder and get their embeddings.
Methods run on Linux, and on Windows and macOS (Apple silicon) for the
methods with a Windows or
macOS environment. On an Intel Mac, mtb.scan
shows the plan and its commands.
| Name | Summary |
|---|---|
mtb.scan |
Check which methods can run on a dataset, and how run would run them. |
mtb.run |
Run the runnable methods of a category on a dataset folder, and score them. |
mtb.load_run |
Reload a saved run result. |
mtb.RunResult |
The result of mtb.run: its summary table, long table and figure. |
mtb.scan
¶
scan(
dataset: str,
category: str | None = None,
*,
methods: list[str] | None = None,
modalities: list[str] | str | None = None,
params: dict | None = None,
assume_gpu: bool = False,
verbose: bool = True,
command: bool = False,
) -> DataFrame
Check which methods can run on a dataset, and how run would run them.
Nothing is executed and no file is written. Call it first on a new dataset: it prints what the folder holds and why each method that cannot run cannot.
| PARAMETERS | DESCRIPTION |
|---|---|
dataset
|
Dataset folder: its path, or its name under
type
|
category
|
Integration category to scan;
type
|
methods
|
Method ids to include, as a list;
type
|
modalities
|
Modality tokens of one combination, e.g.
type
|
params
|
Parameters per method, as
type
|
assume_gpu
|
type
|
verbose
|
Print what the folder holds, the cells per file and each reason.
type
|
command
|
type
|
| RETURNS | DESCRIPTION |
|---|---|
DataFrame
|
One row per method (per method and category when |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
The dataset folder does not exist; the message lists the folders present. |
ValueError
|
Unknown |
KeyError
|
Unknown method id, or a |
Examples
>>> import multibench as mtb
>>> mtb.scan("D11", "vertical")
>>> mtb.scan("data/MYCITE", "vertical", methods=["Matilda"],
... params={"Matilda": {"epochs": 10}}, command=True).command[0]
See Also
mtb.run : runs the runnable methods.
mtb.run
¶
run(
dataset: str,
category: str,
out_dir,
*,
methods=None,
modalities=None,
params: dict | None = None,
batch=None,
evaluate: bool = True,
timeout: float | None = None,
skip_existing: bool = False,
verbose: bool = True,
) -> "RunResult"
Run the runnable methods of a category on a dataset folder, and score them.
Only the methods mtb.scan marks runnable run. A method's failure is
recorded, never raised, and the results are saved under out_dir after
each method.
methods=["Matilda"] runs one method. mtb.scan shows the plan
first, with each command.
| PARAMETERS | DESCRIPTION |
|---|---|
dataset
|
Dataset folder: its path, or its name under
type
|
category
|
Integration category:
type
|
out_dir
|
Output root, one
type
|
methods
|
Method ids to run, as a list;
type
|
modalities
|
Modality tokens of one combination, e.g.
type
|
params
|
Parameters per method,
type
|
batch
|
One batch per cell; a Series or a barcode-indexed CSV is aligned by
barcode.
type
|
evaluate
|
Score each embedding;
type
|
timeout
|
Wall-clock limit per method, in seconds;
type
|
skip_existing
|
Reuse an output already in
type
|
verbose
|
Print
type
|
| RETURNS | DESCRIPTION |
|---|---|
RunResult
|
Read |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
The dataset folder does not exist; the message lists the folders present. |
ValueError
|
Nothing in the selection can run, an |
KeyError
|
Unknown method id, or a |
Examples
mtb.load_run
¶
Reload a saved run result.
| PARAMETERS | DESCRIPTION |
|---|---|
out_dir
|
A
type
|
methods
|
Methods whose records to keep;
type
|
| RETURNS | DESCRIPTION |
|---|---|
RunResult
|
The reloaded result; |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
|
KeyError
|
A name in |
| WARNS | DESCRIPTION |
|---|---|
UserWarning
|
The scores were made by multibench 0.3.x; |
Examples
>>> import multibench as mtb
>>> res = mtb.load_run("out/")
>>> res.summary
>>> mtb.load_run("out/", methods=["Matilda", "scMM"]).plot()
Notes
A copied run folder can be re-scored: each record's output folder
is looked for next to batch_result.json when its recorded path is
gone, and the dataset folder under the recorded data root, then
mtb.config.data_path.
mtb.RunResult
¶
The result of mtb.run: its summary table, long table and figure.
mtb.run and mtb.load_run build it. It keeps one record per
method, which rescore and plot read.
| PARAMETERS | DESCRIPTION |
|---|---|
records
|
One record per method run, as
type
|
dataset
|
Dataset folder name the run used.
type
|
category
|
Integration category of the run.
type
|
out_dir
|
Where the run wrote its outputs;
type
|
| ATTRIBUTES | DESCRIPTION |
|---|---|
records |
One dict per method, as saved in
type
|
dataset |
Dataset folder name.
type
|
category |
Integration category.
type
|
out_dir |
The run's output root, or
type
|
Examples
>>> import multibench as mtb
>>> res = mtb.run("D11", "vertical", out_dir="out/", timeout=3600)
>>> res.summary[["method", "status", "ARI", "NMI"]]
>>> res.summary[res.summary.status != "scored"] # what did not finish, and why
>>> fig = res.plot(metrics=["ARI", "NMI", "ASW"])
>>> res.rescore().save() # nothing re-run
Notes
len(res) is the number of method records.
See Also
mtb.run : produces one.
mtb.load_run : reloads one from save()'s folder.
mtb.plot.bubble : the figure plot draws from long.
summary
property
¶
One row per method: status, timing, label matching and every metric.
| RETURNS | DESCRIPTION |
|---|---|
DataFrame
|
Examples
See Also
mtb.load_run : reads a saved folder; its batch_result.json holds every
record, with the label orders tried.
long
property
¶
The scores as a long table, for plotting.
This is what plot and mtb.plot.bubble consume.
| RETURNS | DESCRIPTION |
|---|---|
DataFrame
|
Columns |
Examples
>>> res = mtb.load_run("out/")
>>> mtb.plot.bubble(res.long, metrics=["ARI", "NMI"])
>>> res.long.pivot_table(index="method", columns="metric", values="value")
Notes
Source of the rows. Each record contributes the unrounded frame
run attached (or long.csv via mtb.load_run) when
present. Otherwise the record contributes its metrics dict.
See Also
RunResult.plot : draws the bubble figure from this frame.
plot
¶
Bubble figure of every method that produced metrics.
Rows are methods, best first. Circle size shows the rank within a column (bigger is better). Fill compares the values in a column: the lightest is the lowest in this figure, not zero.
| PARAMETERS | DESCRIPTION |
|---|---|
**kw
|
Keyword arguments of
default
|
| RETURNS | DESCRIPTION |
|---|---|
Figure
|
Save it with |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
Nothing was scored ( |
Examples
>>> res = mtb.load_run("out/")
>>> fig = res.plot()
>>> fig = res.plot(metrics=["ARI", "NMI", "ASW"], title="D11 vertical")
>>> fig.savefig("D11_vertical.png", dpi=200)
Notes
Keywords. metrics= sets the column order and methods= a
subset.
Reading the figure. Size and colour are relative to the methods
in this figure, so with few methods a small gap fills the whole colour
scale. Check the values in summary.
See Also
mtb.plot.bubble : the underlying function and its full keyword list.
RunResult.long : the frame handed to it.
rescore
¶
Score the saved outputs again with new labels, batch or metrics.
No method is re-run. Each record's embedding is read back from its
out_dir. The Leiden clusters depend on the cell order and the machine,
so the same embedding can score a few hundredths differently.
| PARAMETERS | DESCRIPTION |
|---|---|
batch
|
Batch ids in the order of the dataset's label files, or a
barcode-indexed Series or CSV. Every cell needs a batch.
type
|
labels
|
One cell-type label per cell; a Series or a barcode-indexed CSV is
aligned by barcode.
type
|
metrics
|
Metric family (
type
|
verbose
|
Print one line per method, and one before each label-order ranking.
type
|
| RETURNS | DESCRIPTION |
|---|---|
RunResult
|
A new result; this one is untouched. Call |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
The output of no method is found; the message gives the first
method's error. A method that cannot be scored otherwise gets the
status |
ValueError
|
|
| WARNS | DESCRIPTION |
|---|---|
UserWarning
|
A Series or barcode-indexed CSV cannot be aligned and is matched by position. |
Examples
See Also
mtb.evaluate : the scoring function applied per record. RunResult.save : persist the re-scored result.
embedding
¶
The embedding of one method, as an AnnData.
| PARAMETERS | DESCRIPTION |
|---|---|
method
|
A method of this result.
type
|
| RETURNS | DESCRIPTION |
|---|---|
AnnData
|
|
| RAISES | DESCRIPTION |
|---|---|
KeyError
|
No record for |
ValueError
|
The method did not run, or it writes a graph instead of an embedding. |
FileNotFoundError
|
Its output folder is gone. |
Examples
save
¶
Write this result to disk so it outlives the process.
| PARAMETERS | DESCRIPTION |
|---|---|
out_dir
|
Target folder, created if missing;
type
|
| RETURNS | DESCRIPTION |
|---|---|
Path
|
The folder written. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
The folder holds a saved result of another dataset or category. |
Examples
>>> res = mtb.load_run("out/")
>>> res.rescore(metrics="clustering").save("out/clustering_only")
>>> mtb.load_run("out/clustering_only").summary
Notes
Files written.
summary.csv- thesummaryframe.long.csv- thelongframe, only when some method produced metrics.batch_result.json- dataset, category and the per-method records.batch_<hash>.csv- a batch vector the records were scored with.
Saving into a folder that has a result. When the folder already
holds batch_result.json for the same dataset and category, the
records are merged. A method in this result replaces its earlier
record, unless its status is skipped and the earlier one's is not; the other
earlier records are kept, every file in the list above is rewritten
from the merged set, and a line [save] Merged with 1 earlier record in <folder>
(StabMap). names them. This result object is not changed. Jobs
running at the same time should use one out_dir each; combine
their long tables with pandas.concat to plot them together.
See Also
mtb.load_run : reads the folder back.