Skip to content

Metrics (stored results)

Read the stored results and rank the methods of a category. Are my numbers comparable? says what to expect when you add your own runs.

Name Summary
mtb.load_results Load the stored results as one long table.
mtb.rank The overall score and row order of mtb.plot.bubble, best first.

mtb.load_results

load_results(
    category: str | None = None,
    *,
    dataset: str | list[str] | None = None,
    methods: str | list[str] | None = None,
    metrics=None,
) -> DataFrame

Load the stored results as one long table.

The stored results are the package's own runs of its methods on the benchmark datasets, scored with mtb.evaluate. The table concatenates with the long table mtb.evaluate returns.

PARAMETERS DESCRIPTION
category

Integration category (vertical, diagonal, mosaic or cross); None = every category.

type str default None

dataset

Dataset id(s), e.g. "D11" or ["D11", "D28"]; None = every dataset of the category. mtb.catalog.datasets() lists them.

type str or list of str default None

methods

Method(s) to keep, in any case or spelling ("mofa+" -> MOFA2); None = every method.

type str or list of str default None

metrics

None / "all" = every metric; or "clustering", "batch", "classification", or a list of codes such as ["ARI", "NMI"].

type None, str or list of str default None

RETURNS DESCRIPTION
DataFrame

One row per score: metric, value, method, dataset, category, source, scored_with. attrs["rerun_version"] holds the package version that scored the stored rows.

RAISES DESCRIPTION
FileNotFoundError

No stored results for a requested category or dataset; the message names the datasets there are.

KeyError

Unknown method name in methods.

ValueError

Unknown category or metrics, or metrics names a family or code the stored results have no rows of, for example metrics="classification" on vertical.

WARNS DESCRIPTION
UserWarning

methods leaves no rows.

Examples

>>> import multibench as mtb
>>> long = mtb.load_results("vertical", dataset="D11")
>>> mtb.load_results("diagonal", dataset="D28", metrics="clustering")
See Also

mtb.evaluate : scores your own embedding into this shape. mtb.plot.bubble : draws the table.

mtb.rank

rank(
    category: str,
    *,
    modalities: list[str] | None = None,
    methods: list[str] | None = None,
    metrics=None,
    long_df: DataFrame | None = None,
) -> DataFrame

The overall score and row order of mtb.plot.bubble, best first.

Within each dataset the methods are ranked on each metric; each per-dataset score is the mean rank over the metrics, min-max scaled so that the best stored method scores 1 and the worst 0, and grand_score is the mean over the datasets. Your rows are ranked against the stored rows, so these keep their scores and order. Without long_df=, the methods the package runs for the category without stored results are listed last.

PARAMETERS DESCRIPTION
category

Integration category: vertical, diagonal, mosaic or cross.

type str

modalities

Keep only methods that read all of these modalities, e.g. ["rna", "adt"], as in mtb.find_methods; None = no filter.

type list of str default None

methods

Rank only these methods, among themselves; None = every method.

type list of str default None

metrics

None = every metric present, as mtb.plot.bubble; or a family, a code, or a list of codes and families.

type None, str or list of str default None

long_df

A long table to rank instead of the stored results, such as pd.concat([mtb.load_results(category), mine]).

type DataFrame default None

RETURNS DESCRIPTION
DataFrame

One row per method, best first: method, grand_score, needs_labels, runtime_tier, worst_sec and env.

RAISES DESCRIPTION
ValueError

Nothing left to rank, metrics naming a family or code the table has no rows of, long_df repeating stored method/dataset rows, your rows on another dataset than the stored rows, or modalities naming data that none of the ranked datasets measured (pass such scores as long_df=).

KeyError

Unknown method name in methods.

WARNS DESCRIPTION
UserWarning

One message: partial coverage, methods left out and why.

Examples

>>> import multibench as mtb
>>> mtb.rank("vertical", modalities=["rna", "adt"])
>>> free = mtb.find_methods("vertical", needs_labels=False)
>>> mtb.rank("vertical", methods=free)
>>> both = pd.concat([mtb.load_results("vertical"), mine])
>>> mtb.rank("vertical", long_df=both)