Metrics (stored results)¶
Read the stored results and rank the methods of a category. Are my numbers comparable? says what to expect when you add your own runs.
| Name | Summary |
|---|---|
mtb.load_results |
Load the stored results as one long table. |
mtb.rank |
The overall score and row order of mtb.plot.bubble, best first. |
mtb.load_results
¶
load_results(
category: str | None = None,
*,
dataset: str | list[str] | None = None,
methods: str | list[str] | None = None,
metrics=None,
) -> DataFrame
Load the stored results as one long table.
The stored results are the package's own runs of its methods on the
benchmark datasets, scored with mtb.evaluate. The table concatenates
with the long table mtb.evaluate returns.
| PARAMETERS | DESCRIPTION |
|---|---|
category
|
Integration category (
type
|
dataset
|
Dataset id(s), e.g.
type
|
methods
|
Method(s) to keep, in any case or spelling (
type
|
metrics
|
type
|
| RETURNS | DESCRIPTION |
|---|---|
DataFrame
|
One row per score: |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
No stored results for a requested category or dataset; the message names the datasets there are. |
KeyError
|
Unknown method name in |
ValueError
|
Unknown |
| WARNS | DESCRIPTION |
|---|---|
UserWarning
|
|
Examples
>>> import multibench as mtb
>>> long = mtb.load_results("vertical", dataset="D11")
>>> mtb.load_results("diagonal", dataset="D28", metrics="clustering")
See Also
mtb.evaluate : scores your own embedding into this shape. mtb.plot.bubble : draws the table.
mtb.rank
¶
rank(
category: str,
*,
modalities: list[str] | None = None,
methods: list[str] | None = None,
metrics=None,
long_df: DataFrame | None = None,
) -> DataFrame
The overall score and row order of mtb.plot.bubble, best first.
Within each dataset the methods are ranked on each metric; each
per-dataset score is the mean rank over the metrics, min-max scaled so
that the best stored method scores 1 and the worst 0, and
grand_score is the mean over the datasets. Your rows are ranked
against the stored rows, so these keep their scores and order. Without
long_df=, the methods the package runs for the category without
stored results are listed last.
| PARAMETERS | DESCRIPTION |
|---|---|
category
|
Integration category:
type
|
modalities
|
Keep only methods that read all of these modalities, e.g.
type
|
methods
|
Rank only these methods, among themselves;
type
|
metrics
|
type
|
long_df
|
A long table to rank instead of the stored results, such as
type
|
| RETURNS | DESCRIPTION |
|---|---|
DataFrame
|
One row per method, best first: |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
Nothing left to rank, |
KeyError
|
Unknown method name in |
| WARNS | DESCRIPTION |
|---|---|
UserWarning
|
One message: partial coverage, methods left out and why. |
Examples