Load and plot results¶
mtb.load_results
reads the stored results, the package's own runs of its methods, into one
long table.
mtb.plot.bubble draws
the paper's bubble table from it.
Summary
Step 1: Load a results table¶
load_results returns a long table: one row per method, dataset and
metric. Every plotting function takes this table.
import multibench as mtb
df = mtb.load_results(
"vertical", # vertical | diagonal | mosaic | cross
dataset="D11", # a list for several; omit for all
)
df.head()
Details
methods= and metrics= narrow the table and accept aliases, such as
"mofa+" for MOFA2. metrics= takes the same values as in
evaluate.
The stored results cover these datasets:
| Category | datasets |
|---|---|
| vertical | D11 |
| diagonal | D28 |
| mosaic | D45 |
| cross | D52 |
The stored results have no kBET scores.
Step 2: Draw the bubble table¶
mtb.plot.bubble draws one row per method, best on top, and one column per
metric. It returns a Matplotlib Figure.
fig = mtb.plot.bubble(
df,
metrics=["ARI", "NMI", "ASW", "cLISI"], # column order within each family
title="Vertical integration, D11",
save="d11_bubble.pdf",
)
Details
The reference lists every option. These are used most often:
methods=[...]keeps only these methods, before ranking.metrics="clustering"draws one family.
Reading the figure¶
Circle size shows the rank within a column. A bigger circle is better.
Fill colour is scaled within each column of this figure. The lightest
fill is the lowest value in that column. Each metric family starts with
an Overall bar, and rows are in the order of mtb.rank.
Details
Blue columns are the clustering family, green columns the batch-correction metrics, and pink columns the classification metrics.
A grey column holds the same value in every row. A figure with one method is grey in every column. A warning names the grey columns.
A dash marks a metric not computed for that method. Its Overall averages the metrics it has.
The chips give each method's language, Py or R. Matilda (this
run) has Matilda's. ? marks a name the package does not know, such
as your own method.
Several datasets¶
mtb.plot.bar ranks the methods within each dataset and draws one bar per
method with its score across the datasets, in the colour of the metrics=
family: "clustering", "batch" or "classification". mtb.rank(category,
long_df=df, metrics="clustering") gives the same scores, in the same order, as a
table.
import pandas as pd
# res_a, res_b: mtb.run on two dataset folders
df = pd.concat([res_a.long, res_b.long])
fig = mtb.plot.bar(df, metrics="clustering", title="Two datasets")
mtb.rank("vertical", long_df=df, metrics="clustering")
Details
The stored results hold one dataset per category: a bar chart of one of them draws one bar per method.
plot.bubble of several datasets averages each method's values per
column. Each row label shows the method's dataset, or how many
datasets it averages (Matilda ยท 2 ds). A warning says so. When the
datasets share no method, plot each dataset on its own.
A method missing from a dataset is left out there: its score is the
mean over the datasets it was scored on. A warning lists these methods,
and methods= keeps the ones scored on every dataset.
Add your own method¶
Score your run with mtb.evaluate and concatenate it with the stored
results.
Are my numbers comparable? says
what to expect.
import pandas as pd
import multibench as mtb
bench = mtb.load_results("vertical", dataset="D11")
# emb: your embedding, cells x dims, in the order of the D11 labels
mine = mtb.evaluate(emb, dataset="D11", category="vertical", method="MyMethod")
both = pd.concat([bench, mine])
metrics = ["ARI", "NMI", "ASW", "cLISI"]
fig = mtb.plot.bubble(both, metrics=metrics,
title="Vertical integration, D11 + MyMethod",
save="d11_vertical.pdf")
ranks = mtb.rank("vertical", long_df=both, metrics=metrics)
print(list(ranks.method).index("MyMethod") + 1) # its row in the figure
Details
Give your run a name the stored results do not use, such as
"Matilda (this run)" for a run of a package method. It reads Matilda's
cell order. Duplicate
(method, dataset, metric) rows raise instead of being averaged.
To put your embedding next to your runs on your own folder, pass that
folder as dataset=, such as dataset="mydata/MYCITE".
The stored methods set the scale of the overall score, so your rows can score above 1 or below 0.
Try it with a PCA of D11¶
To try the recipe without a method of your own, use a PCA of D11's RNA
as emb. Put these lines above bench = ... in d11_panel.py, then run
the script.