Skip to content

Load and plot results

mtb.load_results reads the stored results, the package's own runs of its methods, into one long table. mtb.plot.bubble draws the paper's bubble table from it.

Summary

import multibench as mtb

df = mtb.load_results("vertical", dataset="D11")
fig = mtb.plot.bubble(df, metrics=["ARI", "NMI", "ASW", "cLISI"],
                      title="Vertical integration, D11", save="d11.pdf")

Step 1: Load a results table

load_results returns a long table: one row per method, dataset and metric. Every plotting function takes this table.

import multibench as mtb

df = mtb.load_results(
    "vertical",            # vertical | diagonal | mosaic | cross
    dataset="D11",         # a list for several; omit for all
)
df.head()
Details

methods= and metrics= narrow the table and accept aliases, such as "mofa+" for MOFA2. metrics= takes the same values as in evaluate.

The stored results cover these datasets:

Category datasets
vertical D11
diagonal D28
mosaic D45
cross D52

The stored results have no kBET scores.


Step 2: Draw the bubble table

mtb.plot.bubble draws one row per method, best on top, and one column per metric. It returns a Matplotlib Figure.

fig = mtb.plot.bubble(
    df,
    metrics=["ARI", "NMI", "ASW", "cLISI"],   # column order within each family
    title="Vertical integration, D11",
    save="d11_bubble.pdf",
)
Details

The reference lists every option. These are used most often:

  • methods=[...] keeps only these methods, before ranking.
  • metrics="clustering" draws one family.

Reading the figure

Circle size shows the rank within a column. A bigger circle is better. Fill colour is scaled within each column of this figure. The lightest fill is the lowest value in that column. Each metric family starts with an Overall bar, and rows are in the order of mtb.rank.

Details

Blue columns are the clustering family, green columns the batch-correction metrics, and pink columns the classification metrics.

A grey column holds the same value in every row. A figure with one method is grey in every column. A warning names the grey columns.

A dash marks a metric not computed for that method. Its Overall averages the metrics it has.

The chips give each method's language, Py or R. Matilda (this run) has Matilda's. ? marks a name the package does not know, such as your own method.

Several datasets

mtb.plot.bar ranks the methods within each dataset and draws one bar per method with its score across the datasets, in the colour of the metrics= family: "clustering", "batch" or "classification". mtb.rank(category, long_df=df, metrics="clustering") gives the same scores, in the same order, as a table.

import pandas as pd

# res_a, res_b: mtb.run on two dataset folders
df = pd.concat([res_a.long, res_b.long])
fig = mtb.plot.bar(df, metrics="clustering", title="Two datasets")
mtb.rank("vertical", long_df=df, metrics="clustering")
Details

The stored results hold one dataset per category: a bar chart of one of them draws one bar per method.

plot.bubble of several datasets averages each method's values per column. Each row label shows the method's dataset, or how many datasets it averages (Matilda ยท 2 ds). A warning says so. When the datasets share no method, plot each dataset on its own.

A method missing from a dataset is left out there: its score is the mean over the datasets it was scored on. A warning lists these methods, and methods= keeps the ones scored on every dataset.


Add your own method

Score your run with mtb.evaluate and concatenate it with the stored results. Are my numbers comparable? says what to expect.

d11_panel.py
import pandas as pd
import multibench as mtb

bench = mtb.load_results("vertical", dataset="D11")
# emb: your embedding, cells x dims, in the order of the D11 labels
mine = mtb.evaluate(emb, dataset="D11", category="vertical", method="MyMethod")

both = pd.concat([bench, mine])
metrics = ["ARI", "NMI", "ASW", "cLISI"]
fig = mtb.plot.bubble(both, metrics=metrics,
                      title="Vertical integration, D11 + MyMethod",
                      save="d11_vertical.pdf")
ranks = mtb.rank("vertical", long_df=both, metrics=metrics)
print(list(ranks.method).index("MyMethod") + 1)   # its row in the figure
Details

Give your run a name the stored results do not use, such as "Matilda (this run)" for a run of a package method. It reads Matilda's cell order. Duplicate (method, dataset, metric) rows raise instead of being averaged.

To put your embedding next to your runs on your own folder, pass that folder as dataset=, such as dataset="mydata/MYCITE".

The stored methods set the scale of the overall score, so your rows can score above 1 or below 0.

Try it with a PCA of D11

To try the recipe without a method of your own, use a PCA of D11's RNA as emb. Put these lines above bench = ... in d11_panel.py, then run the script.

import scanpy as sc

d = mtb.data.fetch("D11")
rna = mtb.io.read(d / "rna.h5")
sc.pp.normalize_total(rna)
sc.pp.log1p(rna)
sc.pp.highly_variable_genes(rna, n_top_genes=2000)
sc.pp.pca(rna)
emb = rna.obsm["X_pca"]