Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions docs/source/developer_guide/add_benchmarks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -501,3 +501,36 @@ Building those components and their interactivity should become increasingly aut
but less standard plots/interactions will need setting up.

For now, please contact us to help with this process.

Framework credit tags
+++++++++++++++++++++

If a benchmark comes from an external benchmarking framework (for example,
MLIP Arena), add a framework credit tag as follows:

1. Add/update the framework entry in ``ml_peg/app/utils/frameworks.yml``.

.. code-block:: yaml

mlip_arena:
label: MLIP Arena
color: "#0f766e"
text_color: "#ecfeff"
url: "https://huggingface.co/spaces/atomind/mlip-arena"
logo: "https://.../logo.svg"

2. Set ``framework_id`` in the benchmark app constructor.

.. code-block:: python3

return SomeBenchmarkApp(
name="SomeBenchmark",
...,
framework_id="mlip_arena",
)

That is all that is required. The benchmark header badge and framework filter
dropdown are populated automatically from this metadata.

The ``logo`` field is optional. It can point to a remote image URL or a local
Dash asset path such as ``assets/frameworks/my_framework_logo.png``.
9 changes: 9 additions & 0 deletions ml_peg/app/base_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

from ml_peg.app.utils.build_components import build_test_layout
from ml_peg.app.utils.load import rebuild_table
from ml_peg.app.utils.utils import normalize_framework_id


class BaseApp(ABC):
Expand All @@ -28,6 +29,9 @@ class BaseApp(ABC):
List of other Dash components to add to app.
docs_url
URL for online documentation. Default is None.
framework_id
Framework identifier used for benchmark attribution tags. Default is
``"ml_peg"``.
"""

def __init__(
Expand All @@ -37,6 +41,7 @@ def __init__(
table_path: Path,
extra_components: list[Component],
docs_url: str | None = None,
framework_id: str = "ml_peg",
):
"""
Initiaise class.
Expand All @@ -53,12 +58,15 @@ def __init__(
List of other Dash components to add to app.
docs_url
URL to online documentation. Default is None.
framework_id
Framework identifier used for benchmark attribution tags.
"""
self.name = name
self.description = description
self.table_path = table_path
self.extra_components = extra_components
self.docs_url = docs_url
self.framework_id = normalize_framework_id(framework_id)
self.table_id = f"{self.name}-table"
self.table = rebuild_table(
self.table_path, id=self.table_id, description=description
Expand All @@ -80,6 +88,7 @@ def build_layout(self) -> Div:
name=self.name,
description=self.description,
docs_url=self.docs_url,
framework_id=self.framework_id,
table=self.table,
thresholds=getattr(self.table, "thresholds", None),
extra_components=self.extra_components,
Expand Down
Loading
Loading