New leaderboard prototype

.github/workflows/push_spaces.yaml

@@ -1,6 +1,7 @@
 name: Deploy
 
 on:
+  workflow_dispatch:
   push:
     branches:
       - master
@@ -34,6 +35,7 @@ jobs:
         env:
           HF_TOKEN: ${{ secrets.HF_TOKEN }}
         run: |
+          git lfs install
           for i in 1 2 3 4 5; do
             git push -f https://jaywonchung:[email protected]/spaces/ml-energy/leaderboard master:main && break || sleep 5;
           done

.gitignore

@@ -12,7 +12,9 @@ pyrightconfig.json
 # Python
 *.egg-info
 **/__pycache__
+**/.ipynb_checkpoints
 build/
+**.ipynb
 
 # Data files
 *.log

README.md

@@ -15,49 +15,31 @@ tags: ["energy", "leaderboard"]
 [![Deploy](https://github.com/ml-energy/leaderboard/actions/workflows/push_spaces.yaml/badge.svg?branch=web)](https://github.com/ml-energy/leaderboard/actions/workflows/push_spaces.yaml)
 [![Apache-2.0 License](https://custom-icon-badges.herokuapp.com/github/license/ml-energy/leaderboard?logo=law)](/LICENSE)
 
-How much energy do LLMs consume?
+How much energy do GenAI models like LLMs and Diffusion models consume?
 
 This README focuses on explaining how to run the benchmark yourself.
 The actual leaderboard is here: https://ml.energy/leaderboard.
 
-## Colosseum
-
-We instrumented [Hugging Face TGI](https://github.com/huggingface/text-generation-inference) so that it measures and returns GPU energy consumption.
-Then, our [controller](/spitfight/colosseum/controller) server receives user prompts from the [Gradio app](/app.py), selects two models randomly, and streams model responses back with energy consumption.
-
-## Setup for benchmarking
-
-### Model weights
-
-- For models that are directly accessible in Hugging Face Hub, you don't need to do anything.
-- For other models, convert them to Hugging Face format and put them in `/data/leaderboard/weights/lmsys/vicuna-13B`, for example. The last two path components (e.g., `lmsys/vicuna-13B`) are taken as the name of the model.
-
-### Docker container
+## Repository Organization
 
-We have our pre-built Docker image published with the tag `mlenergy/leaderboard:latest` ([Dockerfile](/Dockerfile)).
-
-```console
-$ docker run -it \
-    --name leaderboard0 \
-    --gpus '"device=0"' \
-    -v /path/to/your/data/dir:/data/leaderboard \
-    -v $(pwd):/workspace/leaderboard \
-    mlenergy/leaderboard:latest bash
+```
+ leaderboard/
+├──  benchmark/      # Benchmark scripts & instructions
+├──  data/           # Benchmark results
+├──  deployment/     # Colosseum deployment files
+├──  spitfight/      # Python package for the Colosseum
+├──  app.py          # Leaderboard Gradio app definition
+└──  index.html      # Embeds the leaderboard HuggingFace Space
 ```
 
-The container internally expects weights to be inside `/data/leaderboard/weights` (e.g., `/data/leaderboard/weights/lmsys/vicuna-7B`), and sets the Hugging Face cache directory to `/data/leaderboard/hfcache`.
-If needed, the repository should be mounted to `/workspace/leaderboard` to override the copy of the repository inside the container.
-
-## Running the benchmark
+## Colosseum
 
-We run benchmarks using multiple nodes and GPUs using [Pegasus](https://github.com/jaywonchung/pegasus). Take a look at [`pegasus/`](/pegasus) for details.
+We instrumented [Hugging Face TGI](https://github.com/huggingface/text-generation-inference) so that it measures and returns GPU energy consumption.
+Then, our [controller](/spitfight/colosseum/controller) server receives user prompts from the [Gradio app](/app.py), selects two models randomly, and streams model responses back with energy consumption.
 
-You can still run benchmarks without Pegasus like this:
+## Running the Benchmark
 
-```console
-$ docker exec leaderboard0 python scripts/benchmark.py --model-path /data/leaderboard/weights/lmsys/vicuna-13B --input-file sharegpt/sg_90k_part1_html_cleaned_lang_first_sampled_sorted.json
-$ docker exec leaderboard0 python scripts/benchmark.py --model-path databricks/dolly-v2-12b --input-file sharegpt/sg_90k_part1_html_cleaned_lang_first_sampled_sorted.json
-```
+We open-sourced the entire benchmark with instructions here: [`./benchmark`](./benchmark)
 
 ## Citation
 

_config.yml

@@ -1,6 +1,12 @@
 exclude:
+  - benchmark/
   - deployment/
-  - pegasus/
-  - scripts/
-  - sharegpt/
+  - spitfight/
+  - docs/
   - tests/
+  - .gitignore
+  - app.py
+  - LICENSE
+  - README.md
+  - requirements.txt
+  - setup.py

app.py

@@ -1,5 +1,12 @@
+"""Gradio app for the ML.ENERGY leaderboard.
+
+Everything is in a single file. Search for `gr.Blocks` to find the place
+where UI elements are actually defined.
+"""
+
 from __future__ import annotations
 
+from abc import abstractmethod
 import copy
 import json
 import random
@@ -9,16 +16,13 @@ import itertools
 import contextlib
 import argparse
 import os
-from typing import Literal
+from pathlib import Path
+from typing import Literal, Any
 from dateutil import parser, tz
 
 import numpy as np
 import gradio as gr
 import pandas as pd
-import plotly.io as pio
-import plotly.express as px
-from pandas.api.types import is_numeric_dtype, is_float_dtype
-pio.templates.default = "plotly_white"
 
 from spitfight.colosseum.client import ControllerClient
 
@@ -28,8 +32,499 @@ COLOSSUMM_YOUTUBE_DEMO_EMBED_HTML = '<div style="width: 100%; min-width: 400px;"
 
 
 class TableManager:
+    """Manages the data for the leaderboard tables for tasks."""
+
+    def __init__(self, data_dir: str) -> None:
+        """Load leaderboard data from files in `data_dir`.
+
+        Expected directory structure: `data_dir/gpu_model`.
+        Inside the innermost (GPU) directory, there should be:
+        - `models.json`: JSON file that maps huggingface model IDs to model info.
+              Some models listed in this file may not have benchmark results.
+        - `model_org/model_name/*.json`: JSON files containing the benchmark results.
+        """
+        self.data_dir = Path(data_dir)
+
+    def __str__(self) -> str:
+        return f"{self.__class__}(data_dir={self.data_dir})"
+
+    def _wrap_model_name(self, url: str, model_name: str) -> str:
+        """Wrap the model name in an HTML anchor."""
+        return f'<a style="text-decoration: underline; text-decoration-style: dotted" target="_blank" href="{url}">{model_name}</a>'
+
+    def _unwrap_model_name(self, model_name: str) -> str:
+        """Unwrap the model name from an HTML anchor."""
+        return model_name.split(">")[1].split("<")[0]
+
+    @abstractmethod
+    def get_tab_name(self) -> str:
+        """Return the name of the leaderboard."""
+
+    @abstractmethod
+    def get_intro_text(self) -> tuple[str, str]:
+        """Return the type of the introduction text and the introduction text."""
+
+    @abstractmethod
+    def get_detail_text(self) -> tuple[str, str]:
+        """Return the type of the detail text and the detail text."""
+
+    def get_benchmark_checkboxes(self) -> dict[str, list[str]]:
+        """Return data for the benchmark selection checkboxes."""
+        return {}
+
+    def get_benchmark_sliders(self) -> dict[str, tuple[float, float, float, float]]:
+        """Return data for the benchmark selection sliders.
+
+        Dictionary values are tuples of the form (min, max, step, default).
+        """
+        return {}
+
+    @abstractmethod
+    def get_all_models(self) -> list[str]:
+        """Return all available models."""
+
+    @abstractmethod
+    def set_filter_get_df(self, *filters) -> pd.DataFrame:
+        """Set the current set of filters and return the filtered DataFrame."""
+
+
+class LLMTableManager(TableManager):
+    def __init__(self, data_dir: str, task_name: str) -> None:
+        """Load leaderboard data from files in `data_dir`.
+
+        Under `data_dir`, there should be:
+        - `models.json`: JSON file that maps huggingface model IDs to model info.
+              Some models listed in this file may not have benchmark results.
+        - `schema.yaml`: YAML file containing the schema of the benchmark.
+
+        Then, benchmark data files are nested under `data_dir` according to the schema.
+        One directory hierarchy for each choice in the schema and then two more -- the
+        model's HuggingFace hub organization and the model name.
+        """
+        super().__init__(data_dir)
+
+        self.task_name = task_name
+
+        # Read in the data into a Pandas DataFrame.
+        # Important: The ordering `self.schema` determines the directory structure.
+        self.schema = yaml.safe_load(open(self.data_dir / "schema.yaml"))
+        models: dict[str, dict[str, Any]] = json.load(
+            open(self.data_dir / "models.json")
+        )
+        res_df = pd.DataFrame()
+        for choice in itertools.product(*self.schema.values()):
+            result_dir = self.data_dir / "/".join(choice)
+            with contextlib.suppress(FileNotFoundError):
+                for model_id, model_info in models.items():
+                    for file in (result_dir / model_id).glob("*.json"):
+                        model_df = pd.DataFrame([json.load(open(file))])
+                        # Sanity checks and standardization of schema values.
+                        assert model_df["Model"].iloc[0] == model_id
+                        for key, val in zip(self.schema.keys(), choice):
+                            assert (
+                                str(val).lower() in str(model_df[key].iloc[0]).lower()
+                            )
+                            model_df[key] = val
+                        # Format the model name as an HTML anchor.
+                        model_df["Model"] = self._wrap_model_name(model_info["url"], model_info["nickname"])
+                        model_df["Params"] = model_info["params"]
+                        res_df = pd.concat([res_df, model_df])
+
+        if res_df.empty:
+            raise ValueError(
+                f"No benchmark JSON files were read from {self.data_dir=}."
+            )
+
+        # Order columns
+        columns = res_df.columns.to_list()
+        cols_to_order = ["Model", "Params"]
+        cols_to_order.extend(self.schema.keys())
+        columns = cols_to_order + [col for col in columns if col not in cols_to_order]
+        res_df = res_df[columns]
+
+        # Order rows
+        res_df = res_df.sort_values(by=["Model", *self.schema.keys(), "Energy/req (J)"])
+
+        self.cur_df = self.full_df = res_df.round(2)
+
+        # We need to set the default view separately when `gr.State` is forked.
+        self.set_filter_get_df()
+
+    def get_benchmark_checkboxes(self) -> dict[str, list[str]]:
+        return self.schema
+
+    def get_benchmark_sliders(self) -> dict[str, tuple[float, float, float, float]]:
+        return {"Target Time Per Output Token (TPOT) (s)": (0.0, 0.5, 0.01, 0.2)}
+
+    def get_all_models(self) -> list[str]:
+        return self.full_df["Model"].apply(self._unwrap_model_name).unique().tolist()
+
+    def set_filter_get_df(self, *filters) -> pd.DataFrame:
+        """Set the current set of filters and return the filtered DataFrame.
+
+        Filters can either be completely empty, or be a concatenated list of
+        choices from all checkboxes and all sliders.
+        """
+        # If the filter is empty, we default to the first choice for each checkbox.
+        if not filters:
+            checkboxes = [choices[:1] for choices in self.schema.values()]
+            sliders = [slider[3] for slider in self.get_benchmark_sliders().values()]
+            filters = checkboxes + sliders
+
+        index = np.full(len(self.full_df), True)
+        # Checkboxes
+        for setup, choice in zip(self.schema, filters):
+            index = index & self.full_df[setup].isin(choice)
+        self.cur_df = self.full_df.loc[index]
+
+        # Sliders (We just have TPOT for now.)
+        # For each `Model`, we want to first filter out rows whose `Avg TPOT (s)` is greater than the slider value.
+        # Finally, only just leave the row whose `Energy/req (J)` is the smallest.
+        tpot_slo = filters[-1]
+        self.cur_df = (
+            self.cur_df
+                .groupby("Model")[self.cur_df.columns]
+                .apply(lambda x: x[x["Avg TPOT (s)"] <= tpot_slo], include_groups=True)
+                .sort_values(by="Energy/req (J)")
+                .reset_index(drop=True)
+                .groupby("Model")
+                .head(1)
+        )
+
+        return self.cur_df
+
+    def get_detail_text(self) -> tuple[str, str]:
+        text = """
+            Columns
+            - **Model**: The name of the model.
+            - **GPU**: Name of the GPU model used for benchmarking.
+            - **Params**: Number of parameters in the model.
+            - **TP**: Tensor parallelism degree.
+            - **PP**: Pipeline parallelism degree. (TP * PP is the total number of GPUs used.)
+            - **Energy/req (J)**: Energy consumed per request in Joules.
+            - **Avg TPOT (s)**: Average time per output token in seconds.
+            - **Token tput (toks/s)**: Average number of tokens generated by the engine per second.
+            - **Avg Output Tokens**: Average number of output tokens in the LLM's response.
+            - **Avg BS**: Average batch size of the serving engine over time.
+            - **Max BS**: Maximum batch size configuration of the serving engine.
+
+            For more detailed information, please take a look at the **About** tab.
+            """
+        return "markdown", text
+
+
+class LLMChatTableManager(LLMTableManager):
+    """LLM table manager for chat tasks."""
+
+    def get_tab_name(self) -> str:
+        return "LLM Chat"
+
+    def get_intro_text(self) -> tuple[str, str]:
+        text = """
+            <h2>How much energy do GenAI models consume?</h2>
+
+            <h3>LLM chatbot response generation</h3>
+
+            <p style="font-size: 16px">
+            We used <a href="https://ml.energy/zeus">Zeus</a> to benchmark various instruction-tuned LLMs in terms of how much time and energy they consume for inference.
+            </p>
+
+            <p style="font-size: 16px">
+            An average Time Per Output Token (TPOT) of 0.20 seconds roughly corresponds to a person reading at 240 words per minute and 1.3 tokens per word.
+            </p>
+            """
+        return "html", text
+
+
+class LLMCodeTableManager(LLMTableManager):
+    """LLM table manager for coding tasks."""
+
+    def get_tab_name(self) -> str:
+        return "LLM Code"
+
+    def get_intro_text(self) -> tuple[str, str]:
+        text = """
+            <h2>How much energy do GenAI models consume?</h2>
+
+            <h3>LLM code generation</h3>
+
+            <p style="font-size: 16px">
+            We used <a href="https://ml.energy/zeus">Zeus</a> to benchmark various LLMs specialized for coding in terms of how much time and energy they consume for inference.
+            </p>
+
+            <p style="font-size: 16px">
+            An average Time Per Output Token (TPOT) of 0.20 seconds roughly corresponds to a person reading at 240 words per minute and 1.3 tokens per word.
+            </p>
+            """
+        return "html", text
+
+
+class VLMChatTableManager(LLMTableManager):
+    """VLM table manager for chat tasks."""
+
+    def get_tab_name(self) -> str:
+        return "VLM Visual Chat"
+
+    def get_intro_text(self) -> tuple[str, str]:
+        text = """
+            <h2>How much energy do GenAI models consume?</h2>
+
+            <h3>VLM visual chatbot response generation</h3>
+
+            <p style="font-size: 16px">
+            We used <a href="https://ml.energy/zeus">Zeus</a> to benchmark various Vision Language Models (VLMs) in terms of how much time and energy they consume for inference.
+            </p>
+
+            <p style="font-size: 16px">
+            A Time Per Output Token (TPOT) of 0.2 seconds roughly corresponds to a person reading at 240 words per minute and 1.3 tokens per word.
+            </p>
+            """
+        return "html", text
+
+
+class DiffusionTableManager(TableManager):
+    def __init__(self, data_dir: str, task_name: str) -> None:
+        """Load leaderboard data from files in `data_dir`.
+
+        Under `data_dir`, there should be:
+        - `models.json`: JSON file that maps huggingface model IDs to model info.
+              Some models listed in this file may not have benchmark results.
+        - `schema.yaml`: YAML file containing the schema of the benchmark.
+
+        Then, benchmark data files are nested under `data_dir` according to the schema.
+        One directory hierarchy for each choice in the schema and then two more -- the
+        model's HuggingFace hub organization and the model name.
+        """
+        super().__init__(data_dir)
+
+        self.task_name = task_name
+
+        if "to video" in task_name.lower():
+            self.energy_col = "Energy/video (J)"
+        elif "to image" in task_name.lower():
+            self.energy_col = "Energy/image (J)"
+        else:
+            raise ValueError(f"Unknown task name: {task_name=}")
+
+        # Read in the data into a Pandas DataFrame.
+        # Important: The ordering `self.schema` determines the directory structure.
+        self.schema = yaml.safe_load(open(self.data_dir / "schema.yaml"))
+        models: dict[str, dict[str, Any]] = json.load(
+            open(self.data_dir / "models.json")
+        )
+        res_df = pd.DataFrame()
+        for choice in itertools.product(*self.schema.values()):
+            result_dir = self.data_dir / "/".join(choice)
+            with contextlib.suppress(FileNotFoundError):
+                for model_id, model_info in models.items():
+                    for file in (result_dir / model_id).glob("*.json"):
+                        model_df = pd.DataFrame([json.load(open(file))])
+                        # Sanity checks and standardization of schema values.
+                        assert model_df["Model"].iloc[0] == model_id
+                        for key, val in zip(self.schema.keys(), choice):
+                            assert (
+                                str(val).lower() in str(model_df[key].iloc[0]).lower()
+                            )
+                            model_df[key] = val
+                        # Format the model name as an HTML anchor.
+                        model_df["Model"] = self._wrap_model_name(model_info["url"], model_info["nickname"])
+                        model_df["Total params"] = model_info["total_params"]
+                        model_df["Denoising params"] = model_info["denoising_params"]
+                        model_df["Resolution"] = model_info["resolution"]
+                        res_df = pd.concat([res_df, model_df])
+
+        if res_df.empty:
+            raise ValueError(
+                f"No benchmark JSON files were read from {self.data_dir=}."
+            )
+
+        # Order columns
+        columns = res_df.columns.to_list()
+        cols_to_order = ["Model", "Denoising params", "Total params"]
+        cols_to_order.extend(self.schema.keys())
+        columns = cols_to_order + [col for col in columns if col not in cols_to_order]
+        res_df = res_df[columns]
+
+        # Order rows
+        res_df = res_df.sort_values(by=["Model", *self.schema.keys(), self.energy_col])
+
+        self.cur_df = self.full_df = res_df.round(2)
+
+        # We need to set the default view separately when `gr.State` is forked.
+        self.set_filter_get_df()
+
+    def get_benchmark_checkboxes(self) -> dict[str, list[str]]:
+        return self.schema
+
+    def get_all_models(self) -> list[str]:
+        return self.full_df["Model"].apply(self._unwrap_model_name).unique().tolist()
+
+    def set_filter_get_df(self, *filters) -> pd.DataFrame:
+        """Set the current set of filters and return the filtered DataFrame.
+
+        Filters can either be completely empty, or be a concatenated list of
+        choices from all checkboxes and all sliders.
+        """
+        # If the filter is empty, we default to the first choice for each key.
+        if not filters:
+            checkboxes = [choices[:1] for choices in self.schema.values()]
+            sliders = [slider[3] for slider in self.get_benchmark_sliders().values()]
+            filters = checkboxes + sliders
+
+        index = np.full(len(self.full_df), True)
+        # Checkboxes
+        for setup, choice in zip(self.schema, filters):
+            index = index & self.full_df[setup].isin(choice)
+        self.cur_df = self.full_df.loc[index]
+
+        # Sliders (We just have Batch latency for now.)
+        # For each `Model`, we want to first filter out rows whose `Batch latency (s)` is greater than the slider value.
+        # Finally, only just leave the row whose `Energy/image (J)` or `Energy/video (J)` is the smallest.
+        batch_latency = filters[-1]
+        self.cur_df = (
+            self.cur_df
+                .groupby("Model")[self.cur_df.columns]
+                .apply(
+                    lambda x: x[x["Batch latency (s)"] <= batch_latency],
+                    include_groups=True,
+                )
+                .sort_values(by=self.energy_col)
+                .reset_index(drop=True)
+                .groupby("Model")
+                .head(1)
+        )
+
+        return self.cur_df
+
+
+class DiffusionT2ITableManager(DiffusionTableManager):
+    """Diffusion table manager for text-to-image tasks."""
+
+    def get_tab_name(self) -> str:
+        return "Diffusion Text to image"
+
+    def get_intro_text(self) -> tuple[str, str]:
+        text = """
+            <h2>Diffusion text-to-image generation</h2></br>
+
+            <p style="font-size: 16px">
+            We used <a href="https://ml.energy/zeus">Zeus</a> to benchmark various open source LLMs in terms of how much time and energy they consume for inference.
+            </p>
+
+            <p style="font-size: 16px">
+            The time and energy consumption of Diffusion models are affected by not only the size of the model, but also the number of denoising steps and the resolution of the generated images.
+            </p>
+            """
+        return "html", text
+
+    def get_detail_text(self) -> tuple[str, str]:
+        text = """
+            Columns
+            - **Model**: The name of the model.
+            - **Denoising params**: Number of parameters in the denosing module (e.g., UNet, Transformer).
+            - **Total params**: Total number of parameters in the model, including encoders and decoders.
+            - **GPU**: Name of the GPU model used for benchmarking.
+            - **Energy/image (J)**: Energy consumed per generated image in Joules.
+            - **Batch latency (s)**: Time taken to generate a batch of images in seconds.
+            - **Batch size**: Number of prompts/images in a batch.
+            - **Denoising steps**: Number of denoising steps used for the diffusion model.
+            - **Resolution**: Resolution of the generated image.
+
+            For more detailed information, please take a look at the **About** tab.
+            """
+        return "markdown", text
+
+    def get_benchmark_sliders(self) -> dict[str, tuple[float, float, float, float]]:
+        return {"Batch latency (s)": (0.0, 60.0, 1.0, 10.0)}
+
+
+class DiffusionT2VTableManager(DiffusionTableManager):
+    """Diffusion table manager for text-to-video tasks."""
+
+    def get_tab_name(self) -> str:
+        return "Diffusion Text to video"
+
+    def get_intro_text(self) -> tuple[str, str]:
+        text = """
+            <h2>Diffusion text-to-video generation</h2></br>
+
+            <p style="font-size: 16px">
+            We used <a href="https://ml.energy/zeus">Zeus</a> to benchmark various open source LLMs in terms of how much time and energy they consume for inference.
+            </p>
+
+            <p style="font-size: 16px">
+            The time and energy consumption of Diffusion models are affected by not only the size of the model, but also the number of denoising steps, the resolution of the generated video, and the total number of frames in the video.
+            </p>
+            """
+        return "html", text
+
+    def get_detail_text(self) -> tuple[str, str]:
+        text = """
+            Columns
+            - **Model**: The name of the model.
+            - **Denoising params**: Number of parameters in the denosing module (e.g., UNet, Transformer).
+            - **Total params**: Total number of parameters in the model, including encoders and decoders.
+            - **GPU**: Name of the GPU model used for benchmarking.
+            - **Energy/video (J)**: Energy consumed per generated video in Joules.
+            - **Batch latency (s)**: Time taken to generate a batch of videos in seconds.
+            - **Batch size**: Number of prompts/videos in a batch.
+            - **Denoising steps**: Number of denoising steps used for the diffusion model.
+            - **Frames**: Number of frames in the generated video.
+            - **Resolution**: Resolution of the generated video.
+
+            For more detailed information, please take a look at the **About** tab.
+            """
+        return "markdown", text
+
+    def get_benchmark_sliders(self) -> dict[str, tuple[float, float, float, float]]:
+        return {"Batch latency (s)": (0.0, 60.0, 1.0, 10.0)}
+
+
+class DiffusionI2VTableManager(DiffusionTableManager):
+    """Diffusion table manager for image-to-video tasks."""
+
+    def get_tab_name(self) -> str:
+        return "Diffusion Image to video"
+
+    def get_intro_text(self) -> tuple[str, str]:
+        text = """
+            <h2>Diffusion image-to-video generation</h2></br>
+
+            <p style="font-size: 16px">
+            We used <a href="https://ml.energy/zeus">Zeus</a> to benchmark various open source LLMs in terms of how much time and energy they consume for inference.
+            </p>
+
+            <p style="font-size: 16px">
+            The time and energy consumption of Diffusion models are affected by not only the size of the model, but also the number of denoising steps, the resolution of the generated video, and the total number of frames in the video.
+            </p>
+            """
+        return "html", text
+
+    def get_detail_text(self) -> tuple[str, str]:
+        text = """
+            Columns
+            - **Model**: The name of the model.
+            - **Denoising params**: Number of parameters in the denosing module (e.g., UNet, Transformer).
+            - **Total params**: Total number of parameters in the model, including encoders and decoders.
+            - **GPU**: Name of the GPU model used for benchmarking.
+            - **Energy/video (J)**: Energy consumed per generated video in Joules.
+            - **Batch latency (s)**: Time taken to generate a batch of videos in seconds.
+            - **Batch size**: Number of prompts/videos in a batch.
+            - **Denoising steps**: Number of denoising steps used for the diffusion model.
+            - **Frames**: Number of frames in the generated video.
+            - **Resolution**: Resolution of the generated video.
+
+            For more detailed information, please take a look at the **About** tab.
+            """
+        return "markdown", text
+
+    def get_benchmark_sliders(self) -> dict[str, tuple[float, float, float, float]]:
+        return {"Batch latency (s)": (0.0, 120.0, 1.0, 45.0)}
+
+
+class LegacyTableManager:
     def __init__(self, data_dir: str) -> None:
-        """Load leaderboard data from CSV files in data_dir.
+        """Load the legacy LLM leaderboard data from CSV files in data_dir.
 
         Inside `data_dir`, there should be:
         - `models.json`: a JSON file containing information about each model.
@@ -58,6 +553,7 @@ class TableManager:
                 f'<a style="text-decoration: underline; text-decoration-style: dotted" '
                 f'target="_blank" href="{url}">{nickname}</a>'
             )
+
         df["model"] = df["model"].apply(format_model_link)
 
         # Sort by our 'energy efficiency' score.
@@ -110,63 +606,6 @@ class TableManager:
         """Formats into HTML that prints in Monospace font."""
         return f"<pre style='font-family: monospace'>{text}</pre>"
 
-    def add_column(self, column_name: str, formula: str):
-        """Create and add a new column with the given formula."""
-        # If the user did not provide the name of the new column,
-        # generate a unique name for them.
-        if not column_name:
-            counter = 1
-            while (column_name := f"custom{counter}") in self.full_df.columns:
-                counter += 1
-
-        # If the user did not provide a formula, return an error message.
-        if not formula:
-            return self.cur_df, self._format_msg("Please enter a formula.")
-
-        # If there is an equal sign in the formula, `df.eval` will
-        # return an entire DataFrame with the new column, instead of
-        # just the new column. This is not what we want, so we check
-        # for this case and return an error message.
-        if "=" in formula:
-            return self.cur_df, self._format_msg("Invalid formula: expr cannot contain '='.")
-
-        # The user may want to update an existing column.
-        verb = "Updated" if column_name in self.full_df.columns else "Added"
-
-        # Evaluate the formula and catch any error.
-        try:
-            # Give the users some helper functions that can be used in the formula
-            # like "@sum(response_length)". Also wipe out some global variables.
-            col = self.full_df.eval(
-                formula,
-                local_dict={"sum": sum, "len": len, "max": max, "min": min},
-                global_dict={"global_tbm": None},
-            )
-        except Exception as exc:
-            return self.cur_df, self._format_msg(f"Invalid formula: {exc}")
-
-        # If the result is a numeric scalar, make it a Series.
-        # We may have deleted some models (rows) form the full dataframe when we
-        # called dropna, so we need to query the maximum index instead of taking len.
-        if isinstance(col, (int, float)):
-            col = pd.Series([col] * (self.full_df.index.max() + 1))
-        # We only accept numeric columns.
-        if not is_numeric_dtype(col):
-            return self.cur_df, self._format_msg("Invalid formula: result must be numeric.")
-        # Round if it's floating point.
-        if is_float_dtype(col):
-            col = col.round(2)
-
-        # If the column already exists, update it.
-        if column_name in self.full_df.columns:
-            self.full_df[column_name] = col
-        else:
-            self.full_df.insert(len(self.schema) + 1, column_name, col)
-
-        # If adding a column succeeded, `self.cur_df` should also be updated.
-        self.cur_df = self.full_df.loc[self.cur_index]
-        return self.cur_df, self._format_msg(f"{verb} column '{column_name}'.")
-
     def get_dropdown(self):
         columns = self.full_df.columns.tolist()[1:]
         return [
@@ -196,51 +635,40 @@ class TableManager:
         self.cur_index = index
         return self.cur_df
 
-    def plot_scatter(self, width, height, x, y, z):
-        # The user did not select either x or y.
-        if not x or not y:
-            return None, width, height, self._format_msg("Please select both X and Y.")
-
-        # Width and height may be an empty string. Then we set them to 600.
-        if not width and not height:
-            width, height = "600", "600"
-        elif not width:
-            width = height
-        elif not height:
-            height = width
-        try:
-            width, height = int(width), int(height)
-        except ValueError:
-            return None, width, height, self._format_msg("Width and height should be positive integers.")
-
-        # Strip the <a> tag from model names.
-        text = self.cur_df["model"].apply(lambda x: x.split(">")[1].split("<")[0])
-        # Hide model names since they clutter the plots, and only show them on hover.
-        if z is None or z == "None" or z == "":
-            fig = px.scatter(self.cur_df, x=x, y=y, hover_name=text)
-        else:
-            fig = px.scatter_3d(self.cur_df, x=x, y=y, z=z, hover_name=text)
-        fig.update_traces(marker=dict(size=12, line=dict(width=2, color="DarkSlateGrey")))
-        fig.update_layout(width=width, height=height)
+    def get_intro_text(self) -> str:
+        """Return the leaderboard's introduction text in HTML."""
+        return """
+            <div align="center">
+              <h2 style="color: #23d175">This is the legacy ML.ENERGY LLM leaderboard. This will be removed by the end of the year.</h2>
+            </div>
+
+            <h3>How much energy do modern Large Language Models (LLMs) consume for inference?</h3>
+
+            <p style="font-size: 16px">
+            We used <a href="https://ml.energy/zeus">Zeus</a> to benchmark various open source LLMs in terms of how much time and energy they consume for inference.
+            </p>
+
+            <p style="font-size: 16px">
+            For more detailed information, please take a look at the <b>About</b> tab.
+            Every benchmark is limited in some sense -- Before you interpret the results, please take a look at the <b>Limitations</b> section there, too.
+            </p>
+            """
 
-        return fig, width, height, ""
 
 # The global instance of the TableManager should only be used when
 # initializing components in the Gradio interface. If the global instance
 # is mutated while handling user sessions, the change will be reflected
 # in every user session. Instead, the instance provided by gr.State should
 # be used.
-global_tbm = TableManager("data")
-
-# Fetch the latest update date of the leaderboard repository.
-resp = requests.get("https://api.github.com/repos/ml-energy/leaderboard/commits/master")
-if resp.status_code != 200:
-    current_date = "[Failed to fetch]"
-    print("Failed to fetch the latest release date of the leaderboard repository.")
-    print(resp.json())
-else:
-    current_datetime = parser.parse(resp.json()["commit"]["author"]["date"])
-    current_date = current_datetime.astimezone(tz.gettz("US/Eastern")).strftime("%Y-%m-%d")
+global_ltbm = LegacyTableManager("data/legacy")
+global_tbms = [
+    LLMChatTableManager("data/llm_text_generation/chat", "Chat"),
+    LLMCodeTableManager("data/llm_text_generation/code", "Code"),
+    VLMChatTableManager("data/mllm_text_generation/chat", "Visual chat"),
+    DiffusionT2ITableManager("data/diffusion/text-to-image", "Text to image"),
+    DiffusionT2VTableManager("data/diffusion/text-to-video", "Text to video"),
+    DiffusionI2VTableManager("data/diffusion/image-to-video", "Image to video"),
+]
 
 # Custom JS.
 # XXX: This is a hack to make the model names clickable.
@@ -254,11 +682,14 @@ else:
 dataframe_update_js = f"""
 function format_model_link() {{
     // Iterate over the cells of the first column of the leaderboard table.
-    for (let index = 1; index <= {len(global_tbm.full_df)}; index++) {{
-        // Get the cell.
-        var cell = document.querySelector(
-            `#tab-leaderboard > div > div > div > table > tbody > tr:nth-child(${{index}}) > td:nth-child(1) > div > span`
-        );
+    var table_element = document.querySelectorAll(".tab-leaderboard");
+    for (var table of table_element) {{
+    for (let index = 1; index <= {len(global_ltbm.full_df) + sum(len(tbm.full_df) for tbm in global_tbms)}; index++) {{
+        // Get the cell from `table`.
+        var cell = table.querySelector(`div > div > div > table > tbody > tr:nth-child(${{index}}) > td:nth-child(1) > div > span`);
+        // var cell = document.querySelector(
+        //     `.tab-leaderboard > div > div > div > table > tbody > tr:nth-child(${{index}}) > td:nth-child(1) > div > span`
+        // );
 
         // If nothing was found, it likely means that now the visible table has less rows
         // than the full table. This happens when the user filters the table. In this case,
@@ -282,6 +713,7 @@ function format_model_link() {{
         // Replace the innerHTML of the cell with the interpreted HTML.
         cell.replaceChildren(model_anchor);
     }}
+    }}
 
     // Return all arguments as is.
     return arguments
@@ -365,25 +797,26 @@ table th:first-child {
 }
 """
 
-intro_text = """
-<h2>How much energy do modern Large Language Models (LLMs) consume for inference?</h2>
-
-<p style="font-size: 16px">We used <a href="https://ml.energy/zeus">Zeus</a> to benchmark various open source LLMs in terms of how much time and energy they consume for inference.
-Time and energy are of course not the only things we care about -- so we also benchmarked all of the models on a variety of NLP datasets,
-including the ARC Challenge (reasoning), HellaSwag (common sense), and TruthfulQA (truthfulness).</p>
-
-<p style="font-size: 16px">For more detailed information, please take a look at the <b>About</b> tab.
-Every benchmark is limited in some sense -- Before you interpret the results, please take a look at the <b>Limitations</b> section there, too.</p>
-"""
-
 # The app will not start without a controller address set.
 controller_addr = os.environ.get("COLOSSEUM_CONTROLLER_ADDR")
 if controller_addr is None:
     COLOSSEUM_UP = False
-    COLOSSEUM_DOWN_MESSAGE = "<br/><h2 style='text-align: center'>Disabled Colosseum for local testing.</h2>"
+    COLOSSEUM_DOWN_MESSAGE = "<br/><h2 style='text-align: center'>Local testing mode. Colosseum disabled.</h2>"
     controller_addr = "localhost"
 global_controller_client = ControllerClient(controller_addr=controller_addr, timeout=15)
 
+# Fetch the latest update date of the leaderboard repository.
+resp = requests.get("https://api.github.com/repos/ml-energy/leaderboard/commits/master")
+if resp.status_code != 200:
+    current_date = "[Failed to fetch]"
+    print("Failed to fetch the latest release date of the leaderboard repository.")
+    print(resp.json())
+else:
+    current_datetime = parser.parse(resp.json()["commit"]["author"]["date"])
+    current_date = current_datetime.astimezone(tz.gettz("US/Eastern")).strftime(
+        "%Y-%m-%d"
+    )
+
 # Load the list of models. To reload, the app should be restarted.
 RANDOM_MODEL_NAME = "Random"
 RANDOM_USER_PREFERENCE = "Two random models"
@@ -392,12 +825,19 @@ model_name_to_user_pref = {model: f"One is {model}" for model in global_availabl
 model_name_to_user_pref[RANDOM_MODEL_NAME] = RANDOM_USER_PREFERENCE
 user_pref_to_model_name = {v: k for k, v in model_name_to_user_pref.items()}
 
+
 # Colosseum helper functions.
-def enable_interact():
-    return [gr.update(interactive=True)] * 2
+def enable_interact(num: int):
+    def inner():
+        return [gr.update(interactive=True)] * num
+    return inner
+
+
+def disable_interact(num: int):
+    def inner():
+        return [gr.update(interactive=False)] * num
+    return inner
 
-def disable_interact():
-    return [gr.update(interactive=False)] * 2
 
 def consumed_less_energy_message(energy_a, energy_b):
     """Return a message that indicates that the user chose the model that consumed less energy.
@@ -410,6 +850,7 @@ def consumed_less_energy_message(energy_a, energy_b):
     how_much = f"{1 / factor:.1f}x" if factor <= 0.5 else f"{100 - factor * 100:.1f}%"
     return f"<h2>That response also <span class='green-text'>consumed {how_much} less energy</span> ({energy_a:,.0f} J vs. {energy_b:,.0f} J)!</h2>"
 
+
 def consumed_more_energy_message(energy_a, energy_b):
     """Return a message that indicates that the user chose the model that consumed more energy.
 
@@ -421,14 +862,23 @@ def consumed_more_energy_message(energy_a, energy_b):
     how_much = f"{factor:.1f}x" if factor >= 2.0 else f"{factor * 100 - 100:.1f}%"
     return f"<h2>That response <span class='red-text'>consumed {how_much} more energy</span> ({energy_a:,.0f} J vs. {energy_b:,.0f} J).</h2>"
 
+
 # Colosseum event handlers
 def on_load():
     """Intialize the dataframe, shuffle the model preference dropdown choices."""
-    dataframe = global_tbm.set_filter_get_df()
+    dataframe = global_ltbm.set_filter_get_df()
+    dataframes = [global_tbm.set_filter_get_df() for global_tbm in global_tbms]
     available_models = copy.deepcopy(global_available_models)
     random.shuffle(available_models)
     available_models.insert(0, RANDOM_MODEL_NAME)
-    return dataframe, gr.Dropdown.update(choices=[model_name_to_user_pref[model] for model in available_models])
+    return (
+        dataframe,
+        *dataframes,
+        gr.Dropdown.update(
+            choices=[model_name_to_user_pref[model] for model in available_models]
+        ),
+    )
+
 
 def add_prompt_disable_submit(prompt, history_a, history_b):
     """Add the user's prompt to the two model's history and disable further submission."""
@@ -442,12 +892,17 @@ def add_prompt_disable_submit(prompt, history_a, history_b):
         client,
     ]
 
+
 def generate_responses(client: ControllerClient, user_preference, history_a, history_b):
     """Generate responses for the two models."""
     model_preference = user_pref_to_model_name[user_preference]
     for resp_a, resp_b in itertools.zip_longest(
-        client.prompt(prompt=history_a[-1][0], index=0, model_preference=model_preference),
-        client.prompt(prompt=history_b[-1][0], index=1, model_preference=model_preference),
+        client.prompt(
+            prompt=history_a[-1][0], index=0, model_preference=model_preference
+        ),
+        client.prompt(
+            prompt=history_b[-1][0], index=1, model_preference=model_preference
+        ),
     ):
         if resp_a is not None:
             history_a[-1][1] += resp_a
@@ -455,8 +910,10 @@ def generate_responses(client: ControllerClient, user_preference, history_a, his
             history_b[-1][1] += resp_b
         yield [history_a, history_b]
 
+
 def make_resp_vote_func(victory_index: Literal[0, 1]):
     """Return a function that will be called when the user clicks on response preference vote buttons."""
+
     def resp_vote_func(client: ControllerClient):
         vote_response = client.response_vote(victory_index=victory_index)
         model_name_a, model_name_b = map(lambda n: f"## {n}", vote_response.model_names)
@@ -491,10 +948,13 @@ def make_resp_vote_func(victory_index: Literal[0, 1]):
                 # Keep the reset button disabled
                 gr.Button.update(visible=False, interactive=False),
             ]
+
     return resp_vote_func
 
+
 def make_energy_vote_func(is_worth: bool):
     """Return a function that will be called when the user clicks on energy vote buttons."""
+
     def energy_vote_func(client: ControllerClient, energy_message: str):
         vote_response = client.energy_vote(is_worth=is_worth)
         model_name_a, model_name_b = map(lambda n: f"## {n}", vote_response.model_names)
@@ -508,8 +968,10 @@ def make_energy_vote_func(is_worth: bool):
             # Append to the energy comparison message
             energy_message[:-5] + (" Fair enough.</h2>" if is_worth else " Wasn't worth it.</h2>"),
         ]
+
     return energy_vote_func
 
+
 def play_again():
     available_models = copy.deepcopy(global_available_models)
     random.shuffle(available_models)
@@ -524,11 +986,16 @@ def play_again():
         # Hide energy vote buttons and message
         gr.Button.update(visible=False), gr.Button.update(visible=False), gr.Markdown.update(visible=False),
         # Enable model preference dropdown and shuffle choices
-        gr.Dropdown.update(value=RANDOM_USER_PREFERENCE, choices=[model_name_to_user_pref[model] for model in available_models], interactive=True),
+        gr.Dropdown.update(
+            value=RANDOM_USER_PREFERENCE,
+            choices=[model_name_to_user_pref[model] for model in available_models],
+            interactive=True,
+        ),
         # Disable reset button
         gr.Button.update(interactive=False, visible=False),
     ]
 
+
 focus_prompt_input_js = """
 function() {
     for (let textarea of document.getElementsByTagName("textarea")) {
@@ -541,13 +1008,17 @@ function() {
 """
 
 with gr.Blocks(css=custom_css) as block:
-    tbm = gr.State(global_tbm)  # type: ignore
+    tbm = gr.State(global_ltbm)  # type: ignore
+    local_tbms: list[TableManager] = [gr.State(global_tbm) for global_tbm in global_tbms]  # type: ignore
+
     with gr.Box():
-        gr.HTML("<h1><a href='https://ml.energy' class='text-logo'>ML.ENERGY</a> Leaderboard</h1>")
+        gr.HTML(
+            "<h1><a href='https://ml.energy' class='text-logo'>ML.ENERGY</a> Leaderboard</h1>"
+        )
 
     with gr.Tabs():
         # Tab: Colosseum.
-        with gr.TabItem("Colosseum ⚔️️"):
+        with gr.Tab("Colosseum ⚔️️"):
             if COLOSSEUM_UP:
                 gr.Markdown(open("docs/colosseum_top.md").read())
             else:
@@ -587,32 +1058,64 @@ with gr.Blocks(css=custom_css) as block:
                 resp_vote_btn_list: list[gr.component.Component] = []
                 with gr.Column():
                     with gr.Row():
-                        masked_model_names.append(gr.Markdown(visible=False, elem_classes=["model-name-text"]))
+                        masked_model_names.append(
+                            gr.Markdown(visible=False, elem_classes=["model-name-text"])
+                        )
                     with gr.Row():
-                        chatbots.append(gr.Chatbot(label="Model A", elem_id="chatbot", height=400, elem_classes=None if COLOSSEUM_UP else ["greyed-out"]))
+                        chatbots.append(
+                            gr.Chatbot(
+                                label="Model A",
+                                elem_id="chatbot",
+                                height=400,
+                                elem_classes=None if COLOSSEUM_UP else ["greyed-out"],
+                            )
+                        )
                     with gr.Row():
-                        left_resp_vote_btn = gr.Button(value="👈 Model A is better", interactive=False)
+                        left_resp_vote_btn = gr.Button(
+                            value="👈 Model A is better", interactive=False
+                        )
                         resp_vote_btn_list.append(left_resp_vote_btn)
 
                 with gr.Column():
                     with gr.Row():
-                        masked_model_names.append(gr.Markdown(visible=False, elem_classes=["model-name-text"]))
+                        masked_model_names.append(
+                            gr.Markdown(visible=False, elem_classes=["model-name-text"])
+                        )
                     with gr.Row():
-                        chatbots.append(gr.Chatbot(label="Model B", elem_id="chatbot", height=400, elem_classes=None if COLOSSEUM_UP else ["greyed-out"]))
+                        chatbots.append(
+                            gr.Chatbot(
+                                label="Model B",
+                                elem_id="chatbot",
+                                height=400,
+                                elem_classes=None if COLOSSEUM_UP else ["greyed-out"],
+                            )
+                        )
                     with gr.Row():
-                        right_resp_vote_btn = gr.Button(value="👉 Model B is better", interactive=False)
+                        right_resp_vote_btn = gr.Button(
+                            value="👉 Model B is better", interactive=False
+                        )
                         resp_vote_btn_list.append(right_resp_vote_btn)
 
             with gr.Row():
                 energy_comparison_message = gr.HTML(visible=False)
 
             with gr.Row():
-                worth_energy_vote_btn = gr.Button(value="The better response was worth 👍 the extra energy.", visible=False)
-                notworth_energy_vote_btn = gr.Button(value="Not really worth that much more. 👎", visible=False)
-                energy_vote_btn_list: list[gr.component.Component] = [worth_energy_vote_btn, notworth_energy_vote_btn]
+                worth_energy_vote_btn = gr.Button(
+                    value="The better response was worth 👍 the extra energy.",
+                    visible=False,
+                )
+                notworth_energy_vote_btn = gr.Button(
+                    value="Not really worth that much more. 👎", visible=False
+                )
+                energy_vote_btn_list: list[gr.component.Component] = [
+                    worth_energy_vote_btn,
+                    notworth_energy_vote_btn,
+                ]
 
             with gr.Row():
-                play_again_btn = gr.Button("Play again!", visible=False, elem_classes=["btn-submit"])
+                play_again_btn = gr.Button(
+                    "Play again!", visible=False, elem_classes=["btn-submit"]
+                )
 
             gr.Markdown(open("docs/colosseum_bottom.md").read())
 
@@ -622,11 +1125,11 @@ with gr.Blocks(css=custom_css) as block:
             (prompt_input
                 .submit(add_prompt_disable_submit, [prompt_input, *chatbots], [prompt_input, prompt_submit_btn, model_preference_dropdown, *chatbots, controller_client], queue=False)
                 .then(generate_responses, [controller_client, model_preference_dropdown, *chatbots], [*chatbots], queue=True, show_progress="hidden")
-                .then(enable_interact, None, resp_vote_btn_list, queue=False))
+                .then(enable_interact(2), None, resp_vote_btn_list, queue=False))
             (prompt_submit_btn
                 .click(add_prompt_disable_submit, [prompt_input, *chatbots], [prompt_input, prompt_submit_btn, model_preference_dropdown, *chatbots, controller_client], queue=False)
                 .then(generate_responses, [controller_client, model_preference_dropdown, *chatbots], [*chatbots], queue=True, show_progress="hidden")
-                .then(enable_interact, None, resp_vote_btn_list, queue=False))
+                .then(enable_interact(2), None, resp_vote_btn_list, queue=False))
 
             left_resp_vote_btn.click(
                 make_resp_vote_func(victory_index=0),
@@ -663,128 +1166,100 @@ with gr.Blocks(css=custom_css) as block:
                 )
                 .then(None, _js=focus_prompt_input_js, queue=False))
 
+        # Tab: Leaderboards.
+        dataframes = []
+        for global_tbm, local_tbm in zip(global_tbms, local_tbms):
+            with gr.Tab(global_tbm.get_tab_name()):
+                # Box: Introduction text.
+                with gr.Box():
+                    intro_text_type, intro_text = global_tbm.get_intro_text()
+                    if intro_text_type not in ["markdown", "html"]:
+                        raise ValueError(f"Invalid text type '{intro_text_type}' from {local_tbm}")
+                    if intro_text_type == "markdown":
+                        gr.Markdown(intro_text)
+                    else:
+                        gr.HTML(intro_text)
+
+                # Block: Checkboxes and sliders to select benchmarking parameters.
+                with gr.Row():
+                    checkboxes: list[gr.CheckboxGroup] = []
+                    for key, choices in global_tbm.get_benchmark_checkboxes().items():
+                        # Check the first element by default.
+                        checkboxes.append(gr.CheckboxGroup(choices=choices, value=choices[:1], label=key))
+
+                    sliders: list[gr.Slider] = []
+                    for key, (min_val, max_val, step, default) in global_tbm.get_benchmark_sliders().items():
+                        sliders.append(gr.Slider(minimum=min_val, maximum=max_val, value=default, step=step, label=key))
+
+                # Block: Leaderboard table.
+                with gr.Row():
+                    dataframe = gr.Dataframe(
+                        type="pandas",
+                        elem_classes=["tab-leaderboard"],
+                        interactive=False,
+                    )
+                    dataframes.append(dataframe)
 
-        # Tab: Leaderboard.
-        with gr.Tab("Leaderboard"):
+                    # Make sure the models have clickable links.
+                    dataframe.change(
+                        None, None, None, _js=dataframe_update_js, queue=False
+                    )
+                    # Table automatically updates when users check or uncheck any checkbox or move any slider.
+                    for element in [*checkboxes, *sliders]:
+                        element.change(
+                            global_tbm.__class__.set_filter_get_df,
+                            inputs=[local_tbm, *checkboxes, *sliders],
+                            outputs=dataframe,
+                            queue=False,
+                        )
+
+                # Block: More details about the leaderboard.
+                with gr.Box():
+                    detail_text_type, detail_text = global_tbm.get_detail_text()
+                    if detail_text_type not in ["markdown", "html"]:
+                        raise ValueError(f"Invalid text type '{detail_text_type}' from {local_tbm}")
+                    if detail_text_type == "markdown":
+                        gr.Markdown(detail_text)
+                    else:
+                        gr.HTML(detail_text)
+
+                # Block: Leaderboard date.
+                with gr.Row():
+                    gr.HTML(
+                        f"<h3 style='color: gray'>Last updated: {current_date}</h3>"
+                    )
+
+        # Tab: Legacy leaderboard.
+        with gr.Tab("LLM Leaderboard (legacy)"):
             with gr.Box():
-                gr.HTML(intro_text)
+                gr.HTML(global_ltbm.get_intro_text())
 
             # Block: Checkboxes to select benchmarking parameters.
             with gr.Row():
                 with gr.Box():
                     gr.Markdown("### Benchmark results to show")
                     checkboxes: list[gr.CheckboxGroup] = []
-                    for key, choices in global_tbm.schema.items():
+                    for key, choices in global_ltbm.schema.items():
                         # Specifying `value` makes everything checked by default.
-                        checkboxes.append(gr.CheckboxGroup(choices=choices, value=choices[:1], label=key))
+                        checkboxes.append(
+                            gr.CheckboxGroup(
+                                choices=choices, value=choices[:1], label=key
+                            )
+                        )
 
             # Block: Leaderboard table.
             with gr.Row():
-                dataframe = gr.Dataframe(type="pandas", elem_id="tab-leaderboard", interactive=False)
+                dataframe = gr.Dataframe(
+                    type="pandas", elem_classes=["tab-leaderboard"], interactive=False
+                )
             # Make sure the models have clickable links.
             dataframe.change(None, None, None, _js=dataframe_update_js, queue=False)
             # Table automatically updates when users check or uncheck any checkbox.
             for checkbox in checkboxes:
-                checkbox.change(TableManager.set_filter_get_df, inputs=[tbm, *checkboxes], outputs=dataframe, queue=False)
-
-            # Block: Allow users to add new columns.
-            with gr.Box():
-                gr.Markdown("### Add custom columns to the table")
-                with gr.Row():
-                    with gr.Column(scale=3):
-                        with gr.Row():
-                            colname_input = gr.Textbox(lines=1, label="Custom column name")
-                            formula_input = gr.Textbox(lines=1, label="Formula (@sum, @len, @max, and @min are supported)")
-                    with gr.Column(scale=1):
-                        with gr.Row():
-                            add_col_btn = gr.Button("Add to table (⏎)", elem_classes=["btn-submit"])
-                        with gr.Row():
-                            clear_input_btn = gr.Button("Clear")
-                with gr.Row():
-                    add_col_message = gr.HTML("")
-                gr.Examples(
-                    examples=[
-                        ["power", "energy / latency"],
-                        ["token_per_joule", "response_length / energy"],
-                        ["verbose", "response_length > @sum(response_length) / @len(response_length)"],
-                    ],
-                    inputs=[colname_input, formula_input],
-                )
-                colname_input.submit(
-                    TableManager.add_column,
-                    inputs=[tbm, colname_input, formula_input],
-                    outputs=[dataframe, add_col_message],
-                    queue=False,
-                )
-                formula_input.submit(
-                    TableManager.add_column,
-                    inputs=[tbm, colname_input, formula_input],
-                    outputs=[dataframe, add_col_message],
-                    queue=False,
-                )
-                add_col_btn.click(
-                    TableManager.add_column,
-                    inputs=[tbm, colname_input, formula_input],
-                    outputs=[dataframe, add_col_message],
-                    queue=False,
-                )
-                clear_input_btn.click(
-                    lambda: (None, None, None),
-                    inputs=None,
-                    outputs=[colname_input, formula_input, add_col_message],
-                    queue=False,
-                )
-
-            # Block: Allow users to plot 2D and 3D scatter plots.
-            with gr.Box():
-                gr.Markdown("### Scatter plot (Hover over marker to show model name)")
-                with gr.Row():
-                    with gr.Column(scale=3):
-                        with gr.Row():
-                            # Initialize the dropdown choices with the global TableManager with just the original columns.
-                            axis_dropdowns = global_tbm.get_dropdown()
-                    with gr.Column(scale=1):
-                        with gr.Row():
-                            plot_btn = gr.Button("Plot", elem_classes=["btn-submit"])
-                        with gr.Row():
-                            clear_plot_btn = gr.Button("Clear")
-                with gr.Accordion("Plot size (600 x 600 by default)", open=False):
-                    with gr.Row():
-                        plot_width_input = gr.Textbox("600", lines=1, label="Width (px)")
-                        plot_height_input = gr.Textbox("600", lines=1, label="Height (px)")
-                with gr.Row():
-                    plot = gr.Plot(value=global_tbm.plot_scatter(
-                        plot_width_input.value,
-                        plot_height_input.value,
-                        x=axis_dropdowns[0].value,
-                        y=axis_dropdowns[1].value,
-                        z=axis_dropdowns[2].value,
-                    )[0])  # type: ignore
-                with gr.Row():
-                    plot_message = gr.HTML("")
-                add_col_btn.click(TableManager.update_dropdown, inputs=tbm, outputs=axis_dropdowns, queue=False)  # type: ignore
-                plot_width_input.submit(
-                    TableManager.plot_scatter,
-                    inputs=[tbm, plot_width_input, plot_height_input, *axis_dropdowns],
-                    outputs=[plot, plot_width_input, plot_height_input, plot_message],
-                    queue=False,
-                )
-                plot_height_input.submit(
-                    TableManager.plot_scatter,
-                    inputs=[tbm, plot_width_input, plot_height_input, *axis_dropdowns],
-                    outputs=[plot, plot_width_input, plot_height_input, plot_message],
-                    queue=False,
-                )
-                plot_btn.click(
-                    TableManager.plot_scatter,
-                    inputs=[tbm, plot_width_input, plot_height_input, *axis_dropdowns],
-                    outputs=[plot, plot_width_input, plot_height_input, plot_message],
-                    queue=False,
-                )
-                clear_plot_btn.click(
-                    lambda: (None,) * 7,
-                    None,
-                    outputs=[*axis_dropdowns, plot, plot_width_input, plot_height_input, plot_message],
+                checkbox.change(
+                    LegacyTableManager.set_filter_get_df,
+                    inputs=[tbm, *checkboxes],
+                    outputs=dataframe,
                     queue=False,
                 )
 
@@ -794,8 +1269,7 @@ with gr.Blocks(css=custom_css) as block:
 
         # Tab: About page.
         with gr.Tab("About"):
-            # Read in LEADERBOARD.md
-            gr.Markdown(open("docs/leaderboard.md").read())
+            gr.Markdown(open("docs/about.md").read())
 
     # Citation
     with gr.Accordion("📚  Citation", open=False, elem_id="citation-header"):
@@ -809,13 +1283,21 @@ with gr.Blocks(css=custom_css) as block:
         )
 
     # Load the table on page load.
-    block.load(on_load, outputs=[dataframe, model_preference_dropdown], queue=False)
+    block.load(
+        on_load,
+        outputs=[dataframe, *dataframes, model_preference_dropdown],
+        queue=False,
+    )
 
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
-    parser.add_argument("--share", action="store_true", help="Specify if sharing is enabled")
+    parser.add_argument(
+        "--share", action="store_true", help="Specify if sharing is enabled"
+    )
     parser.add_argument("--concurrency", type=int, default=50)
 
     args = parser.parse_args()
-    block.queue(concurrency_count=args.concurrency, api_open=False).launch(share=args.share, show_error=True)
+    block.queue(concurrency_count=args.concurrency, api_open=False).launch(
+        share=args.share, show_error=True
+    )

benchmark/.gitignore

@@ -0,0 +1 @@
+**/results/

benchmark/README.md

@@ -0,0 +1,14 @@
+# ML.ENERGY Leaderboard Benchmark Suite
+
+```
+ benchmark/
+├──  common/
+├──  diffusion/
+│  └──  text-to-image/
+└──  llm_text_generation/
+   ├──  chat/
+   └──  code/
+```
+
+The `common` directory is for utilities that are common to all benchmarking tasks.
+Other than that, there is one directory for each type of model and subdirectories for more specific tasks.

benchmark/common/download_weights.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+QUEUE_FILE="$1"
+
+for model in $(tail -n +4 $QUEUE_FILE  | awk '{print $2}'); do
+  HF_HOME=/data/leaderboard/hfcache huggingface-cli download $model --revision $(cat models/$model/revision.txt)
+done

benchmark/common/start_nvml_container.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+docker run -dit --gpus all --cap-add SYS_ADMIN --name nvml nvidia/cuda:12.3.1-base-ubuntu22.04 bash

benchmark/diffusion/image-to-video/.dockerignore

@@ -0,0 +1 @@
+README.md

benchmark/diffusion/image-to-video/Dockerfile

@@ -0,0 +1,20 @@
+FROM nvidia/cuda:12.1.0-base-ubuntu22.04
+
+# Basic installs
+ARG DEBIAN_FRONTEND=noninteractive
+ENV TZ='America/Detroit'
+RUN apt-get update -qq \
+    && apt-get -y --no-install-recommends install python3-pip \
+    && apt-get clean all \
+    && rm -r /var/lib/apt/lists/*
+
+# HuggingFace cache dir
+ENV HF_HOME=/root/.cache/huggingface
+
+# Copy over benchmark suite and install dependencies
+ADD . /workspace/image-to-video
+WORKDIR /workspace/image-to-video
+RUN pip install -r requirements.txt
+
+# Benchmark script to run
+ENTRYPOINT ["python3", "scripts/benchmark_one_datapoint.py"]

benchmark/diffusion/image-to-video/README.md

@@ -0,0 +1,51 @@
+# Diffusion model (Image to Video)
+
+This benchmark suite benchmarks diffusion models with the image-to-video task.
+
+## Setup
+
+### Docker images
+
+```sh
+docker build -t mlenergy/leaderboard:diffusion-i2v .
+```
+
+### HuggingFace cache directory
+
+The scripts assume the HuggingFace cache directory will be under `/data/leaderboard/hfcache` on the node that runs this benchmark.
+
+## Benchmarking
+
+### Obtaining one datapoint
+
+The Docker image we've build runs `python scripts/benchmark_one_datapoint.py` as its `ENTRYPOINT`.
+
+```sh
+docker run \
+  --gpus '"device=0"' \
+  --cap-add SYS_ADMIN \
+  -v /data/leaderboard/hfcache:/root/.cache/huggingface 
+  -v $(pwd):/workspace/image-to-video \
+  mlenergy/leaderboard:diffusion-i2v \
+  --result-root results \
+  --batch-size 2 \
+  --power-limit 300 \
+  --save-every 5 \
+  --model ali-vilab/i2vgen-xl \
+  --dataset-path sharegpt4video/sharegpt4video_100.json \
+  --add-text-prompt \
+  --num-frames 16 \
+  --fps 16 \
+  --huggingface-token $HF_TOKEN
+```
+
+### Obtaining all datapoints for a single model
+
+Export your HuggingFace hub token as environment variable `$HF_TOKEN`.
+
+Run `scripts/benchmark_one_model.py`.
+
+### Running the entire suite with Pegasus
+
+You can use [`pegasus`](https://github.com/jaywonchung/pegasus) to run the entire benchmark suite.
+Queue and host files are in [`./pegasus`](./pegasus).

benchmark/diffusion/image-to-video/models/ali-vilab/i2vgen-xl/kwargs.json

@@ -0,0 +1,4 @@
+{
+  "torch_dtype": "torch.float16",
+  "variant": "fp16"
+}

benchmark/diffusion/image-to-video/models/ali-vilab/i2vgen-xl/revision.txt

@@ -0,0 +1 @@
+39e1979ea27be737b0278c06755e321f2b4360d5

benchmark/diffusion/image-to-video/models/stabilityai/stable-video-diffusion-img2vid-xt/kwargs.json

@@ -0,0 +1,4 @@
+{
+  "torch_dtype": "torch.float16",
+  "variant": "fp16"
+}

benchmark/diffusion/image-to-video/models/stabilityai/stable-video-diffusion-img2vid-xt/revision.txt

@@ -0,0 +1 @@
+9e43909513c6714f1bc78bcb44d96e733cd242aa

benchmark/diffusion/image-to-video/models/stabilityai/stable-video-diffusion-img2vid/kwargs.json

@@ -0,0 +1,4 @@
+{
+  "torch_dtype": "torch.float16",
+  "variant": "fp16"
+}

benchmark/diffusion/image-to-video/models/stabilityai/stable-video-diffusion-img2vid/revision.txt

@@ -0,0 +1 @@
+9cf024d5bfa8f56622af86c884f26a52f6676f2e

benchmark/diffusion/image-to-video/pegasus/A100/hosts_1gpu.yaml

@@ -0,0 +1,11 @@
+- hostname:
+    - localhost
+  gpu:
+    - 0
+    - 1
+    - 2
+    - 3
+    - 4
+    - 5
+    - 6
+    - 7

benchmark/diffusion/image-to-video/pegasus/A100/queue_1gpu.yaml

@@ -0,0 +1,6 @@
+- command:
+    - "python scripts/benchmark_one_model.py {{ model }} --result-root results/joule --dataset-path sharegpt4video/sharegpt4video_100.json --gpu-ids {{ gpu }} --batch-sizes 8 4 2 1 --power-limits 400 --num-inference-steps 25"
+  model:
+    - '--model ali-vilab/i2vgen-xl --num-frames 16 --add-text-prompt'
+    - '--model stabilityai/stable-video-diffusion-img2vid --num-frames 14'
+    - '--model stabilityai/stable-video-diffusion-img2vid-xt --num-frames 25'

benchmark/diffusion/image-to-video/pegasus/H100/hosts_1gpu.yaml

@@ -0,0 +1,11 @@
+- hostname:
+    - localhost
+  gpu:
+    - 0
+    - 1
+    - 2
+    - 3
+    - 4
+    - 5
+    - 6
+    - 7

benchmark/diffusion/image-to-video/pegasus/H100/queue_1gpu.yaml

@@ -0,0 +1,6 @@
+- command:
+    - "python scripts/benchmark_one_model.py {{ model }} --result-root results/joule --dataset-path sharegpt4video/sharegpt4video_700.json --gpu-ids {{ gpu }} --batch-sizes 64 32 16 8 4 2 1 --power-limits 700 --num-inference-steps 25"
+  model:
+    - '--model ali-vilab/i2vgen-xl --num-frames 16 --add-text-prompt'
+    - '--model stabilityai/stable-video-diffusion-img2vid --num-frames 14'
+    - '--model stabilityai/stable-video-diffusion-img2vid-xt --num-frames 25'

benchmark/diffusion/image-to-video/requirements.txt

@@ -0,0 +1,7 @@
+torch
+diffusers==0.29.2
+accelerate
+transformers
+pillow
+nvidia-ml-py
+zeus-ml

benchmark/diffusion/image-to-video/scripts/aggregate_leaderboard_data.py

@@ -0,0 +1,38 @@
+import json
+from glob import glob
+from pathlib import Path
+
+import tyro
+
+
+FIELDS = {
+    "model": "Model",
+    "gpu_model": "GPU",
+    "energy_per_video": "Energy/video (J)",
+    "average_batch_latency": "Batch latency (s)",
+    "batch_size": "Batch size",
+    "num_inference_steps": "Denoising steps",
+    "num_frames": "Frames",
+}
+
+def main(results_dir: Path, output_dir: Path) -> None:
+    print(f"{results_dir} -> {output_dir}")
+
+    for model_dir in sorted(glob(f"{results_dir}/*/*")):
+        model_name = "/".join(model_dir.split("/")[-2:])
+        print(f"  {model_name}")
+        (output_dir / model_name).mkdir(parents=True, exist_ok=True)
+        for file in sorted(glob(f"{model_dir}/bs*+results.json")):
+            raw_data = json.load(open(file))
+            raw_data["energy_per_video"] = raw_data["average_batch_energy"] / raw_data["batch_size"]
+
+            data = {}
+            for field1, field2 in FIELDS.items():
+                data[field2] = raw_data.pop(field1)
+
+            filename = f"bs{data['Batch size']}+steps{data['Denoising steps']}+frames{data['Frames']}.json"
+            json.dump(data, open(output_dir / model_name/  filename, "w"), indent=2)
+
+
+if __name__ == "__main__":
+    tyro.cli(main)

benchmark/diffusion/image-to-video/scripts/aggregate_leaderboard_models.py

@@ -0,0 +1,36 @@
+import json
+from glob import glob
+from pathlib import Path
+
+import tyro
+
+def raw_params_to_readable(params: int) -> str:
+    return f"{params/1e9:.1f}B"
+
+def main(results_dir: Path, output_file: Path) -> None:
+    output_file.parent.mkdir(parents=True, exist_ok=True)
+    print(f"{results_dir} -> {output_file}")
+
+    models = {}
+    for model_dir in sorted(glob(f"{results_dir}/*/*")):
+        model_name = "/".join(model_dir.split("/")[-2:])
+        print(f"  {model_name}")
+        result_file_cand = glob(f"{model_dir}/bs1+*+results.json")
+        assert len(result_file_cand) == 1, model_name
+        results_data = json.load(open(result_file_cand[0]))
+        denosing_module_name = "unet" if "unet" in results_data["num_parameters"] else "transformer"
+        model_info = dict(
+            url=f"https://huggingface.co/{model_name}",
+            nickname=model_name.split("/")[-1].replace("-", " ").title(),
+            total_params=raw_params_to_readable(sum(results_data["num_parameters"].values())),
+            denoising_params=raw_params_to_readable(results_data["num_parameters"][denosing_module_name]),
+            resolution="NA",
+        )
+        assert model_name not in models
+        models[model_name] = model_info
+
+    json.dump(models, open(output_file, "w"), indent=2)
+
+
+if __name__ == "__main__":
+    tyro.cli(main)

benchmark/diffusion/image-to-video/scripts/benchmark_one_datapoint.py

@@ -0,0 +1,300 @@
+from __future__ import annotations
+
+import json
+import inspect
+import argparse
+from pprint import pprint
+from pathlib import Path
+from contextlib import suppress
+from dataclasses import dataclass, field, asdict
+from typing import Any
+
+import torch
+import pynvml
+import numpy as np
+from PIL import Image
+from transformers.trainer_utils import set_seed
+from diffusers import ModelMixin, DiffusionPipeline  # type: ignore
+from diffusers.utils import load_image, export_to_gif  # pyright: reportPrivateImportUsage=false
+from zeus.monitor import ZeusMonitor
+
+# Disable torch gradients globally
+torch.set_grad_enabled(False)
+
+
+@dataclass
+class Results:
+    model: str
+    num_parameters: dict[str, int]
+    gpu_model: str
+    num_infernece_steps: int
+    num_frames: int
+    power_limit: int
+    batch_size: int
+    num_prompts: int
+    total_runtime: float = 0.0
+    total_energy: float = 0.0
+    average_batch_latency: float = 0.0
+    average_images_per_second: float = 0.0
+    average_batch_energy: float = 0.0
+    average_power_consumption: float = 0.0
+    peak_memory: float = 0.0
+    results: list[Result] = field(default_factory=list, repr=False)
+
+
+@dataclass
+class ResultIntermediateBatched:
+    prompts: list[str]
+    images: list[Image.Image]
+    batch_latency: float = 0.0
+    batch_energy: float = 0.0
+    frames: np.ndarray | list[list[Image.Image]] = np.empty(0)
+
+
+@dataclass
+class Result:
+    batch_latency: float
+    sample_energy: float
+    prompt: str
+    video_path: str | None
+
+
+def get_pipeline(model_id: str):
+    """Instantiate a Diffusers pipeline from a modes's HuggingFace Hub ID."""
+    # Load args to give to `from_pretrained` from the model's kwargs.json file
+    kwargs = json.load(open(f"models/{model_id}/kwargs.json"))
+    with suppress(KeyError):
+        kwargs["torch_dtype"] = eval(kwargs["torch_dtype"])
+
+    # Add additional args
+    kwargs["safety_checker"] = None
+    kwargs["revision"] = open(f"models/{model_id}/revision.txt").read().strip()
+
+    pipeline = DiffusionPipeline.from_pretrained(model_id, **kwargs).to("cuda:0")
+    print("\nInstantiated pipeline via DiffusionPipeline:\n", pipeline)
+
+    return pipeline
+
+
+def load_text_image_prompts(
+    path: str,
+    batch_size: int,
+    num_batches: int | None = None,
+) -> tuple[int, list[tuple[list[str], list[Image.Image]]]]:
+    """Load the dataset to feed the model and return it as a list of batches of prompts.
+
+    Depending on the batch size, the final batch may not be full. The final batch
+    is dropped in that case. If `num_batches` is not None, only that many batches
+    is returned. If `num_batches` is None, all batches are returned.
+
+    Returns:
+        Total number of prompts and a list of batches of prompts.
+    """
+    dataset = json.load(open(path))
+    assert len(dataset["caption"]) == len(dataset["video_id"])
+
+    if num_batches is not None:
+        if len(dataset["caption"]) < num_batches * batch_size:
+            raise ValueError("Not enough data for the requested number of batches.")
+        dataset["caption"] = dataset["caption"][: num_batches * batch_size]
+        dataset["video_id"] = dataset["video_id"][: num_batches * batch_size]
+
+    image_path = Path(path).parent / "first_frame"
+    dataset["first_frame"] = [
+        load_image(str(image_path / f"{video_id}.jpg")) for video_id in dataset["video_id"]
+    ]
+
+    batched = [
+        (dataset["caption"][i : i + batch_size], dataset["first_frame"][i : i + batch_size])
+        for i in range(0, len(dataset["caption"]), batch_size)
+    ]
+    if len(batched[-1]) < batch_size:
+        batched.pop()
+
+    return len(batched) * batch_size, batched
+
+
+def count_parameters(pipeline) -> dict[str, int]:
+    """Count the number of parameters in the given pipeline."""
+    num_params = {}
+    for name, attr in vars(pipeline).items():
+        if isinstance(attr, ModelMixin):
+            num_params[name] = attr.num_parameters(only_trainable=False, exclude_embeddings=True)
+        elif isinstance(attr, torch.nn.Module):
+            num_params[name] = sum(p.numel() for p in attr.parameters())
+    return num_params
+
+
+def benchmark(args: argparse.Namespace) -> None:
+    if args.model.startswith("models/"):
+        args.model = args.model[len("models/") :]
+    if args.model.endswith("/"):
+        args.model = args.model[:-1]
+
+    set_seed(args.seed)
+
+    results_dir = Path(args.result_root) / args.model
+    results_dir.mkdir(parents=True, exist_ok=True)
+    benchmark_name = str(results_dir / f"bs{args.batch_size}+pl{args.power_limit}")
+    video_dir = results_dir / f"bs{args.batch_size}+pl{args.power_limit}+generated"
+    video_dir.mkdir(exist_ok=True)
+
+    arg_out_filename = f"{benchmark_name}+args.json"
+    with open(arg_out_filename, "w") as f:
+        f.write(json.dumps(vars(args), indent=2))
+    print(args)
+    print("Benchmark args written to", arg_out_filename)
+
+    zeus_monitor = ZeusMonitor()
+
+    pynvml.nvmlInit()
+    handle = pynvml.nvmlDeviceGetHandleByIndex(0)
+    gpu_model = pynvml.nvmlDeviceGetName(handle)
+    pynvml.nvmlDeviceSetPersistenceMode(handle, pynvml.NVML_FEATURE_ENABLED)
+    pynvml.nvmlDeviceSetPowerManagementLimit(handle, args.power_limit * 1000)
+    pynvml.nvmlShutdown()
+
+    num_prompts, batched_prompts = load_text_image_prompts(args.dataset_path, args.batch_size, args.num_batches)
+
+    pipeline = get_pipeline(args.model)
+
+    # Warmup
+    print("Warming up with two batches...")
+    for i in range(2):
+        params: dict[str, Any] = dict(
+            image=batched_prompts[i][1],
+            num_frames=args.num_frames,
+            num_inference_steps=args.num_inference_steps,
+        )
+        if args.add_text_prompt:
+            params["prompt"] = batched_prompts[i][0]
+
+        _ = pipeline(**params)
+
+    rng = torch.manual_seed(args.seed)
+
+    # Some models require a text prompt alongside the image (e.g., I2VGen-XL)
+    # In that case, `prompts` will not be passed to the model.
+    intermediates: list[ResultIntermediateBatched] = [
+        ResultIntermediateBatched(prompts=text, images=image) for text, image in batched_prompts
+    ]
+
+    # Different pipelines use different names for the FPS parameter
+    gen_signature= inspect.signature(pipeline.__call__)
+    fps_param_name_candidates = list(filter(lambda x: "fps" in x, gen_signature.parameters))
+    if not fps_param_name_candidates:
+        raise ValueError("No parameter with 'fps' in its name found in the pipeline's signature.")
+    if len(fps_param_name_candidates) > 1:
+        raise ValueError("Multiple parameters with 'fps' in their name found in the pipeline's signature.")
+    fps_param_name = fps_param_name_candidates[0]
+
+    torch.cuda.reset_peak_memory_stats(device="cuda:0")
+    zeus_monitor.begin_window("benchmark", sync_cuda=False)
+
+    # Build common parameter dict for all batches
+    params: dict[str, Any] = dict(
+        num_frames=args.num_frames,
+        num_inference_steps=args.num_inference_steps,
+        generator=rng,
+    )
+    params[fps_param_name] = args.fps
+    if args.height is not None:
+        params["height"] = args.height
+    if args.width is not None:
+        params["width"] = args.width
+
+    for ind, intermediate in enumerate(intermediates):
+        print(f"Batch {ind + 1}/{len(intermediates)}")
+
+        params["image"] = intermediate.images
+        if args.add_text_prompt:
+            params["prompt"] = intermediate.prompts
+
+        zeus_monitor.begin_window("batch", sync_cuda=False)
+        frames = pipeline(**params).frames
+        batch_measurements = zeus_monitor.end_window("batch", sync_cuda=False)
+
+        intermediate.frames = frames
+        intermediate.batch_latency = batch_measurements.time
+        intermediate.batch_energy = batch_measurements.total_energy
+
+    measurements = zeus_monitor.end_window("benchmark", sync_cuda=False)
+    peak_memory = torch.cuda.max_memory_allocated(device="cuda:0")
+
+    results: list[Result] = []
+    ind = 0
+    for intermediate in intermediates:
+        # Some pipelines just return a giant numpy array for all frames.
+        # In that case, scale frames to uint8 [0, 256] and convert to PIL.Image
+        if isinstance(intermediate.frames, np.ndarray):
+            frames = []
+            for video in intermediate.frames:
+                frames.append(
+                    [Image.fromarray((frame * 255).astype(np.uint8)) for frame in video]
+                )
+            intermediate.frames = frames
+
+        for frames, prompt in zip(intermediate.frames, intermediate.prompts, strict=True):
+            if ind % args.save_every == 0:
+                video_path = str(video_dir / f"{prompt[:200]}.gif")
+                export_to_gif(frames, video_path, fps=args.fps)
+            else:
+                video_path = None
+
+            results.append(
+                Result(
+                    batch_latency=intermediate.batch_latency,
+                    sample_energy=intermediate.batch_energy / len(intermediate.prompts),
+                    prompt=prompt,
+                    video_path=video_path,
+                )
+            )
+            ind += 1
+
+    final_results = Results(
+        model=args.model,
+        num_parameters=count_parameters(pipeline),
+        gpu_model=gpu_model,
+        num_infernece_steps=args.num_inference_steps,
+        num_frames=args.num_frames,
+        power_limit=args.power_limit,
+        batch_size=args.batch_size,
+        num_prompts=num_prompts,
+        total_runtime=measurements.time,
+        total_energy=measurements.total_energy,
+        average_batch_latency=measurements.time / len(batched_prompts),
+        average_images_per_second=num_prompts / measurements.time,
+        average_batch_energy=measurements.total_energy / len(batched_prompts),
+        average_power_consumption=measurements.total_energy / measurements.time,
+        peak_memory=peak_memory,
+        results=results,
+    )
+
+    with open(f"{benchmark_name}+results.json", "w") as f:
+        f.write(json.dumps(asdict(final_results), indent=2))
+    print("Benchmark results written to", f"{benchmark_name}+results.json")
+
+    print("Benchmark results:")
+    pprint(final_results)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", type=str, required=True, help="The model to benchmark.")
+    parser.add_argument("--dataset-path", type=str, required=True, help="Path to the dataset to use.")
+    parser.add_argument("--add-text-prompt", action="store_true", help="Input text prompt alongside image.")
+    parser.add_argument("--result-root", type=str, help="The root directory to save results to.")
+    parser.add_argument("--batch-size", type=int, default=1, help="The size of each batch of prompts.")
+    parser.add_argument("--power-limit", type=int, default=300, help="The power limit to set for the GPU in Watts.")
+    parser.add_argument("--num-inference-steps", type=int, default=50, help="The number of denoising steps.")
+    parser.add_argument("--num-frames", type=int, default=1, help="The number of frames to generate.")
+    parser.add_argument("--fps", type=int, default=16, help="Frames per second for micro-conditioning.")
+    parser.add_argument("--height", type=int, help="Height of the generated video.")
+    parser.add_argument("--width", type=int, help="Width of the generated video.")
+    parser.add_argument("--num-batches", type=int, default=None, help="The number of batches to use from the dataset.")
+    parser.add_argument("--save-every", type=int, default=10, help="Save generations to file every N prompts.")
+    parser.add_argument("--seed", type=int, default=0, help="The seed to use for the RNG.")
+    parser.add_argument("--huggingface-token", type=str, help="The HuggingFace token to use.")
+    args = parser.parse_args()
+
+    benchmark(args)

benchmark/diffusion/image-to-video/scripts/benchmark_one_model.py

@@ -0,0 +1,84 @@
+from __future__ import annotations
+
+import os
+import argparse
+import subprocess
+
+
+def print_and_write(outfile, line: str, flush: bool = False):
+    print(line, end="", flush=flush)
+    outfile.write(line)
+    if flush:
+        outfile.flush()
+
+
+def main(args: argparse.Namespace) -> None:
+    assert len(args.gpu_ids) == 1
+
+    hf_token = os.environ["HF_TOKEN"]
+
+    if args.model.startswith("models/"):
+        outdir = f"{args.result_root}/{args.model[len('models/'):]}"
+    else:
+        outdir = f"{args.result_root}/{args.model}"
+    os.makedirs(outdir, exist_ok=True)
+
+    outfile = open(f"{outdir}/gpus{''.join(args.gpu_ids)}.out.txt", "w")
+
+    print_and_write(outfile, f"Benchmarking {args.model}\n")
+    print_and_write(outfile, f"Batch sizes: {args.batch_sizes}\n")
+    print_and_write(outfile, f"Power limits: {args.power_limits}\n")
+
+    for batch_size in args.batch_sizes:
+        for power_limit in args.power_limits:
+            print_and_write(outfile, f"{batch_size=}, {power_limit=}\n", flush=True)
+            with subprocess.Popen(
+                args=[
+                    "docker", "run",
+                    "--gpus", '"device=' + ','.join(args.gpu_ids) + '"',
+                    "--cap-add", "SYS_ADMIN",
+                    "--name", f"leaderboard-i2v-{''.join(args.gpu_ids)}",
+                    "--rm",
+                    "-v", "/data/leaderboard/hfcache:/root/.cache/huggingface",
+                    "-v", f"{os.getcwd()}:/workspace/image-to-video",
+                    "mlenergy/leaderboard:diffusion-i2v",
+                    "--dataset-path", args.dataset_path,
+                    "--result-root", args.result_root,
+                    "--batch-size", batch_size,
+                    "--num-batches", "10",
+                    "--power-limit", power_limit,
+                    "--model", args.model,
+                    "--huggingface-token", hf_token,
+                    "--num-frames", args.num_frames,
+                    "--num-inference-steps", args.num_inference_steps,
+                ] + (["--add-text-prompt"] if args.add_text_prompt else []),
+                stdout=subprocess.PIPE,
+                stderr=subprocess.STDOUT,
+                text=True,
+            ) as proc:
+                if proc.stdout:
+                    i = 0
+                    for line in proc.stdout:
+                        print_and_write(outfile, line, flush=i % 50 == 0)
+                        i += 1
+
+            # If proc exited with non-zero status, it's probably an OOM.
+            # Move on to the next batch size.
+            if proc.returncode != 0:
+                break
+
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", type=str, help="ID of the model to benchmark")
+    parser.add_argument("--result-root", type=str, help="Root directory to store the results")
+    parser.add_argument("--gpu-ids", type=str, nargs="+", help="GPU IDs to use")
+    parser.add_argument("--batch-sizes", type=str, nargs="+", default=["8", "4", "2", "1"], help="Batch sizes to benchmark")
+    parser.add_argument("--power-limits", type=str, nargs="+", default=["400", "300", "200"], help="Power limits to benchmark")
+    parser.add_argument("--num-frames", type=str, help="Number of frames to generate")
+    parser.add_argument("--num-inference-steps", type=str, help="Number of denoising steps")
+    parser.add_argument("--add-text-prompt", action="store_true", help="Input text prompt alongside image.")
+    parser.add_argument("--dataset-path", type=str, help="Path to the dataset JSON file.")
+    args = parser.parse_args()
+    main(args)

benchmark/diffusion/image-to-video/sharegpt4video/.gitignore

@@ -0,0 +1 @@
+first_frame/

benchmark/diffusion/image-to-video/sharegpt4video/README.md

@@ -0,0 +1,32 @@
+# ShareGPT4Video dataset
+
+For the image-to-video task, we sample 100 video-caption pairs from the ShareGPT4Video datset to feed to the diffusion model to generate videos.
+
+## Filtering the dataset
+
+Download the dataset with captions and video paths.
+
+```sh
+wget https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video/resolve/main/sharegpt4video_40k.jsonl
+```
+
+Sample video-caption pairs.
+You can adjust the `NUM_SAMPLES` variable in the script to change the size of the generated dataset. By default, 100 pairs will be sampled and saved as `sharegpt4video_100.json`.
+
+```sh
+python sample.py
+```
+
+Download and unzip the chunk of videos.
+
+```sh
+wget https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video/resolve/main/zip_folder/panda/panda_videos_1.zip
+unzip panda_videos_1.zip -d panda
+```
+
+Extract the first frame of the video and save under `first_frame/`.
+
+```sh
+pip install opencv-python
+python extract_first_frame.py
+```

benchmark/diffusion/image-to-video/sharegpt4video/extract_first_frame.py

@@ -0,0 +1,21 @@
+import os
+import json
+
+import cv2
+
+DATASET_PATH = "sharegpt4video_700.json"
+
+
+def main() -> None:
+    os.makedirs("first_frame", exist_ok=True)
+
+    for video_id in json.load(open(DATASET_PATH))["video_id"]:
+        cap = cv2.VideoCapture(f"panda/{video_id}.mp4")
+        ret, frame = cap.read()
+        assert ret, f"failed to read first frame of video {video_id}"
+        cv2.imwrite(f"first_frame/{video_id}.jpg", frame)
+        cap.release()
+
+
+if __name__ == "__main__":
+    main()

benchmark/diffusion/image-to-video/sharegpt4video/sample.py

@@ -0,0 +1,29 @@
+import json
+import random
+
+DATASET_PATH = "sharegpt4video_40k.jsonl"
+VIDEO_SHARD_NAME = "panda_videos_1.zip"
+NUM_SAMPLES = 700
+SEED = 1
+
+
+def main() -> None:
+    dataset = [json.loads(line) for line in open(DATASET_PATH) if VIDEO_SHARD_NAME in line]
+    random.seed(SEED)
+    random.shuffle(dataset)
+
+    sampled = dict(caption=[], video_id=[])
+    for sample in dataset[:NUM_SAMPLES]:
+        assert sample["zip_folder"] == VIDEO_SHARD_NAME, f"sample from wrong video shard: {sample}"
+        whole_video_caption = next(
+            (c for c in sample["captions"] if c["idx"] == "-1"), None
+        )
+        assert whole_video_caption is not None, f"whole video caption not found for sample: {sample}"
+        sampled["caption"].append(whole_video_caption["content"])
+        sampled["video_id"].append(sample["video_id"])
+
+    json.dump(sampled, open(f"sharegpt4video_{NUM_SAMPLES}.json", "w"))
+
+
+if __name__ == "__main__":
+    main()

benchmark/diffusion/text-to-image/.dockerignore

@@ -0,0 +1 @@
+README.md

benchmark/diffusion/text-to-image/Dockerfile

@@ -0,0 +1,20 @@
+FROM nvidia/cuda:12.1.0-base-ubuntu22.04
+
+# Basic installs
+ARG DEBIAN_FRONTEND=noninteractive
+ENV TZ='America/Detroit'
+RUN apt-get update -qq \
+    && apt-get -y --no-install-recommends install python3-pip \
+    && apt-get clean all \
+    && rm -r /var/lib/apt/lists/*
+
+# HuggingFace cache dir
+ENV HF_HOME=/root/.cache/huggingface
+
+# Copy over benchmark suite and install dependencies
+ADD . /workspace/text-to-image
+WORKDIR /workspace/text-to-image
+RUN pip install -r requirements.txt
+
+# Benchmark script to run
+ENTRYPOINT ["python3", "scripts/benchmark_one_datapoint.py"]

benchmark/diffusion/text-to-image/README.md

@@ -0,0 +1,48 @@
+# Diffusion model (Text to Image)
+
+This benchmark suite benchmarks diffusion models with the text-to-image task.
+
+## Setup
+
+### Docker images
+
+```sh
+docker build -t mlenergy/leaderboard:diffusion-t2i .
+```
+
+### HuggingFace cache directory
+
+The scripts assume the HuggingFace cache directory will be under `/data/leaderboard/hfcache` on the node that runs this benchmark.
+
+## Benchmarking
+
+### Obtaining one datapoint
+
+The Docker image we've build runs `python scripts/benchmark_one_datapoint.py` as its `ENTRYPOINT`.
+
+```sh
+docker run \
+  --gpus '"device=0"' \
+  --cap-add SYS_ADMIN \
+  -v /data/leaderboard/hfcache:/root/.cache/huggingface 
+  -v $(pwd):/workspace/text-to-image \
+  mlenergy/leaderboard:diffusion-t2i \
+  --result-root results \
+  --batch-size 2 \
+  --power-limit 300 \
+  --image-save-every 5 \
+  --num-inference-steps 25 \
+  --model stabilityai/stable-diffusion-2-1 \
+  --huggingface-token $HF_TOKEN
+```
+
+### Obtaining all datapoints for a single model
+
+Export your HuggingFace hub token as environment variable `$HF_TOKEN`.
+
+Run `scripts/benchmark_one_model.py`.
+
+### Running the entire suite with Pegasus
+
+You can use [`pegasus`](https://github.com/jaywonchung/pegasus) to run the entire benchmark suite.
+Queue and host files are in [`./pegasus`](./pegasus).

benchmark/diffusion/text-to-image/models/SimianLuo/LCM_Dreamshaper_v7/kwargs.json

@@ -0,0 +1,3 @@
+{
+  "torch_dtype": "torch.float16"
+}

benchmark/diffusion/text-to-image/models/SimianLuo/LCM_Dreamshaper_v7/revision.txt

@@ -0,0 +1 @@
+4721097975058205c4edcdece2cc574b7dd7bc04

benchmark/diffusion/text-to-image/models/kandinsky-community/kandinsky-2-2-decoder/kwargs.json

@@ -0,0 +1,3 @@
+{
+  "torch_dtype": "torch.float16"
+}

benchmark/diffusion/text-to-image/models/kandinsky-community/kandinsky-2-2-decoder/revision.txt

@@ -0,0 +1 @@
+main

benchmark/diffusion/text-to-image/models/kandinsky-community/kandinsky-3/kwargs.json

@@ -0,0 +1,4 @@
+{
+  "torch_dtype": "torch.float16",
+  "variant": "fp16"
+}

benchmark/diffusion/text-to-image/models/kandinsky-community/kandinsky-3/revision.txt

@@ -0,0 +1 @@
+bf79e6c219da8a94abb50235fdc4567eb8fb4632

benchmark/diffusion/text-to-image/models/prompthero/openjourney-v4/kwargs.json

@@ -0,0 +1,3 @@
+{
+  "torch_dtype": "torch.float16"
+}

benchmark/diffusion/text-to-image/models/prompthero/openjourney-v4/revision.txt

@@ -0,0 +1 @@
+b195ed2d503f3eb29637050a886d77bd81d35f0e

benchmark/diffusion/text-to-image/models/segmind/SSD-1B/kwargs.json

@@ -0,0 +1,4 @@
+{
+  "torch_dtype": "torch.float16",
+  "variant": "fp16"
+}

benchmark/diffusion/text-to-image/models/segmind/SSD-1B/revision.txt

@@ -0,0 +1 @@
+60987f37e94cd59c36b1cba832b9f97b57395a10

benchmark/diffusion/text-to-image/models/stabilityai/sdxl-turbo/kwargs.json

@@ -0,0 +1,4 @@
+{
+  "torch_dtype": "torch.float16",
+  "variant": "fp16"
+}

benchmark/diffusion/text-to-image/models/stabilityai/sdxl-turbo/revision.txt

@@ -0,0 +1 @@
+f4b0486b498f84668e828044de1d0c8ba486e05b

benchmark/diffusion/text-to-image/models/stabilityai/stable-cascade/kwargs.json

@@ -0,0 +1,4 @@
+{
+  "torch_dtype": "torch.bfloat16",
+  "variant": "bf16"
+}

benchmark/diffusion/text-to-image/models/stabilityai/stable-cascade/revision.txt

@@ -0,0 +1 @@
+main

benchmark/diffusion/text-to-image/models/stabilityai/stable-diffusion-2-1/kwargs.json

@@ -0,0 +1,4 @@
+{
+  "torch_dtype": "torch.float16",
+  "variant": "fp16"
+}