Dcg Bench Eval

---
name: dcg-bench-eval
description: Evaluates multimodal large language models' ability to generate executable HTML/JavaScript code for dynamic chart animations from text or video prompts. It probes instruction following, code executability, and fine-grained semantic alignment between generated visualizations and input specifications. Use when the user wants to benchmark on DCG-8K, or asks about evaluating this task. Reports Execution Pass Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.03341
  bibtex_key: li2025opusanimation
  confidence: high
---

# dcg-bench-eval

> OpusAnimation: Code-Based Dynamic Chart Generation — Bozheng Li et al. (2025) (arXiv:2510.03341, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to generate executable HTML/JavaScript code for dynamic chart animations from text or video prompts. It probes instruction following, code executability, and fine-grained semantic alignment between generated visualizations and input specifications.

## Datasets

- **DCG-8K** — total 8000; splits: test (-1)

## Metrics

- `Execution Pass Rate` **(primary)** — range: percent
  - Binary indicator (1 if generated code renders a valid, non-blank animation video, 0 otherwise). Overall rate is the proportion of successful executions across test samples.
- `QA-based Score` — range: [0, 1]
  - Average of binary answers from an MLLM evaluator (Gemini-2.5-Pro) to ~10 targeted QA pairs per modality. Formulas: S_code = (1/N_c) * sum(MLLM_eval(c_g, QA_code_i)) and S_video = (1/N_v) * sum(MLLM_eval(v_g, QA_video_j)). Returns 0 if code fails to render.

## Input / output format

**Input**: A query q containing a data sequence d and a modality-specific description: either a detailed text prompt (t_d), a simple text prompt (t_s), or a reference animation video (v).

**Output**: Complete HTML/JavaScript code snippet c_g that renders a dynamic chart animation video v_g.

## Scoring recipe

```python
def evaluate(code_output, qa_code, qa_video, evaluator):
    # 1. Execution Pass Rate
    try:
        video = render_html_to_video(code_output)
        is_valid = video is not None and not is_blank(video)
    except Exception:
        is_valid = False
    pass_rate = 1.0 if is_valid else 0.0

    # 2. QA-based Scores
    if not is_valid:
        return pass_rate, 0.0, 0.0

    code_score = sum(1.0 if evaluator(code_output, qa) else 0.0 for qa in qa_code) / len(qa_code)
    video_score = sum(1.0 if evaluator(video, qa) else 0.0 for qa in qa_video) / len(qa_video)
    return pass_rate, code_score, video_score
```

## Common pitfalls

- The QA-based scores default to 0 if the generated code fails to render, so syntactically valid but non-executable code is heavily penalized.
- Evaluation relies on a proprietary MLLM (Gemini-2.5-Pro) for semantic scoring, making exact reproduction of QA-based scores difficult without identical evaluator versions and prompts.
- The exact size of the test split (DCG-Bench) is not explicitly stated, requiring careful handling of split boundaries when re-running.

## Evidence (verbatim from paper)

> Execution Pass Rate. We first assess basic executability by verifying whether generated code $c_{g}$ renders a valid, non-blank animation video $v_{g}$. Each instance then receives a binary indicator, and overall pass rate is computed as proportion of successful executions across the test samples. QA-based Scores. To better assess the semantic alignment between generated outputs and the input specifications in dynamic chart generation, we propose a QA-based metric using powerful MLLM Team et al. ([2023]). For each chart, we construct around ten targeted QA pairs based on its reference code or video, focusing on fine-grained aspects such as animation order, element appearance, and timing consistency... Based on these QA sets, we define the evaluation scores $S_{\text{code}}$ and $S_{\text{video}}$ as: $S_{code}(q,c_{g}) = \frac{1}{N_{c}}\sum^{N_{c}}_{i=1}\text{MLLM}_{eval}(c_{g},QA^{(i)}_{code})$ ... Here, $\text{MLLM}_{eval}$ employs Gemini2.5-Pro Team et al. ([2023]) and returns 1 if the artifact meets the corresponding QA requirement, and 0 otherwise.

## Citation

```bibtex
@misc{li2025opusanimation,
  title={OpusAnimation: Code-Based Dynamic Chart Generation},
  author={Bozheng Li et al. (2025)},
  year={2025},
  note={arXiv:2510.03341}
}
```

- arXiv: 2510.03341