Event Driven Storytelling Eval

---
name: event-driven-storytelling-eval
description: Evaluates an LLM's ability to perform spatial and contextual reasoning for multi-agent planning in 3D scenes. It tests object arrangement, regional context alignment, and scene state tracking to generate plausible character actions and positions. Use when the user wants to benchmark on Event-Driven Storytelling Benchmark, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.19232
  bibtex_key: lim2025eventdrivenstorytelling
  confidence: high
---

# event-driven-storytelling-eval

> Event-Driven Storytelling with Multiple Lifelike Humans in a 3D Scene — Lim et al. (2025) (arXiv:2507.19232, 2025)

## What this evaluates

Evaluates an LLM's ability to perform spatial and contextual reasoning for multi-agent planning in 3D scenes. It tests object arrangement, regional context alignment, and scene state tracking to generate plausible character actions and positions.

## Datasets

- **Event-Driven Storytelling Benchmark** — total 40; splits: test (40)

## Metrics

- `success rate` **(primary)** — range: [0, 1]
  - Proportion of test case runs that pass all three conditions: a plan is generated, the action label is valid and matches expectations, and the character's position falls within the expected area or coordinate.
- `execution rate` — range: [0, 1]
  - Proportion of test case runs that complete without runtime errors, such as the LLM referencing nonexistent objects or generating syntactically invalid responses.

## Input / output format

**Input**: Scene description, test scenario (T), and mock planning history (H) simulating prior agent actions.

**Output**: Next plan τ_g = (a_g, p_g or 	ilde{p}_g), where a_g is an action label, and p_g is a coordinate or 	ilde{p}_g is a semantic area.

## Scoring recipe

```python
def evaluate(generated_plan, expected_plan, available_actions, expected_area, expected_coords):
    if not generated_plan:
        return False
    action = generated_plan.action
    if action not in available_actions or action not in expected_plan.actions:
        return False
    if expected_plan.is_area_level:
        return generated_plan.area in expected_area
    else:
        return generated_plan.coord in expected_coords
```

## Common pitfalls

- Runtime errors from LLM hallucinations or syntax issues automatically mark a run as failed, regardless of reasoning quality.
- Test cases are split into area-level and coordinate-level outputs; using the wrong comparison logic causes false negatives.
- Mock planning history must accurately simulate prior steps; incorrect context breaks state-reasoning evaluations.

## Evidence (verbatim from paper)

> Our benchmark employs two evaluation metrics: success rate and execution rate. The success rate measures the proportion of passed runs among all test case runs. On the other hand, the execution rate measures the proportion of runs that are executed without any runtime errors, which are included in the parenthesis next to the success rate in tables.

## Citation

```bibtex
@misc{lim2025eventdrivenstorytelling,
  title={Event-Driven Storytelling with Multiple Lifelike Humans in a 3D Scene},
  author={Lim et al. (2025)},
  year={2025},
  note={arXiv:2507.19232}
}
```

- arXiv: 2507.19232