Streambench Eval

---
name: streambench-eval
description: Evaluates real-time streaming video understanding and multi-turn dialogue capabilities. It measures semantic correctness, factual accuracy, dialogue coherence, and system latency across diverse video types and query formats. Use when the user wants to benchmark on STREAMBENCH, or asks about evaluating this task. Reports Acc..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.13468
  bibtex_key: xiong2025streamchat
  confidence: high
---

# streambench-eval

> Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge — Haomiao Xiong et al. (2025) (arXiv:2501.13468, 2025)

## What this evaluates

Evaluates real-time streaming video understanding and multi-turn dialogue capabilities. It measures semantic correctness, factual accuracy, dialogue coherence, and system latency across diverse video types and query formats.

## Datasets

- **STREAMBENCH** — total ?; splits: test (-1); repo https://github.com/hmxiong/StreamChat

## Metrics

- `Acc.` **(primary)** — range: [0, 1]
  - Accuracy percentage of the model's response matching the ground truth or expected answer.
- `Sco.` — range: [0, 5]
  - Semantic correctness score assigned by LLaMA-3 on a scale of [0, 5], where higher scores indicate closer alignment with expected answers.
- `Coh.` — range: other
  - Coherence score measured by the fluctuation of Sco. across dialogue turns; smaller fluctuations indicate smoother dialogue.
- `RPD` — range: seconds
  - Request Processing Delay in seconds, calculated as the time from user request submission to the start of response generation.
- `FPS` — range: other
  - Frames per second processed by the streaming system.

## Input / output format

**Input**: Streaming video frames (processed in chunks/groups) concatenated with a user query or multi-turn dialogue history.

**Output**: Natural language text response to the user query.

## Scoring recipe

```python
def compute_metrics(predictions, golds, timestamps):
    # Sco.: LLaMA-3 judge scores response vs expected answer [0,5]
    sco = [llama3_score(p, g) for p, g in zip(predictions, golds)]
    # Acc.: Binary match or percentage
    acc = [1.0 if p == g else 0.0 for p, g in zip(predictions, golds)]
    # Coh.: Standard deviation of Sco. across turns (lower is better)
    coh = [np.std(turn_scores) for turn_scores in predictions]
    # RPD: Time from request to start of generation
    rpd = [ts['start_gen'] - ts['request'] for ts in timestamps]
    return {'Sco.': np.mean(sco), 'Acc.': np.mean(acc), 'Coh.': np.mean(coh), 'RPD': np.mean(rpd)}
```

## Common pitfalls

- Coherence (Coh.) is not a direct quality score but a measure of score fluctuation across turns; lower values indicate better coherence.
- RPD only measures latency until the start of response generation, not the full generation time.
- Semantic scoring (Sco.) relies on LLaMA-3 as a judge, which may introduce bias or differ from human evaluation.

## Evidence (verbatim from paper)

> We evaluate semantic similarity in single conversations using the LLaMA-3 model, which assigns a semantic correctness score (Sco.) ranging from [0,5], where higher scores reflect responses that more closely align with the expected answers. For assessing coherence in multi-turn dialogues, we compute score fluctuations across turns; smaller fluctuations (Coh.) indicate a smoother dialogue experience. Additionally, we measure request processing delay (RPD), defined as the time (in seconds) from user request submission to the start of response generation.

## Citation

```bibtex
@misc{xiong2025streamchat,
  title={Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge},
  author={Haomiao Xiong et al. (2025)},
  year={2025},
  note={arXiv:2501.13468}
}
```

- arXiv: 2501.13468