Audiocrag Eval

---
name: audiocrag-eval
description: Evaluates speech-in-speech-out dialogue systems on their ability to accurately answer spoken queries using external tools, measuring both answer correctness and system latency under streaming versus open-book settings. Use when the user wants to benchmark on AudioCRAG, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.02044
  bibtex_key: arora2025streamrag
  confidence: high
---

# audiocrag-eval

> Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage — Arora et al. (2025) (arXiv:2510.02044, 2025)

## What this evaluates

Evaluates speech-in-speech-out dialogue systems on their ability to accurately answer spoken queries using external tools, measuring both answer correctness and system latency under streaming versus open-book settings.

## Datasets

- **AudioCRAG** — total ?; splits: synthetic (-1), human (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered queries out of the total number of queries in the benchmark.
- `first-token latency` — range: seconds
  - Time in seconds from query submission to the generation of the first output token, reported at P50 and P90 percentiles.

## Input / output format

**Input**: Audio recordings of spoken user queries (synthetic or human-spoken) from the AudioCRAG benchmark.

**Output**: Text responses or audio responses (speech-in-speech-out) generated by the model.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_latency(latency_values):
    sorted_vals = sorted(latency_values)
    p50 = sorted_vals[len(sorted_vals)//2]
    p90 = sorted_vals[int(len(sorted_vals)*0.9)]
    return {'P50': p50, 'P90': p90}
```

## Common pitfalls

- Latency measurements for synthetic audio exclude end-point detection latency, which is mandatory in production systems and would increase reported times.
- Speech output accuracy is consistently lower than text output accuracy due to difficulties in generating uncommon entity nouns directly in audio.
- Post-training is performed exclusively on synthetic data, so human-spoken benchmark results reflect zero-shot generalization rather than direct training.

## Evidence (verbatim from paper)

> All models are assessed on both the AudioCRAG-Synthetic (Syn.) and AudioCRAG-Human (Hum.). In the Closed Book setting, where models rely solely on their internal knowledge without access to external tools (reference length = 0), all models achieve accuracy scores below 20%.

## Citation

```bibtex
@misc{arora2025streamrag,
  title={Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage},
  author={Arora et al. (2025)},
  year={2025},
  note={arXiv:2510.02044}
}
```

- arXiv: 2510.02044