Voicebench Qa Eval

---
name: voicebench-qa-eval
description: Evaluates an agent's reasoning and tool-use capabilities in spoken question answering by requiring it to process audio queries, optionally perform web searches, and generate accurate responses. Use when the user wants to benchmark on OpenBookQA, AlpacaEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.23049
  bibtex_key: maben2025aura
  confidence: high
---

# voicebench-qa-eval

> AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks — Maben et al. (2025) (arXiv:2506.23049, 2025)

## What this evaluates

Evaluates an agent's reasoning and tool-use capabilities in spoken question answering by requiring it to process audio queries, optionally perform web searches, and generate accurate responses.

## Datasets

- **OpenBookQA** — total ?; splits: test (-1)
- **AlpacaEval** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total questions) * 100.
- `score (1-5)` — range: other
  - Open-ended response quality rated on a 1 to 5 scale by GPT-4o-mini.

## Input / output format

**Input**: Audio query (spoken question) transcribed by ASR, passed to LLM agent with tool-use prompts.

**Output**: Text response (or audio via TTS) containing the answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_alpaca_score(predictions):
    # Scored externally by GPT-4o-mini on 1-5 scale
    return average_gpt4o_scores(predictions)
```

## Common pitfalls

- ASR transcription errors can significantly degrade downstream accuracy if not mitigated.
- AlpacaEval relies on GPT-4o-mini for scoring, which may introduce bias or inconsistency compared to human judges.
- Web search prompting is optional but significantly boosts performance; omitting it changes the evaluation conditions.

## Evidence (verbatim from paper)

> We evaluated AURA’s reasoning and tool use capabilities in two VoiceBench QA tasks: AlpacaEval (open-ended, scored 1–5 by GPT-4o-mini) and OpenBookQA (multiple-choice with ground-truth answers). AURA achieves an accuracy of 92.75% on OpenBookQA—competitive with GPT-4o and outperforming all other open-weight systems.

## Citation

```bibtex
@misc{maben2025aura,
  title={AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks},
  author={Maben et al. (2025)},
  year={2025},
  note={arXiv:2506.23049}
}
```

- arXiv: 2506.23049