Voiceagentbench Eval

---
name: voiceagentbench-eval
description: Evaluates speech language models and ASR-LLM pipelines on agentic speech tasks. It probes single/multi-tool orchestration, multi-turn dialogue, and safety refusal capabilities across multiple languages, including English, Hindi, and five Indic languages. Use when the user wants to benchmark on VoiceAgentBench, or asks about evaluating this task. Reports PF (Parameter Filling).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.07978
  bibtex_key: jain2025voiceagentbench
  confidence: high
---

# voiceagentbench-eval

> VoiceAgentBench: Are Voice Assistants ready for agentic tasks? — Dhruv Jain et al. (2025) (arXiv:2510.07978, 2025)

## What this evaluates

Evaluates speech language models and ASR-LLM pipelines on agentic speech tasks. It probes single/multi-tool orchestration, multi-turn dialogue, and safety refusal capabilities across multiple languages, including English, Hindi, and five Indic languages.

## Datasets

- **VoiceAgentBench** — total 5500; splits: English (-1), Hindi (-1), Indic (-1), Source-native (-1), Safety (-1)

## Metrics

- `PF (Parameter Filling)` **(primary)** — range: percent
  - Percentage of queries where the model correctly identifies the target tool, matches the required schema, and accurately fills all necessary parameters. Calculated as the average across task categories.
- `TS (Tool Selection)` — range: percent
  - Binary accuracy for correctly identifying the intended tool from the available set. Often near-perfect for single-tool tasks.
- `TCS (Tool Call Schema)` — range: percent
  - Binary accuracy for correctly structuring the tool call according to the required schema/format, independent of parameter values.
- `Refusal Rate` — range: percent
  - Percentage of safety/harmful queries where the model explicitly refuses to execute the request or generate a tool call.

## Input / output format

**Input**: Synthetic spoken audio queries (or transcribed text for ASR-LLM baselines) accompanied by tool definitions, system instructions, and multi-turn dialogue context.

**Output**: Structured tool call arguments (tool name, schema, parameters) or a natural language refusal response.

## Scoring recipe

```python
def score_instance(pred, gold):
    ts = 1 if pred['tool_name'] == gold['tool_name'] else 0
    tcs = 1 if pred['schema'] == gold['schema'] else 0
    pf = 1 if (tcs == 1 and pred['params'] == gold['params']) else 0
    return {'TS': ts, 'TCS': tcs, 'PF': pf}

def compute_metrics(predictions, golds):
    scores = [score_instance(p, g) for p, g in zip(predictions, golds)]
    return {
        'TS': sum(s['TS'] for s in scores) / len(scores),
        'TCS': sum(s['TCS'] for s in scores) / len(scores),
        'PF': sum(s['PF'] for s in scores) / len(scores)
    }
```

## Common pitfalls

- TS scores are trivially high for single-tool calling tasks and do not reflect true agentic capability; PF is the meaningful metric for tool execution.
- PF performance heavily depends on ASR transcription quality for Indic languages; replacing ground-truth transcripts can boost PF by >24%, masking model reasoning limits.
- Safety/refusal rates are highly sensitive to explicit refusal prompts in the system prompt; removing them causes sharp drops in refusal rates.

## Evidence (verbatim from paper)

> Metrics include TS, TCS, and PF (see Section [3.3] for definitions). For Indic subset we report average across all the 5 Indic languages. TS for Single Tool Calling is trivial, often yielding near-perfect scores.

## Citation

```bibtex
@misc{jain2025voiceagentbench,
  title={VoiceAgentBench: Are Voice Assistants ready for agentic tasks?},
  author={Dhruv Jain et al. (2025)},
  year={2025},
  note={arXiv:2510.07978}
}
```

- arXiv: 2510.07978