Echox Speech Qa Eval

---
name: echox-speech-qa-eval
description: Evaluates the knowledge-based question-answering capabilities of speech-to-speech and speech-to-text models on audio and text inputs. Use when the user wants to benchmark on Llama Questions, Web Questions, TriviaQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.09174
  bibtex_key: zhang2025echox
  confidence: high
---

# echox-speech-qa-eval

> EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs — Zhang et al. (2025) (arXiv:2509.09174, 2025)

## What this evaluates

Evaluates the knowledge-based question-answering capabilities of speech-to-speech and speech-to-text models on audio and text inputs.

## Datasets

- **Llama Questions** — total ?; splits: test (-1)
- **Web Questions** — total ?; splits: test (-1)
- **TriviaQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 100] percent
  - Exact match or F1 score computed by the UltraEval-Audio toolkit on knowledge-based QA benchmarks. Values are reported as percentages averaged across datasets.

## Input / output format

**Input**: Spoken audio questions (for speech-to-speech evaluation) or transcribed text questions (for speech-to-text evaluation).

**Output**: Generated spoken audio responses (for speech-to-speech) or text responses (for speech-to-text), evaluated against gold answers.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    scores = []
    for pred, gold in zip(predictions, gold_answers):
        # UltraEval-Audio handles transcription if needed, then computes exact match/F1
        score = ultra_eval_toolkit.compute_qa_score(pred, gold)
        scores.append(score)
    return sum(scores) / len(scores) * 100
```

## Common pitfalls

- The evaluation relies on the UltraEval-Audio toolkit; using a different toolkit may yield different scores due to variations in ASR transcription or QA scoring logic.
- Speech-to-speech results depend on the unit-based HiFi-GAN vocoder for audio generation, but the reported metric focuses on QA accuracy rather than audio quality.
- The paper reports an 'Avg.' column across the three benchmarks, which should be computed as the mean of the individual dataset scores.

## Evidence (verbatim from paper)

> For evaluation, we use the UltraEval-Audio toolkit. We mainly conduct experiments on the three benchmarks: Llama questions (Nachmani et al., 2023), Web questions (Berant et al., 2013), and TriviaQA (Joshi et al., 2017).

## Citation

```bibtex
@misc{zhang2025echox,
  title={EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.09174}
}
```

- arXiv: 2509.09174