Bss Eval

---
name: bss-eval
description: Evaluates speech language models' ability to process beyond-semantic speech attributes, including dialect comprehension, multi-turn context memory, emotion perception, age-aware response generation, and non-verbal cue handling. It probes whether models can integrate paralinguistic, affective, and contextual signals beyond literal semantic understanding to achieve human-like social interaction. Use when the user wants to benchmark on BoSS Evaluation Datasets, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.17563
  bibtex_key: wang2025bss
  confidence: high
---

# bss-eval

> BoSS: Beyond-Semantic Speech — Wang et al. (2025) (arXiv:2507.17563, 2025)

## What this evaluates

Evaluates speech language models' ability to process beyond-semantic speech attributes, including dialect comprehension, multi-turn context memory, emotion perception, age-aware response generation, and non-verbal cue handling. It probes whether models can integrate paralinguistic, affective, and contextual signals beyond literal semantic understanding to achieve human-like social interaction.

## Datasets

- **BoSS Evaluation Datasets** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact string match against reference substrings for AQA; dialect classification model output (1/0) for chitchat; averaged across instances.
- `Score (%)` — range: [0, 100]
  - GPT-4o or Emotion2Vec judge scores on 0-5 scale, scaled via Avg(S) = (100/|S|) * Σ(s/5)^p, then averaged.

## Input / output format

**Input**: Audio input containing speech (dialect, multi-turn, emotional, age-cued, or with non-verbal sounds), optionally accompanied by text context or reference answers.

**Output**: Text response (Mandarin or target dialect) or audio response, depending on the task.

## Scoring recipe

```python
def score_bss(predictions, golds, task_type, p=1.0):
    scores = []
    for pred, gold in zip(predictions, golds):
        if task_type == 'dialect_aqa':
            scores.append(1.0 if any(sub in pred for sub in gold) else 0.0)
        elif task_type == 'dialect_chitchat':
            scores.append(1.0 if classify_dialect(pred) == gold else 0.0)
        elif task_type in ['open_ended', 'emotion', 'age', 'nonverbal']:
            raw = judge_model.evaluate(pred, gold, scale=5)
            scores.append((raw / 5.0) ** p)
    return sum(scores) / len(scores) * 100 if task_type in ['open_ended', 'emotion', 'age', 'nonverbal'] else sum(scores) / len(scores)
```

## Common pitfalls

- Using LLM-based scoring for dialect AQA introduces ambiguity and boundary issues; exact string matching against reference substrings is required.
- Relying on TTS-generated emotional speech fails to capture authentic acoustic correlates; human-produced emotional audio must be used.
- Open-ended paralinguistic tasks suffer from score compression near the top of the scale; a power scaling function must be applied to raw 0-5 judge scores.

## Evidence (verbatim from paper)

> To assess dialectal comprehension, we adopt a string-matching strategy to determine answer correctness, foregoing conventional LLM-based scoring methods. This approach ensures the consistency of the evaluation and mitigates the ambiguity and boundary issues previously reported in [54]. For each AQA (Answer Questioning and Answering) test case, we construct a set of valid reference substrings. A model's textual response is marked as correct if it exactly matches any entry in this reference set. The overall dialectal performance is then computed as the average accuracy across all test instances.

## Citation

```bibtex
@misc{wang2025bss,
  title={BoSS: Beyond-Semantic Speech},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2507.17563}
}
```

- arXiv: 2507.17563