Brace Hallucination Eval

---
name: brace-hallucination-eval
description: Probes models' robustness in detecting subtle hallucinations in audio captions, specifically those introduced via LLM-driven noun substitution. It measures the ability to identify semantically flawed or factually incorrect descriptions against audio ground truth. Use when the user wants to benchmark on BRACE-Hallucination, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.10403
  bibtex_key: guo2025brace
  confidence: high
---

# brace-hallucination-eval

> BRACE: A Benchmark for Robust Audio Caption Quality Evaluation — Guo et al. (2025) (arXiv:2512.10403, 2025)

## What this evaluates

Probes models' robustness in detecting subtle hallucinations in audio captions, specifically those introduced via LLM-driven noun substitution. It measures the ability to identify semantically flawed or factually incorrect descriptions against audio ground truth.

## Datasets

- **BRACE-Hallucination** — total ?; splits: test (-1); repo https://github.com/HychTus/BRACE_Evaluation

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for identifying hallucinated captions or selecting the non-hallucinated caption in a pair.

## Input / output format

**Input**: Audio clip paired with caption pairs where one may contain subtle hallucinations (e.g., substituted nouns) or a single caption for hallucination detection.

**Output**: Model outputs a preference choice, hallucination flag, or quality score.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- CLAP models overlook fine-grained acoustic details and syntactic errors, leading to inflated similarity scores for flawed captions.
- LALMs exhibit strong position bias and poor instruction following under complex prompts, often outputting invalid responses like 'none'.

## Evidence (verbatim from paper)

> On BRACE-Hallucination, the top-performing model M2D-CLAP reaches an F1-score of 88.26, though performance still varies significantly across models.

## Citation

```bibtex
@misc{guo2025brace,
  title={BRACE: A Benchmark for Robust Audio Caption Quality Evaluation},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2512.10403}
}
```

- arXiv: 2512.10403