Speechmentalmanip Eval

---
name: speechmentalmanip-eval
description: Binary classification of spoken multi-speaker dialogues to detect the presence of mental manipulation tactics. It probes an audio-language model's ability to identify subtle manipulative cues in synthetic speech without relying on text transcripts. Use when the user wants to benchmark on SpeechMentalManip, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08342
  bibtex_key: chen2026detecting
  confidence: high
---

# speechmentalmanip-eval

> Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialogue — Run Chen et al. (arXiv:2601.08342, 2026)

## What this evaluates

Binary classification of spoken multi-speaker dialogues to detect the presence of mental manipulation tactics. It probes an audio-language model's ability to identify subtle manipulative cues in synthetic speech without relying on text transcripts.

## Datasets

- **SpeechMentalManip** — total 699; splits: test (699); repo https://github.com/runjchen/speech_mentalmanip

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified audio clips (manipulative vs. non-manipulative) out of the total evaluation set.
- `recall` — range: [0, 1]
  - True positive rate for the manipulative class, measuring the fraction of actual manipulative clips correctly identified.

## Input / output format

**Input**: Audio clips of multi-speaker synthetic dialogues generated via a two-phase TTS pipeline.

**Output**: Binary classification label: 'manipulative' or 'non-manipulative'.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)

def compute_recall(preds, gold, pos='manipulative'):
    tp = sum(1 for p, g in zip(preds, gold) if p == pos and g == pos)
    actual = sum(1 for g in gold if g == pos)
    return tp / actual if actual > 0 else 0.0
```

## Common pitfalls

- Models may exhibit a systematic bias toward predicting 'manipulative' due to safety alignment mechanisms over-flagging agitated prosody, as noted in preliminary trials with other models.
- The dataset uses synthetic TTS audio rather than original movie recordings, which prioritizes experimental control over ecological realism and may affect prosodic cue perception.

## Evidence (verbatim from paper)

> Prior evaluation on such benchmark indicate that few-shot GPT-4 Turbo reaches 0.724 accuracy and a finetuned LLaMA-2-13B achieves 0.768 accuracy on the core detection task. This process yields 609 manipulative and 90 non-manipulative audio clips used for evaluation.

## Citation

```bibtex
@misc{chen2026detecting,
  title={Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialogue},
  author={Run Chen et al.},
  year={2026},
  note={arXiv:2601.08342}
}
```

- arXiv: 2601.08342