Streammark Eval

---
name: streammark-eval
description: This benchmark evaluates the imperceptibility, robustness to benign audio transformations, and semi-fragility to malicious deepfake manipulations of a deep learning-based audio watermarking system. It specifically probes whether a watermark can survive standard compression and cropping while being deliberately destroyed by semantic-altering AI conversions like voice cloning or speech editing. Use when the user wants to benchmark on LibriSpeech (train_clean100), Test Set A, Test Set B, or asks about evaluating this task. Reports ACC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11917
  bibtex_key: [REDACTED]
  confidence: high
---

# streammark-eval

> StreamMark: A Deep Learning-Based Semi-Fragile Audio Watermarking for Proactive Deepfake Detection — Liu et al. (2026) (arXiv:2604.11917, 2026)

## What this evaluates

This benchmark evaluates the imperceptibility, robustness to benign audio transformations, and semi-fragility to malicious deepfake manipulations of a deep learning-based audio watermarking system. It specifically probes whether a watermark can survive standard compression and cropping while being deliberately destroyed by semantic-altering AI conversions like voice cloning or speech editing.

## Datasets

- **LibriSpeech (train_clean100)** — total ?; splits: train (-1)
- **Test Set A** — total ?; splits: test_A (-1)
- **Test Set B** — total ?; splits: test_B (-1); repo https://github.com/L1uZhentao/deepfake_benchmark

## Metrics

- `SNR` — range: other
  - Signal-to-Noise Ratio in decibels, measuring the power ratio between the clean audio and the watermarked audio.
- `PESQ` — range: other
  - Perceptual Evaluation of Speech Quality, a standardized objective measure of speech quality that correlates with human perception.
- `SECS` — range: [0, 1]
  - Speaker Encoder Cosine Similarity, computed as the cosine similarity between speaker embeddings extracted from the clean and watermarked audio.
- `ACC` **(primary)** — range: percent
  - Message Recovery Accuracy, calculated as the percentage of correctly recovered bits out of the total 16-bit message length.

## Input / output format

**Input**: Clean audio waveform (from LibriSpeech test_clean subset)

**Output**: Watermarked audio waveform (for imperceptibility/robustness evaluation) and/or decoded 16-bit binary message (for ACC evaluation)

## Scoring recipe

```python
def score(predictions, gold):
    clean_audio, original_msg = gold
    watermarked_audio, decoded_msg = predictions
    snr = calculate_snr(clean_audio, watermarked_audio)
    pesq = calculate_pesq(clean_audio, watermarked_audio)
    secs = cosine_similarity(speaker_embed(clean_audio), speaker_embed(watermarked_audio))
    acc = sum(p == g for p, g in zip(decoded_msg, original_msg)) / len(original_msg) * 100
    return {'SNR': snr, 'PESQ': pesq, 'SECS': secs, 'ACC': acc}
```

## Common pitfalls

- Message Recovery Accuracy (ACC) dropping to ~50% under deepfake attacks is the intended 'fragile' behavior (equivalent to random guessing for 16-bit messages), not a model failure.
- The benchmark explicitly distinguishes between benign (semantic-preserving) and malicious (semantic-altering) AI conversions; misclassifying a benign style transfer as a deepfake attack violates the semi-fragility design goal.
- Opus encoding is an out-of-distribution attack not seen during training; high robustness here may not generalize to other codecs without retraining.

## Evidence (verbatim from paper)

> The primary evaluation metrics were Signal-to-Noise Ratio (SNR), Perceptual Evaluation of Speech Quality (PESQ) , and Speaker Encoder Cosine Similarity (SECS) for imperceptibility, and Message Recovery Accuracy (ACC) for robustness and fragility.

## Citation

```bibtex
@misc{liu2026streammark,
  title={StreamMark: A Deep Learning-Based Semi-Fragile Audio Watermarking for Proactive Deepfake Detection},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.11917}
}
```

- arXiv: 2604.11917