Nacsp Eval

---
name: nacsp-eval
description: Evaluates a model's ability to predict discrete neural audio codec parameters (quantizers, sampling rate, bits per second) from audio samples, enabling fine-grained source attribution of AI-generated speech. The protocol frames open-set attribution as a multi-task regression problem rather than binary classification, requiring the model to generalize across both seen and unseen codec configurations. Use when the user wants to benchmark on ST-Codecfake, CodecFake, or asks about evaluating this task. Reports MSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.12627
  bibtex_key: phukan2025towards
  confidence: high
---

# nacsp-eval

> Towards Neural Audio Codec Source Parsing — Phukan et al. (2025) (arXiv:2506.12627, 2025)

## What this evaluates

Evaluates a model's ability to predict discrete neural audio codec parameters (quantizers, sampling rate, bits per second) from audio samples, enabling fine-grained source attribution of AI-generated speech. The protocol frames open-set attribution as a multi-task regression problem rather than binary classification, requiring the model to generalize across both seen and unseen codec configurations.

## Datasets

- **ST-Codecfake** — total ?; splits: train (70000), val (7000), test (158736)
- **CodecFake** — total ?; splits: train (42752), val (735), test (755)

## Metrics

- `MSE` **(primary)** — range: other
  - Mean Squared Error between predicted and ground truth values for quantizers (Q), sampling rate (SR), and bits per second (BPS).

## Input / output format

**Input**: Raw audio waveform samples corresponding to speech synthesized by various neural audio codecs.

**Output**: Three predicted values per sample: quantizers (Q), sampling rate (SR), and bits per second (BPS).

## Scoring recipe

```python
def compute_mse(preds, golds):
    # preds, golds: (N, 3) arrays for [Q, SR, BPS]
    mse_q = np.mean((preds[:, 0] - golds[:, 0]) ** 2)
    mse_sr = np.mean((preds[:, 1] - golds[:, 1]) ** 2)
    mse_bps = np.mean((preds[:, 2] - golds[:, 2]) ** 2)
    return {'MSE_Q': mse_q, 'MSE_SR': mse_sr, 'MSE_BPS': mse_bps}
```

## Common pitfalls

- The protocol explicitly restricts evaluation to fake samples only; including real audio violates the setup.
- The task is framed as multi-task regression for specific codec parameters, not binary real/fake classification.
- Dataset splits for CodecFake are provided per NAC subset; aggregating across subsets without stratification may skew results.

## Evidence (verbatim from paper)

> We use MSE (mean squared error) as loss function for all the three NACSP tasks (Q, BPS, SR) with Adam as the optimizer. We use only the fake samples and follow the distribution given by them for the training and evaluation of the models.

## Citation

```bibtex
@misc{phukan2025towards,
  title={Towards Neural Audio Codec Source Parsing},
  author={Phukan et al. (2025)},
  year={2025},
  note={arXiv:2506.12627}
}
```

- arXiv: 2506.12627