Mavos Dd Eval

---
name: mavos-dd-eval
description: Evaluates deepfake detection models on distinguishing real from fake audio-video content under closed-set and open-set conditions. It specifically probes cross-model and cross-lingual generalization by testing on unseen generation methods and languages. Use when the user wants to benchmark on MAVOS-DD, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.11109
  bibtex_key: croitoru2025mavosdd
  confidence: high
---

# mavos-dd-eval

> MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark — Florinel-Alin Croitoru et al. (arXiv:2505.11109, 2025)

## What this evaluates

Evaluates deepfake detection models on distinguishing real from fake audio-video content under closed-set and open-set conditions. It specifically probes cross-model and cross-lingual generalization by testing on unseen generation methods and languages.

## Datasets

- **MAVOS-DD** — total ?; splits: train (-1), in-domain (-1), open-set model (-1), open-set language (-1), open-set full (-1)

## Metrics

- `mAP` **(primary)** — range: [0, 1]
  - Mean Average Precision across real and fake classes. Computed as the area under the precision-recall curve.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `acc` — range: [0, 1]
  - Accuracy, calculated as the proportion of correctly classified samples (true positives + true negatives) out of the total number of samples.

## Input / output format

**Input**: Synchronized audio and video frames (or video-only for unimodal baselines) representing either real or deepfake content.

**Output**: Binary prediction: 'real' or 'fake' (or a confidence score for thresholding).

## Scoring recipe

```python
def compute_metrics(predictions, labels):
    acc = sum(p == l for p, l in zip(predictions, labels)) / len(labels)
    auc = roc_auc_score(labels, predictions)
    mAP = average_precision_score(labels, predictions)
    return {'acc': acc, 'AUC': auc, 'mAP': mAP}
```

## Common pitfalls

- Pre-trained models perform near random chance (~50%); fine-tuning on the benchmark's training split is mandatory for meaningful evaluation.
- Open-set scenarios (unseen generation models or languages) cause severe performance degradation, especially for unimodal detectors compared to multimodal ones.
- False positives increase significantly when the test audio contains languages absent from the training set, as detectors may flag unfamiliar speech as synthetic.

## Evidence (verbatim from paper)

> In Table[3], we report the results for the three baseline models across three evaluation metrics: mean average precision (mAP), area under the ROC curve (AUC), and accuracy (acc). We report these values on all four test sets: in-domain, open-set model, open-set language and open-set full.

## Citation

```bibtex
@misc{croitoru2025mavosdd,
  title={MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark},
  author={Florinel-Alin Croitoru et al.},
  year={2025},
  note={arXiv:2505.11109}
}
```

- arXiv: 2505.11109