Deepaction Eval

---
name: deepaction-eval
description: This benchmark evaluates the ability of multi-modal embedding classifiers to distinguish real human motion videos from AI-generated ones. It probes semantic consistency detection, robustness to video laundering (resolution/compression), and generalization to unseen generative models. Use when the user wants to benchmark on DeepAction, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.00526
  bibtex_key: bohacek2024humanactionclips
  confidence: high
---

# deepaction-eval

> Human Action CLIPs: Detecting AI-generated Human Motion — Bohacek et al. (2024) (arXiv:2412.00526, 2024)

## What this evaluates

This benchmark evaluates the ability of multi-modal embedding classifiers to distinguish real human motion videos from AI-generated ones. It probes semantic consistency detection, robustness to video laundering (resolution/compression), and generalization to unseen generative models.

## Datasets

- **DeepAction** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Macro-average accuracy: the unweighted mean of per-class accuracies, (Accuracy_real + Accuracy_fake) / 2. Reported separately at frame and video levels, averaged over five random train/test repetitions.

## Input / output format

**Input**: Video frames or entire videos represented as multi-modal semantic embeddings (e.g., CLIP, SigLIP, JinaCLIP, or fine-tuned CLIP).

**Output**: Binary label (real vs. AI-generated) or multi-class label (specific AI model), produced by an SVM classifier operating on the embeddings.

## Scoring recipe

```python
# predictions: list of predicted labels (0=real, 1=fake)
# gold: list of true labels
acc_real = sum(p == g for p, g in zip(predictions, gold) if g == 0) / sum(1 for g in gold if g == 0)
acc_fake = sum(p == g for p, g in zip(predictions, gold) if g == 1) / sum(1 for g in gold if g == 1)
macro_acc = (acc_real + acc_fake) / 2
return macro_acc * 100  # percent
```

## Common pitfalls

- Splits are determined at the action (prompt) level, not the video level, to prevent data leakage across train/test partitions.
- The dataset is imbalanced with significantly more fake videos; under-sampling is applied, but macro-average accuracy must be reported to avoid class-bias.
- Video-level accuracy is evaluated independently from frame-level accuracy, not merely averaged from frame predictions.

## Evidence (verbatim from paper)

> In each case, we report the mean frame- and video-level accuracy on the test set, averaged over five random train/test repetitions. Because our dataset is imbalanced, with significantly more fake than real videos, we under-sample the fake videos. Throughout, we report accuracy as a macro-average by evenly weighting the class accuracies.

## Citation

```bibtex
@misc{bohacek2024humanactionclips,
  title={Human Action CLIPs: Detecting AI-generated Human Motion},
  author={Bohacek et al. (2024)},
  year={2024},
  note={arXiv:2412.00526}
}
```

- arXiv: 2412.00526