Svld Points Ratio Eval

---
name: svld-points-ratio-eval
description: Probes a model's ability to predict social engagement (upvote ratio) from multimodal inputs (images, videos, and text). It evaluates cross-modal fusion and regression capabilities on socially grounded, context-rich data. Use when the user wants to benchmark on SVLD, or asks about evaluating this task. Reports Mean L1point ratio prediction error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2006.08335
  bibtex_key: xue2020svld
  confidence: high
---

# svld-points-ratio-eval

> A Dataset and Benchmarks for Multimedia Social Analysis — Xue et al. (2020) (arXiv:2006.08335, 2020)

## What this evaluates

Probes a model's ability to predict social engagement (upvote ratio) from multimodal inputs (images, videos, and text). It evaluates cross-modal fusion and regression capabilities on socially grounded, context-rich data.

## Datasets

- **SVLD** — total 365336; splits: train (324888), val (40448), test (-1)

## Metrics

- `Mean L1point ratio prediction error` **(primary)** — range: [0, 1]
  - Mean Absolute Error (L1 loss) between the predicted points ratio and the ground-truth ratio (upvotes / total votes). Lower values indicate better regression performance.

## Input / output format

**Input**: Per post: an image (resized to 224x224), a video (up to 64 frames sampled at ≤5 fps, padded with the last frame if needed), and three text fields (description, title, top comment) tokenized with BERT and padded/truncated to 102 tokens.

**Output**: A single scalar float representing the predicted points ratio (upvotes divided by total votes).

## Scoring recipe

```python
def compute_metric(predictions, gold):
    errors = [abs(p - g) for p, g in zip(predictions, gold)]
    return sum(errors) / len(errors)
```

## Common pitfalls

- The test set is explicitly marked as 'unreleased', so external evaluation is not possible without author access.
- The metric is L1 error (lower is better), not accuracy or correlation; readers may mistakenly treat it as a classification score.
- Video preprocessing pads with the last frame when frames are insufficient, which can bias results if not replicated exactly.

## Evidence (verbatim from paper)

> Our goal in this baseline experiment is to model the points-ratio of a sample, the number of upvotes divided by the total number of votes. This is a number between zero and one, which roughly reflects the attitude of the community towards a particular post. Table 3 reports the Mean L1point ratio prediction error.

## Citation

```bibtex
@misc{xue2020svld,
  title={A Dataset and Benchmarks for Multimedia Social Analysis},
  author={Xue et al. (2020)},
  year={2020},
  note={arXiv:2006.08335}
}
```

- arXiv: 2006.08335