Vidas Eval

---
name: vidas-eval
description: Evaluates a model's ability to assess danger levels in videos by identifying risk elements, understanding context, and assigning severity scores. It probes multimodal perception and risk reasoning capabilities. Use when the user wants to benchmark on ViDAS, or asks about evaluating this task. Reports MSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.00477
  bibtex_key: gupta2024vidas
  confidence: high
---

# vidas-eval

> ViDAS: Vision-based Danger Assessment and Scoring — Gupta et al. (2024) (arXiv:2410.00477, 2024)

## What this evaluates

Evaluates a model's ability to assess danger levels in videos by identifying risk elements, understanding context, and assigning severity scores. It probes multimodal perception and risk reasoning capabilities.

## Datasets

- **ViDAS** — total 100; splits: test (100)

## Metrics

- `MSE` **(primary)** — range: other
  - Mean Squared Error between predicted danger ratings and average human ratings. Formula: MSE = (1/n) * Σ(L_i - E_i^(avg))^2, where L_i is the LLM's predicted rating, E_i^(avg) is the average human rating, and n is the number of videos.

## Input / output format

**Input**: Video(s) accompanied by a prompt template (Zero-Shot, Fixed Few-Shot, or N-shot learning) requesting a danger level assessment.

**Output**: A single numerical danger rating score on a 0–10 scale.

## Scoring recipe

```python
def compute_mse(predictions, gold_ratings):
    n = len(predictions)
    mse = sum((p - g) ** 2 for p, g in zip(predictions, gold_ratings)) / n
    return mse
```

## Common pitfalls

- The metric uses average human ratings per video, not individual annotator scores.
- Performance depends heavily on prompt type (ZS vs FFS vs N-shot) and example selection, not just N value.
- LLMs may fail to detect hidden danger elements unless explicitly pointed out in the prompt or video context.

## Evidence (verbatim from paper)

> To compare the danger ratings predicted by a Language Model (LLM) to the average danger ratings given by humans, we use the Mean Squared Error (MSE) as our metric. The MSE provides a measure of the average squared difference between the predicted ratings and the actual average ratings. Let $L_{i}$ represent the danger rating predicted by the LLM for the $i$-th video, and let $E_{i}^{(	ext{avg})}$ denote the average danger rating given by human evaluators for the $i$-th video. The MSE can be formulated as follows: $\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}(L_{i}-E_{i}^{(\text{avg})})^{2}$ where $n$ is the total number of videos. The MSE thus quantifies the prediction accuracy by averaging the squared differences between the predicted and actual ratings overall videos.

## Citation

```bibtex
@misc{gupta2024vidas,
  title={ViDAS: Vision-based Danger Assessment and Scoring},
  author={Gupta et al. (2024)},
  year={2024},
  note={arXiv:2410.00477}
}
```

- arXiv: 2410.00477