Vsdx Eval

---
name: vsdx-eval
description: Evaluates vision-language models' ability to perceive and reason about non-RGB sensor data (thermal, depth, X-ray). It probes low-level perception (existence, counting, position, description) and high-level understanding (contextual reasoning, sensor-specific physical property interpretation). Use when the user wants to benchmark on VS-TDX, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.20750
  bibtex_key: chung2024enhanced
  confidence: high
---

# vsdx-eval

> Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking — Sangyun Chung et al. (2024) (arXiv:2412.20750, 2024)

## What this evaluates

Evaluates vision-language models' ability to perceive and reason about non-RGB sensor data (thermal, depth, X-ray). It probes low-level perception (existence, counting, position, description) and high-level understanding (contextual reasoning, sensor-specific physical property interpretation).

## Datasets

- **VS-TDX** — total 10160; splits: test (10160)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of question-answer pairs where the model's generated response matches the ground-truth positive answer. Scores are reported per sub-task (Existence, Counting, Position, General Description, Contextual Understanding, Sensor Understanding) and aggregated into Perception and Understanding averages.

## Input / output format

**Input**: A single image (thermal, depth, or X-ray modality) paired with a natural language question.

**Output**: A natural language text response answering the question.

## Scoring recipe

```python
correct = 0
total = 0
for qa in dataset:
    pred = model.generate(qa.image, qa.question)
    if normalize(pred) == normalize(qa.gold):
        correct += 1
    total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Models are evaluated on non-RGB modalities that lack color cues, so RGB-biased priors often lead to systematic misinterpretations of physical properties.
- Sub-task scores (e.g., Counting, Position) are averaged to form the Perception and Understanding aggregate metrics rather than being combined into a single global accuracy score.
- The benchmark includes a negative answer set used for training (DNA optimization), but evaluation accuracy is computed solely against the positive answer set.

## Evidence (verbatim from paper)

> The final VS-TDX benchmark comprises approximately 6,791 sensor images, consisting of 1,867 thermal, 2,781 depth, and 2,143 X-ray images. It includes 10,160 diverse question-answer pairs, with an average of 1.50 pairs per image. TABLE I: Evaluation results of various Vision-Language Models (VLMs) on the VS-TDX benchmark, with accuracy as the primary metric.

## Citation

```bibtex
@misc{chung2024enhanced,
  title={Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking},
  author={Sangyun Chung et al. (2024)},
  year={2024},
  note={arXiv:2412.20750}
}
```

- arXiv: 2412.20750