Sdsko Pub Vdr Eval

---
name: sdsko-pub-vdr-eval
description: Evaluates a model's ability to retrieve relevant Korean public document pages given a text query, comparing text-only parsing against multimodal visual understanding. It probes cross-modal reasoning, layout awareness, and the capacity to interpret tables, charts, and complex visual structures in administrative documents. Use when the user wants to benchmark on SDS KoPub VDR, or asks about evaluating this task. Reports Recall@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.04910
  bibtex_key: lee2025sdsko-pub-vdr
  confidence: high
---

# sdsko-pub-vdr-eval

> SDS KoPub VDR: A Benchmark Dataset for Visual Document Retrieval in Korean Public Documents — Jaehoon Lee et al. (2025) (arXiv:2511.04910, 2025)

## What this evaluates

Evaluates a model's ability to retrieve relevant Korean public document pages given a text query, comparing text-only parsing against multimodal visual understanding. It probes cross-modal reasoning, layout awareness, and the capacity to interpret tables, charts, and complex visual structures in administrative documents.

## Datasets

- **SDS KoPub VDR** — total 600; splits: test (600)

## Metrics

- `Recall@k` **(primary)** — range: [0, 1]
  - Binary indicator of whether the ground-truth page appears within the top-k retrieved results. Averaged over all queries. Evaluated at k=1, 3, 5, 10.
- `nDCG@k` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank k. For single-ground-truth queries, it measures how early the correct page appears in the ranked list, discounting relevance by log2(rank+1). Evaluated at k=1, 3, 5, 10.

## Input / output format

**Input**: Task 1: Text query + PDF-extracted text of a candidate page. Task 2: Text query + full-page image of a candidate page.

**Output**: Ranked list of top-10 candidate pages sorted by cosine similarity to the query embedding.

## Scoring recipe

```python
import math
def score(predictions, golds, k=10):
    recalls, ndcgs = [], []
    for pred, gold in zip(predictions, golds):
        rank = pred.index(gold) + 1 if gold in pred else len(pred) + 1
        recalls.append(1.0 if rank <= k else 0.0)
        dcg = 1.0 / math.log2(rank + 1) if rank <= k else 0.0
        idcg = 1.0 / math.log2(2)
        ndcgs.append(dcg / idcg)
    return sum(recalls)/len(recalls), sum(ndcgs)/len(ndcgs)
```

## Common pitfalls

- Relying solely on PDF-extracted text for Task 1 leads to severe performance degradation on pages with tables, charts, or complex layouts due to parsing errors.
- Using non-native embedding dimensions or applying projection layers instead of the model's native configuration introduces memory-based bias and breaks comparability.
- Confusing the input modality: Task 1 requires text-only inputs, while Task 2 requires full-page images; swapping them invalidates the modality-specific evaluation.

## Evidence (verbatim from paper)

> The performance of each retrieval model is evaluated using two standard metrics widely adopted in information-retrieval research: Recall and Normalized Discounted Cumulative Gain (nDCG). Recall@k measures whether the relevant document appears within the top-k retrieved results, thereby assessing the model's coverage of correct answers. It serves as an indicator of whether the retrieval system successfully includes the ground-truth document among its top-ranked candidates. nDCG@k evaluates the ranking quality of the retrieved list by assigning higher scores when the correct document is ranked closer to the top.

## Citation

```bibtex
@misc{lee2025sdsko-pub-vdr,
  title={SDS KoPub VDR: A Benchmark Dataset for Visual Document Retrieval in Korean Public Documents},
  author={Jaehoon Lee et al. (2025)},
  year={2025},
  note={arXiv:2511.04910}
}
```

- arXiv: 2511.04910