Korean Vlm Benchmarks Eval

---
name: korean-vlm-benchmarks-eval
description: Evaluates vision-language models on Korean multimodal comprehension, document/table/chart understanding, and open-ended generation capabilities using translated and newly curated benchmarks. Use when the user wants to benchmark on K-MMBench, K-SEED, K-MMStar, K-DTCBench, K-LLaVA-W, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.19103
  bibtex_key: ju2024varcovision
  confidence: high
---

# korean-vlm-benchmarks-eval

> VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models — Ju et al. (2024) (arXiv:2411.19103, 2024)

## What this evaluates

Evaluates vision-language models on Korean multimodal comprehension, document/table/chart understanding, and open-ended generation capabilities using translated and newly curated benchmarks.

## Datasets

- **K-MMBench** — total ?; splits: dev (-1); HF `NCSOFT/K-MMBench`
- **K-SEED** — total ?; splits: test (-1); HF `NCSOFT/K-SEED`
- **K-MMStar** — total 1500; splits: test (1500); HF `NCSOFT/K-MMStar`
- **K-DTCBench** — total 240; splits: test (240); HF `NCSOFT/K-DTCBench`
- **K-LLaVA-W** — total 60; splits: test (60); HF `NCSOFT/K-LLaVA-W`

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total questions) * 100.
- `JudgeLLM ratio score` — range: percent
  - Ratio of the target model's JudgeLLM score to the baseline model's JudgeLLM score. JudgeLLM (GPT-4o) evaluates helpfulness, relevance, accuracy, level of detail, and Korean generation capability based on provided captions.

## Input / output format

**Input**: An image paired with a Korean question (multiple-choice or open-ended).

**Output**: For MCQA: a single choice letter or answer string. For K-LLaVA-W: a free-form Korean text response.

## Scoring recipe

```python
# For MCQA datasets (K-MMBench, K-SEED, K-MMStar, K-DTCBench)
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
accuracy = (correct / len(golds)) * 100

# For K-LLaVA-W
target_scores = [judge_llm.evaluate(q, img, pred, caption) for q, img, pred, caption in zip(questions, images, predictions, captions)]
baseline_scores = [judge_llm.evaluate(q, img, baseline_pred, caption) for q, img, baseline_pred, caption in zip(questions, images, baselines, captions)]
ratio_score = (sum(target_scores) / sum(baseline_scores)) * 100
```

## Common pitfalls

- Translation artifacts and localization issues require human post-editing to ensure fluency and accuracy.
- K-MMStar originally contained unanswerable cases (e.g., requiring multiple images); questions were modified or recreated to ensure single-image answerability.
- K-LLaVA-W evaluation relies on GPT-4o as both JudgeLLM and baseline generator, which may introduce model-specific biases compared to standard LLM-as-a-judge setups.

## Evidence (verbatim from paper)

> JudgeLLM measures the model’s helpfulness, relevance, accuracy, level of detail, and Korean generation capability. The final K-LLaVA-W score of the target model is calculated as the ratio of the target model’s JudgeLLM score to the baseline model’s JudgeLLM score.

## Citation

```bibtex
@misc{ju2024varcovision,
  title={VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models},
  author={Ju et al. (2024)},
  year={2024},
  note={arXiv:2411.19103}
}
```

- arXiv: 2411.19103