Click Eval

---
name: click-eval
description: Evaluates language models' proficiency in Korean cultural knowledge (e.g., history, law, society, economy) and linguistic competence (e.g., grammar, functional usage) using multiple-choice questions sourced from official Korean exams and textbooks. Use when the user wants to benchmark on CLIcK, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.06412
  bibtex_key: kim2024click
  confidence: high
---

# click-eval

> CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean — Kim et al. (2024) (arXiv:2403.06412, 2024)

## What this evaluates

Evaluates language models' proficiency in Korean cultural knowledge (e.g., history, law, society, economy) and linguistic competence (e.g., grammar, functional usage) using multiple-choice questions sourced from official Korean exams and textbooks.

## Datasets

- **CLIcK** — total 1995; splits: test (1995); repo https://github.com/rladmstn1714/CLIcK

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Instance accuracy is computed as count(correct answers) / 3N, where N is the number of options. The model is evaluated 3 times per instance with cyclically permuted option orders, and the average score across all instances is reported.

## Input / output format

**Input**: Multiple-choice questions with 4 or 5 options (A/B/C/D or A/B/C/D/E). Prompts are provided in two variants: Type 1 includes background context, Type 2 does not. Each question is presented 3 times with cyclically permuted option positions.

**Output**: A single uppercase alphabet corresponding to the selected option (e.g., 'A'). For API models, verbose responses are accepted if they clearly indicate one option per predefined acceptance criteria.

## Scoring recipe

```python
correct_count = 0
for instance in dataset:
    options = instance.options
    gold = instance.gold_answer
    for perm in cyclic_permutations(options, k=3):
        prompt = build_prompt(instance, perm, context=instance.context)
        response = model.generate(prompt, temperature=0)
        pred = extract_answer(response) # applies acceptance criteria
        if pred == gold:
            correct_count += 1
accuracy = correct_count / (3 * len(options))
```

## Common pitfalls

- Failing to average over the 3 cyclic permutations of option orders, which can introduce position bias.
- Using strict exact-match string comparison for API-generated responses, whereas the protocol allows flexible extraction (e.g., 'the answer is A') based on Jin et al. criteria.
- Ignoring the distinction between Type 1 (with context) and Type 2 (without context) prompts, as model performance varies significantly between them.

## Evidence (verbatim from paper)

> Our primary evaluation metric is accuracy. We report the average accuracy over the entire dataset. As we prompt the model 3 times and adopt cyclic permutation for each instance, the total number of experiments per instance is $3N$, where $N$ denotes the “number of options”. Instance accuracy is computed as: $p_{	ext{accuracy}}=rac{	ext{count}(	ext{correct answers})}{3N}$

## Citation

```bibtex
@misc{kim2024click,
  title={CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2403.06412}
}
```

- arXiv: 2403.06412