Cage Korset Eval

---
name: cage-korset-eval
description: Probes large language models' vulnerability to culturally-adapted adversarial prompts across Korean and Khmer contexts. It measures how effectively models resist harmful intent when framed within local socio-technical norms, safety policies, and cultural specifics, rather than generic or translated attacks. Use when the user wants to benchmark on KorSET, or asks about evaluating this task. Reports Attack Success Rate (ASR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.20170
  bibtex_key: kim2026cage
  confidence: high
---

# cage-korset-eval

> CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generation — Kim et al. (2026) (arXiv:2602.20170, 2026)

## What this evaluates

Probes large language models' vulnerability to culturally-adapted adversarial prompts across Korean and Khmer contexts. It measures how effectively models resist harmful intent when framed within local socio-technical norms, safety policies, and cultural specifics, rather than generic or translated attacks.

## Datasets

- **KorSET** — total ?; splits: test (-1); repo https://github.com/selectstar-ai/CAGE-paper

## Metrics

- `Attack Success Rate (ASR)` **(primary)** — range: percent
  - Percentage of adversarial prompts that successfully elicit a policy-violating or harmful response from the target LLM, as determined by an automated judge (GPT-4.1) using a defined safety policy and scoring rubric.

## Input / output format

**Input**: Culturally grounded adversarial prompt (e.g., Korean or Khmer) targeting a specific risk taxonomy, fed to a target LLM.

**Output**: Model-generated text response to the prompt.

## Scoring recipe

```python
def compute_asr(prompts, responses, judge):
    successes = 0
    for prompt, response in zip(prompts, responses):
        judge_input = {
            'safety_policy': policy,
            'harmful_instruction': prompt,
            'model_response': response,
            'rubric': judge_rubric
        }
        if judge.evaluate(judge_input).is_successful:
            successes += 1
    return (successes / len(prompts)) * 100
```

## Common pitfalls

- Automated judging relies on GPT-4.1 with a specific rubric; results may vary significantly if using different LLM judges or default safety filters.
- ASR is highly sensitive to the target model's language specialization (e.g., English-centric vs. Korean-specialized models show divergent vulnerability patterns).
- Adversarial baselines (GCG, TAP, AutoDAN, GPTFuzzer) use their default settings; parameter tuning can drastically change reported ASR.

## Evidence (verbatim from paper)

> For our primary evaluation metric, we use the Attack Success Rate (ASR), following standard practice in prior work*(Li et al., [2024]; Mazeika et al., [2024])*. A higher ASR values indicate greater vulnerability to a given attack.

Automated Evaluation. We utilize GPT-4.1 as an automated judge to determine the success or failure of an attack. The judge prompt integrates the safety policy, harmful instruction, model response, and scoring rubric.

## Citation

```bibtex
@misc{kim2026cage,
  title={CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generation},
  author={Kim et al. (2026)},
  year={2026},
  note={arXiv:2602.20170}
}
```

- arXiv: 2602.20170