Flask Eval

---
name: flask-eval
description: Evaluates LLMs on fine-grained alignment capabilities by decomposing instruction-following performance into 12 sub-skills across four domains (Logical Thinking, Background Knowledge, Problem Handling, User Alignment). It measures how well models adhere to specific quality criteria like factuality, logical robustness, and harmlessness on a per-instance basis. Use when the user wants to benchmark on FLASK, or asks about evaluating this task. Reports FLASK skill score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.10928
  bibtex_key: ye2023flask
  confidence: high
---

# flask-eval

> FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets — Ye et al. (2023) (arXiv:2307.10928, 2023)

## What this evaluates

Evaluates LLMs on fine-grained alignment capabilities by decomposing instruction-following performance into 12 sub-skills across four domains (Logical Thinking, Background Knowledge, Problem Handling, User Alignment). It measures how well models adhere to specific quality criteria like factuality, logical robustness, and harmlessness on a per-instance basis.

## Datasets

- **FLASK** — total ?; splits: test (-1), hard (-1)

## Metrics

- `FLASK skill score` **(primary)** — range: [1, 5]
  - Average of per-instance scores across 12 alignment sub-skills (Logical Robustness, Logical Correctness, Logical Efficiency, Factuality, Commonsense, Comprehension, Insightfulness, Completeness, Metacognition, Readability, Conciseness, Harmlessness). Each skill is rated on a 1–5 scale by GPT-4 acting as the evaluator, then averaged across instances.

## Input / output format

**Input**: Instruction-based text prompt requiring a model response.

**Output**: Model-generated text response.

## Scoring recipe

```python
def compute_flask_score(dataset, model, evaluator='GPT-4'):
    skill_scores = {skill: [] for skill in SKILLS}
    for instance in dataset:
        response = model.generate(instance.prompt)
        for skill in instance.skills:
            score = evaluator.score(response, rubric=skill)  # 1-5 scale
            skill_scores[skill].append(score)
    return {skill: mean(scores) for skill, scores in skill_scores.items()}
```

## Common pitfalls

- GPT-4 as evaluator may exhibit self-preference or style bias, inflating scores for models with similar output patterns.
- Scores on FLASK-HARD can drop by up to 50% compared to the base set, making difficulty calibration critical.
- Averaging across 12 sub-skills can mask severe weaknesses in specific alignment dimensions like Logical Robustness.

## Evidence (verbatim from paper)

> We use GPT-4 as the evaluator (Eval LM) for model-based evaluation. Table 9: Comparison of open-source and proprietary models on the whole FLASK evaluation set. The best performance is shown in bold.

## Citation

```bibtex
@misc{ye2023flask,
  title={FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets},
  author={Ye et al. (2023)},
  year={2023},
  note={arXiv:2307.10928}
}
```

- arXiv: 2307.10928