Cybercertbench Eval

---
name: cybercertbench-eval
description: Evaluates large language models' knowledge of cybersecurity certifications across a spectrum from general IT security to specialized operational technology (OT) and vendor-specific procedural knowledge. It probes whether models can meet professional certification passing standards and identifies gaps in formal, safety-critical industrial protocols. Use when the user wants to benchmark on CyberCertBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20389
  bibtex_key: [REDACTED]
  confidence: high
---

# cybercertbench-eval

> CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge — Keppler et al. (2026) (arXiv:2604.20389, 2026)

## What this evaluates

Evaluates large language models' knowledge of cybersecurity certifications across a spectrum from general IT security to specialized operational technology (OT) and vendor-specific procedural knowledge. It probes whether models can meet professional certification passing standards and identifies gaps in formal, safety-critical industrial protocols.

## Datasets

- **CyberCertBench** — total ?; splits: test (-1); repo https://github.com/GKeppler/CyberCertBench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the number of correctly answered multiple-choice questions divided by the total number of questions. Reported as a percentage or fraction.

## Input / output format

**Input**: Multiple-choice questions (MCQA) derived from official cybersecurity certification exams, covering general IT, OT security, vendor-specific procedures, and formal standards.

**Output**: Model selects the single correct option from the provided choices.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return correct / total
```

## Common pitfalls

- General IT security benchmarks are highly saturated, with top models achieving near-perfect accuracy, making them poor differentiators for state-of-the-art systems.
- Models may demonstrate surface-level conceptual understanding of OT security but fail on formal, standard-specific knowledge (e.g., ISA/IEC 62443) and vendor-specific procedures.
- The 'PRO' benchmark filters questions based on performance of three small baseline models, which may not perfectly correlate with human-perceived difficulty or professional certification standards.

## Evidence (verbatim from paper)

> The accuracy of Qwen2.5 7B falls by 25 percentage points, while the small baseline model, Ministral 3B, falls by 44 points from 62% to just 18%, indicating that it still answers some questions correctly that the other two small models cannot.

## Citation

```bibtex
@misc{keppler2026cybercertbench,
  title={CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge},
  author={Keppler et al. (2026)},
  year={2026},
  note={arXiv:2604.20389}
}
```

- arXiv: 2604.20389