Secbench Eval

---
name: secbench-eval
description: Evaluates large language models' cybersecurity knowledge retention and logical reasoning capabilities across multiple subdomains, languages, and difficulty levels using multiple-choice and short-answer questions. Use when the user wants to benchmark on SecBench, or asks about evaluating this task. Reports correctness percentage.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.20787
  bibtex_key: jing2024secbench
  confidence: high
---

# secbench-eval

> SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity — Jing et al. (2024) (arXiv:2412.20787, 2024)

## What this evaluates

Evaluates large language models' cybersecurity knowledge retention and logical reasoning capabilities across multiple subdomains, languages, and difficulty levels using multiple-choice and short-answer questions.

## Datasets

- **SecBench** — total 47910; splits: test (47910)

## Metrics

- `correctness percentage` **(primary)** — range: [0, 100] percent
  - For MCQs: (number of correctly answered questions / total MCQs) × 100. For SAQs: average score assigned by the GPT-4o-mini grading agent, normalized to a 0–100 scale based on alignment with ground truth.

## Input / output format

**Input**: Multiple-choice question with four options, or open-ended short-answer question in a cybersecurity domain.

**Output**: For MCQ: selected option letter or text. For SAQ: generated natural language answer.

## Scoring recipe

```python
def evaluate_secbench(predictions, golds, q_type='mcq'):
    if q_type == 'mcq':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return (correct / len(golds)) * 100
    else:
        scores = [grade_with_gpt4o_mini(pred, gold) for pred, gold in zip(predictions, golds)]
        return sum(scores) / len(scores)
```

## Common pitfalls

- SAQ evaluation relies on an LLM-based grading agent (GPT-4o-mini) rather than human annotators, which may introduce scoring bias or inconsistency.
- The dataset spans multiple languages (Chinese/English) and nine cybersecurity subdomains; reporting only the overall average masks significant performance disparities across categories.
- MCQ scoring uses strict exact-match correctness, ignoring partial credit or reasoning quality.

## Evidence (verbatim from paper)

> Table [1] presents the benchmarking results for the 44,823 MCQs. The values in each cell represent the correctness percentage for the corresponding category. Table [2] presents the benchmarking results for the 3,087 short-answer questions (SAQs). The values in each cell represent the average score, graded by the grading agent, on a percentage scale for the corresponding category.

## Citation

```bibtex
@misc{jing2024secbench,
  title={SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity},
  author={Jing et al. (2024)},
  year={2024},
  note={arXiv:2412.20787}
}
```

- arXiv: 2412.20787