Secure Eval

---
name: secure-eval
description: Evaluates large language models' capabilities in cybersecurity, specifically focusing on Industrial Control Systems (ICS). It probes knowledge extraction, vulnerability understanding, out-of-distribution reasoning, and risk evaluation using real-world threat intelligence sources. Use when the user wants to benchmark on MAET, CWET, KCV, VOOD, RERT, CPST, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.20441
  bibtex_key: [REDACTED]
  confidence: high
---

# secure-eval

> SECURE: Benchmarking Large Language Models for Cybersecurity — Bhusal et al. (2024) (arXiv:2405.20441, 2024)

## What this evaluates

Evaluates large language models' capabilities in cybersecurity, specifically focusing on Industrial Control Systems (ICS). It probes knowledge extraction, vulnerability understanding, out-of-distribution reasoning, and risk evaluation using real-world threat intelligence sources.

## Datasets

- **MAET** — total ?; splits: test (-1); repo https://github.com/aiforsec/SECURE
- **CWET** — total ?; splits: test (-1); repo https://github.com/aiforsec/SECURE
- **KCV** — total ?; splits: test (-1); repo https://github.com/aiforsec/SECURE
- **VOOD** — total ?; splits: test (-1); repo https://github.com/aiforsec/SECURE
- **RERT** — total ?; splits: test (-1); repo https://github.com/aiforsec/SECURE
- **CPST** — total ?; splits: test (-1); repo https://github.com/aiforsec/SECURE

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted answers out of the total number of instances. Calculated as correct predictions divided by total predictions.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence-based F1 score measuring the overlap between generated text and reference text.
- `MAD` — range: other
  - Mean Absolute Deviation, calculated as the average of the absolute differences between predicted and ground-truth values.

## Input / output format

**Input**: Task-specific prompts containing questions and multiple-choice options (A-D) or embedded knowledge snippets. Prompts are constrained to a maximum of 5000 words to fit model context windows.

**Output**: For MCQ tasks (MAET, CWET, KCV, VOOD): a single uppercase letter (A, B, C, D, or X). For generation tasks (RERT, CPST): free-form text or numerical predictions.

## Scoring recipe

```python
def evaluate(predictions, golds, task_type):
    if task_type in ['MAET', 'CWET', 'KCV', 'VOOD']:
        pred_letters = [re.search(r'[A-DX]', p).group(0) for p in predictions]
        return sum(1 for p, g in zip(pred_letters, golds) if p == g) / len(golds)
    elif task_type == 'RERT':
        return rouge_l_score(predictions, golds)
    elif task_type == 'CPST':
        return mean(abs(float(p) - float(g)) for p, g in zip(predictions, golds))
```

## Common pitfalls

- Models frequently ignore the strict 'return only the letter' instruction and append explanations, requiring manual post-processing for accurate scoring.
- The KCV task selectively includes knowledge snippets to stay under the 5000-word limit, which may introduce sampling bias compared to full-context evaluation.
- Temperature is fixed at 0.7 by default, which can introduce non-deterministic outputs that complicate reproducibility without explicit seeding.

## Evidence (verbatim from paper)

> We employ a range of evaluation metrics tailored to the specific nature of each task within our benchmarking framework: accuracy for MAET, CWET, VOOD and KCV, ROGUE-L [54] for RERT and mean absolute deviation (MAD) for CPST.

## Citation

```bibtex
@misc{bhusal2024secure,
  title={SECURE: Benchmarking Large Language Models for Cybersecurity},
  author={Bhusal et al. (2024)},
  year={2024},
  note={arXiv:2405.20441}
}
```

- arXiv: 2405.20441