Threat Intelligence Eval

---
name: threat-intelligence-eval
description: This benchmark evaluates an AI system's ability to extract actionable insights from threat intelligence reports and perform security reasoning. It probes multi-document comprehension, attack chain reconstruction, and MITRE ATT&CK framework mapping capabilities. Use when the user wants to benchmark on CyberSOCEval Threat Intelligence Reasoning, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.20166
  bibtex_key: deason2025cybersoceval
  confidence: high
---

# threat-intelligence-eval

> CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning — Deason et al. (2025) (arXiv:2509.20166, 2025)

## What this evaluates

This benchmark evaluates an AI system's ability to extract actionable insights from threat intelligence reports and perform security reasoning. It probes multi-document comprehension, attack chain reconstruction, and MITRE ATT&CK framework mapping capabilities.

## Datasets

- **CyberSOCEval Threat Intelligence Reasoning** — total 588; splits: test (588); repo https://github.com/CrowdStrike/cybersoceval_data

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The share of questions for which the system selects all correct options and only the correct options. It is a strict exact-match metric for multi-label multiple-choice questions.

## Input / output format

**Input**: Multiple images representing a threat intelligence report, with one image provided per report page.

**Output**: A set of selected multiple-choice options (up to 6 possible answers per question).

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if set(pred) == set(gold):
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- Partial credit is not awarded; models must select exactly the correct set of options to receive a point for a question.
- The input is provided as images of report pages rather than extractable text, requiring vision-language capabilities instead of pure text processing.

## Evidence (verbatim from paper)

> Evaluation is based on accuracy: the share of questions for which the system selects all correct options and only the correct options.

## Citation

```bibtex
@misc{deason2025cybersoceval,
  title={CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning},
  author={Deason et al. (2025)},
  year={2025},
  note={arXiv:2509.20166}
}
```

- arXiv: 2509.20166