Malware Analysis Eval

---
name: malware-analysis-eval
description: This benchmark evaluates an AI system's ability to analyze low-level process execution logs and identify malicious signals from malware detonations. It probes structured data parsing, security event correlation, and malware family classification capabilities. Use when the user wants to benchmark on CyberSOCEval Malware Analysis, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.20166
  bibtex_key: [REDACTED]
  confidence: high
---

# malware-analysis-eval

> CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning — Deason et al. (2025) (arXiv:2509.20166, 2025)

## What this evaluates

This benchmark evaluates an AI system's ability to analyze low-level process execution logs and identify malicious signals from malware detonations. It probes structured data parsing, security event correlation, and malware family classification capabilities.

## Datasets

- **CyberSOCEval Malware Analysis** — total 609; splits: test (609); repo https://github.com/CrowdStrike/cybersoceval_data

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The share of questions for which the system selects all correct options and only the correct options. It is a strict exact-match metric for multi-label multiple-choice questions.

## Input / output format

**Input**: JSON-formatted system log data from malware detonations, including process execution details, extracted files, and static signature detections.

**Output**: A set of selected multiple-choice options (up to 10 possible answers per question).

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if set(pred) == set(gold):
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- Partial credit is not awarded; models must select exactly the correct set of options to receive a point for a question.
- The input consists of structured JSON logs rather than raw binary files or plain text, requiring the model to parse and interpret system event data.

## Evidence (verbatim from paper)

> Evaluation is based on accuracy: the share of questions for which the system selects all correct options and only the correct options.

## Citation

```bibtex
@misc{deason2025cybersoceval,
  title={CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning},
  author={Deason et al. (2025)},
  year={2025},
  note={arXiv:2509.20166}
}
```

- arXiv: 2509.20166