Romeo Vuln Detection Eval

---
name: romeo-vuln-detection-eval
description: This benchmark evaluates binary vulnerability detection capabilities on assembly language representations of C/C++ functions. It probes whether models can identify security flaws (e.g., buffer overflows, integer overflows) by analyzing machine code semantics and call graph context. Use when the user wants to benchmark on ROMEO, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.06623
  bibtex_key: brust2021romeo
  confidence: high
---

# romeo-vuln-detection-eval

> ROMEO: Exploring Juliet through the Lens of Assembly Language — Brust et al. (2021) (arXiv:2112.06623, 2021)

## What this evaluates

This benchmark evaluates binary vulnerability detection capabilities on assembly language representations of C/C++ functions. It probes whether models can identify security flaws (e.g., buffer overflows, integer overflows) by analyzing machine code semantics and call graph context.

## Datasets

- **ROMEO** — total ?; splits: train (-1), val (-1), test (16764)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly classified instances (vulnerable vs. not vulnerable) out of the total test set.
- `F1 score` — range: percent
  - Harmonic mean of precision and recall for the binary vulnerability classification task.

## Input / output format

**Input**: Tokenized assembly language instructions representing a single function, optionally augmented with call graph context from calling/called functions. Max 512 tokens per instance.

**Output**: Binary label: 'vulnerable' or 'not vulnerable'.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    acc = sum(p == g for p, g in zip(preds, gold)) / len(gold) * 100
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return {'accuracy': acc, 'f1': f1 * 100}
```

## Common pitfalls

- Comparing results across different evaluation granularities (e.g., program slices vs. whole functions) without normalization.
- Assuming random train/val/test splits; the dataset is split proportionally by CWE and flow variant to prevent data leakage, which limits cross-CWE generalization testing.
- Ignoring the context variant; performance drops significantly (~6.7% accuracy) when call graph context is removed, so both variants must be reported.

## Evidence (verbatim from paper)

> With context, the overall accuracy on the held-out test set is 96.9% and the overall F1 score is 94.0%. Without context, the accuracy and F1 score are 90.2% and 81.9%, respectively.

## Citation

```bibtex
@misc{brust2021romeo,
  title={ROMEO: Exploring Juliet through the Lens of Assembly Language},
  author={Brust et al. (2021)},
  year={2021},
  note={arXiv:2112.06623}
}
```

- arXiv: 2112.06623