Trace Classification Eval

---
name: trace-classification-eval
description: Evaluates a model's ability to classify OpenTelemetry workflow traces as benign, suspicious, or malicious, and assesses its knowledge of cybersecurity frameworks via multiple-choice questions. Use when the user wants to benchmark on OpenTelemetry Workflow Traces, or asks about evaluating this task. Reports Overall Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.00848
  bibtex_key: [REDACTED]
  confidence: high
---

# trace-classification-eval

> Temporal Attack Pattern Detection in Multi-Agent AI Workflows: An Open Framework for Training Trace-Based Security Models — Del Rosario (2025) (arXiv:2601.00848, 2025)

## What this evaluates

Evaluates a model's ability to classify OpenTelemetry workflow traces as benign, suspicious, or malicious, and assesses its knowledge of cybersecurity frameworks via multiple-choice questions.

## Datasets

- **OpenTelemetry Workflow Traces** — total 30; splits: test (30)

## Metrics

- `Overall Accuracy` **(primary)** — range: percent
  - Percentage of correctly classified traces out of the total 30 traces. Predictions labeled 'SUSPICIOUS' are treated as incorrect for binary classification metrics.

## Input / output format

**Input**: OpenTelemetry workflow traces representing sequences of agent actions (e.g., query_db, http_request, log_metrics).

**Output**: A classification label (BENIGN, SUSPICIOUS, or MALICIOUS) followed by a textual reasoning explanation.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = correct / len(gold_labels)
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'MALICIOUS' and g == 'MALICIOUS')
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'MALICIOUS' and g == 'BENIGN')
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p != 'MALICIOUS' and g == 'MALICIOUS')
    tn = sum(1 for p, g in zip(predictions, gold_labels) if p == 'BENIGN' and g == 'BENIGN')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return accuracy, precision, recall, f1
```

## Common pitfalls

- 'SUSPICIOUS' predictions are collapsed to incorrect for binary metrics, artificially lowering reported accuracy compared to a 3-class setup.
- Extremely small test set (n=30) yields wide confidence intervals (±18%), limiting statistical generalizability.
- Severe training data imbalance (90% attack-focused) causes high false positive rates that prompt engineering cannot fix.

## Evidence (verbatim from paper)

> We evaluated V4 on 30 traces (15 malicious, 15 benign) with manually labeled ground truth. Model configuration: Q4_K_M quantized (4.92 GB), temperature=0.1, top-p=0.95, standardized prompt requesting BENIGN/SUSPICIOUS/MALICIOUS classification with reasoning. ... Overall Accuracy 30.0% (9/30) | Model correctly classified 30% of all traces ... True Positive Rate 60.0% (9/15) ... False Positive Rate 66.7% (10/15) ... Precision 47.4% ... Recall 60.0% ... F1 Score 0.529

## Citation

```bibtex
@misc{delrosario2025temporal,
  title={Temporal Attack Pattern Detection in Multi-Agent AI Workflows: An Open Framework for Training Trace-Based Security Models},
  author={Del Rosario (2025)},
  year={2025},
  note={arXiv:2601.00848}
}
```

- arXiv: 2601.00848