Flowbench Eval

---
name: flowbench-eval
description: Evaluates computational workflow anomaly detection by benchmarking models on detecting injected CPU and HDD performance anomalies in distributed workflow execution logs. It tests the ability of tabular, graph, and text-based methods to identify anomalous nodes within Directed Acyclic Graph (DAG) workflow executions. Use when the user wants to benchmark on Flow-Bench, or asks about evaluating this task. Reports ROC-AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.09930
  bibtex_key: papadimitriou2023flowbench
  confidence: high
---

# flowbench-eval

> Flow-Bench: A Dataset for Computational Workflow Anomaly Detection — Papadimitriou et al. (2023) (arXiv:2306.09930, 2023)

## What this evaluates

Evaluates computational workflow anomaly detection by benchmarking models on detecting injected CPU and HDD performance anomalies in distributed workflow execution logs. It tests the ability of tabular, graph, and text-based methods to identify anomalous nodes within Directed Acyclic Graph (DAG) workflow executions.

## Datasets

- **Flow-Bench** — total 6352; splits: test (-1); repo https://github.com/PeSeiDon-Workflows/flowbench

## Metrics

- `ROC-AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `AP` — range: [0, 1]
  - Average Precision, summarizing the precision-recall curve across all classification thresholds.
- `Accuracy` — range: [0, 1]
  - Ratio of correctly classified instances (normal and anomalous) to the total number of instances.
- `Recall@k` — range: [0, 1]
  - Proportion of actual anomalous nodes retrieved among the top-k predicted anomalous nodes.

## Input / output format

**Input**: Tabular CSV logs per DAG execution containing job-level execution metrics, infrastructure statistics, and dependency information. The dataset also provides graph (DAG structure) and text (template-generated sentences) representations.

**Output**: Anomaly label per node: normal, cpu_2, cpu_3, cpu_4, hdd_5, or hdd_10. For binary detection, a flag indicating anomalous vs normal.

## Scoring recipe

```python
def compute_rocauc(y_true, y_scores):
    # y_true: binary node labels (0=normal, 1=anomalous)
    # y_scores: predicted anomaly scores
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

def compute_ap(y_true, y_scores):
    precision, recall, _ = precision_recall_curve(y_true, y_scores)
    return auc(recall, precision)
```

## Common pitfalls

- Anomalies are injected at the per-node level, not the whole-DAG level, so evaluation must score or aggregate per node rather than per workflow execution.
- The dataset is highly imbalanced with a vast majority of normal nodes, making accuracy misleading without precision/recall metrics.
- Graph, tabular, and text representations require distinct preprocessing pipelines; mixing them without alignment causes evaluation errors.

## Evidence (verbatim from paper)

> Table 2: Unsupervised Model Examples 

<table><tr><td rowspan=2>Workflow</td><td colspan=5>GAE (PyGOD)</td><td colspan=5>GMM (PyOD)</td></tr><tr><td>Accuracy</td><td>AP</td><td>ROC-AUC</td><td>Recall@k</td><td>Train time(sec.)</td><td>Accuracy</td><td>AP</td><td>ROC-AUC</td><td>Recall@k</td><td>Train time(sec.)</td></tr></table>

## Citation

```bibtex
@misc{papadimitriou2023flowbench,
  title={Flow-Bench: A Dataset for Computational Workflow Anomaly Detection},
  author={Papadimitriou et al. (2023)},
  year={2023},
  note={arXiv:2306.09930}
}
```

- arXiv: 2306.09930