Benchmarking

---
name: benchmarking
description: "Guides Logparser benchmark runs, dataset evaluation, and
  accuracy/F1 comparison workflows."
disable-model-invocation: true
metadata:
  disco-role: operating
license: NOASSERTION
---

# Benchmarking

Use this sub-skill for any workflow that compares parser output against ground
truth or runs a Loghub-style benchmark harness.

## Include here

- Evaluating `*_structured.csv` against ground truth.
- Interpreting `F1_measure`, `Accuracy`, parsing accuracy, grouping accuracy,
  and template accuracy.
- Adapting the parser-specific `benchmark.py` files.
- Reasoning about `data/loghub_2k/` and `data/loghub_2k_corrected/` layouts.

## Exclude from here

- Choosing or tuning a parser for a raw log file; use `../parsing/SKILL.md`.
- Import shims, compilers, APIs, and other special parser issues; use
  `../specialized-parsers/SKILL.md`.

## Read these references

- `references/benchmark-workflows.md` for the benchmark flow and dataset notes.
- `references/evaluator-reference.md` for the metric helper and output meaning.
- `references/troubleshooting.md` for result-path, dataset, and metric issues.

## Run this script

- `scripts/evaluate_csvs.py` — quick metric helper for a ground-truth / parsed
  CSV pair.

## When to route here

Choose this sub-skill when the request says things like:

- "benchmark Drain"
- "compare parsers"
- "compute F1/accuracy"
- "evaluate the structured CSV"
- "run the Loghub benchmark"
- "inspect the benchmark output"

## Working notes

- The benchmark flow usually reuses the same parser configuration as the parsing
  flow, so read the parsing sub-skill if you need to adjust the parser first.
- The benchmark scripts sometimes assume a dataset directory layout; confirm the
  input paths before running a large job.