Skip to content
Back to skills

Benchmark Framework

ASecurity

Build, configure, and execute multi-dataset benchmarks (cli bench) with custom adapters, document-to-knowledge-graph pipelines, multimodal image understanding, agent harness evaluation, LLM-as-judge scoring, and TUI inspection.

  • 4 stars
  • 0 votes
  • 0 copies
  • 1 view
  • Added September 22, 2026
businesspythongobashreactnodedebugging

Works with

  • cli

Security analysis

A100/100

Scanned November 9, 2026

npx -y skills add tclatos/genai-graph --skill benchmark-framework --agent claude-code

Installs into .claude/skills of the current project.

Are you the author of Benchmark Framework?

Add the live security badge to your README. It updates with every re-scan.

Security grade badge for Benchmark Framework
[![Security: A — Skills Directory](https://www.skillsdirectory.com/api/skills/tclatos-benchmark-framework/badge)](https://www.skillsdirectory.com/skills/tclatos-benchmark-framework)

More formats (shields.io, HTML) on the badges page. Keep it an A: scan every change in CI with Pro.

Download with Pro
SKILL.md
---
name: benchmark-framework
description: Build, configure, and execute multi-dataset benchmarks (cli bench) with custom adapters, document-to-knowledge-graph pipelines, multimodal image understanding, agent harness evaluation, LLM-as-judge scoring, and TUI inspection.
tags: [benchmark, evaluation, cli, llm-as-judge, adapter, docgraph, datasets, multimodal]
version: "1.1"
author: ""
category: development
---

# Benchmark Framework Guide (`cli bench`)

The Unified Benchmark Framework (`genai_graph.bench`) provides a standardized, dataset-agnostic pipeline for evaluating autonomous agents and Document Graph retrieval systems across diverse datasets (e.g., **FinanceBench**, **OfficeQA**, **MMLongBench-Doc**).

## When to Use

- Implementing or scaffolding a new benchmark evaluation suite (e.g. SEC filings, legal contracts, technical manuals, multimodal survey reports).
- Adding or configuring benchmark profiles in `config/bench.yaml`.
- Running end-to-end evaluation runs, re-grading existing agent trajectories, or inspecting reports via CLI or TUI.
- Authoring custom `BaseBenchmarkAdapter` classes with domain-specific dataset loaders, document fetchers, and judge rubrics.
- Debugging evaluation failures, numeric equivalence issues, OCR gaps, visual chart understanding, or trajectory errors.

---

## CLI Commands Reference

Benchmark commands are registered via `genai_graph.core.commands_bench.BenchCommands`:

```bash
# 1. List configured benchmark profiles
uv run cli bench list

# 2. Run benchmark evaluation
uv run cli bench run -p mistral_glm
uv run cli bench run -p mistral_glm -q docqa_0207,docqa_0684           # run specific questions
uv run cli bench run -p mistral_glm -d doc_name_1,doc_name_2          # filter by document name
uv run cli bench run -p mistral_glm --skip fetch --skip build         # skip fetch/build when graph exists
uv run cli bench run -p mistral_glm --rerun                           # force re-running agent turns
uv run cli bench run -p mistral_glm --no-judge                        # run agent only without grading
uv run cli bench run -p mistral_glm -n 5                              # limit to first N questions

# 3. Re-grade existing runs without re-running agent
uv run cli bench grade -p mistral_glm
uv run cli bench grade -p mistral_glm --force                         # force re-grade all questions

# 4. Download dataset and documents
uv run cli bench download --docs -n 10                                # download questions + 10 PDFs
uv run cli bench download --no-docs                                   # download questions only

# 5. Generate aggregated metric summary report
uv run cli bench report -p mistral_glm
uv run cli bench report -p mistral_glm --json                         # export summary JSON

# 6. Question and trajectory inspection
uv run cli bench questions -p mistral_glm                             # tabular view
uv run cli bench questions -p mistral_glm -t                          # interactive Textual TUI
uv run cli bench questions -p mistral_glm -t --trajectory             # TUI with live trajectory replay
```

---

## 6-Stage Pipeline Architecture

```
[1. Fetch Dataset & Documents]
         │
         ▼
[2. Route-Selected Ingestion] (ingest route table -> Mistral OCR / Docling / MarkItDown)
         │
         ▼
[3. Hierarchical Graph Ingestion] (Ladybug DB + Sections + Image nodes + BM25 & FTS)
         │
         ▼
[4. Autonomous Agent Execution] (DeepAgent / React harness + Document Graph + VLM tools)
         │
         ▼
[5. LLM-as-Judge Evaluation] (Domain-specific rubric + Mafin 2.5 numeric equivalence + groundedness)
         │
         ▼
[6. Metrics & Summary Aggregation] (Accuracy, numeric match, groundedness, error breakdown)
```

1. **Fetch**: Retrieves questions from HuggingFace, CSV, or local files; caches raw PDFs.
2. **Markdownize**: Converts documents by dispatching each source through the profile's ingest route table when `ingest_routes` is set (config/ingest_routes.yaml), else the profile-configured strategy (`fast`, `medium`, `best`). For multimodal documents, `MistralOCRConverter` extracts images with xxHash32 hex names into `images_dir`, detects and converts lossless HTML tables (`table_processor.py`), and describes uncaptioned images via VLM with KV-store caching (`image_describer.py`).
3. **Build Graph**: Parses markdown into `Folder ──CONTAINS──▶ Document ──HAS_SECTION──▶ MarkdownSection (──HAS_CHUNK──▶ SectionChunk)` inside Ladybug DB, extracting section `keywords` via BAML and truncating large tables (>30 lines) in prompt context.
4. **Run Agent**: Dispatches questions to the agent harness with graph tools (`get_folder_toc`, `get_document_toc`, `get_section_content`, `search_sections`, `query_image`). Records tool calls, thinking traces, and token usage into `runs.jsonl`.
5. **Judge**: Evaluates outputs against `gold_answer` and `evidence` using domain rubrics, outputting `scores.jsonl`.
6. **Aggregate**: Generates `scores_summary.json` with accuracy tiers, numeric match rates, groundedness rates, and failure taxonomy.

---

## Multimodal & Image Understanding in Document Graph

For document understanding benchmarks requiring visual chart, diagram, or line plot interpretation:

1. **OCR Configuration (`config/markdownize.yaml`)**:
   Enable `include_image_base64: true` so the OCR converter extracts images to disk and generates metadata annotations.
2. **Embedded Section Context & Keyword Extraction**:
   Image descriptions and captions are embedded directly inside the enclosing `MarkdownSection` markdown and reflected in section `keywords`.
3. **Navigation Tools**:
   - `get_section_content(section_ids)` / `search_sections(query)`: Locate and read sections containing table data or image descriptions.
   - `query_image(image, question, model)`: Dispatch targeted visual question answering directly to a Vision-Language Model (e.g. `glm_5.3_flash@openrouter`) passing base64 image data. Enforces a per-task budget (max 3 calls) to control latency and token costs.

---

## Agent Harnessing & Dependency Prerequisites

When using `type: deep` agents with sandboxing and filesystem capabilities (e.g. DeepAgents SDK):
- Ensure the `harnessing` extra dependencies are installed:
  ```bash
  uv sync --extra harnessing
  ```
- This provides `deepagents`, `deerflow-harness`, and `agent-sandbox` needed for deep agent planning and execution.

---

## Creating a Custom Benchmark Adapter

Create an adapter by subclassing `BaseBenchmarkAdapter` from `genai_graph.bench.adapters.base`:

```python
from pathlib import Path
from genai_graph.bench.adapters.base import BaseBenchmarkAdapter, download_hf_file, load_hf_dataset_to_pandas
from genai_graph.bench.models import BenchQuestion


class MyBenchmarkAdapter(BaseBenchmarkAdapter):
    """Adapter for MyBenchmark dataset."""

    DATASET_NAME = "my_org/my_benchmark"

    def load_dataset(self, split: str | None = "test", cache_dir: Path | None = None) -> list[BenchQuestion]:
        df = load_hf_dataset_to_pandas(self.DATASET_NAME, split=split or "test", cache_dir=cache_dir)
        questions: list[BenchQuestion] = []
        for _, row in df.iterrows():
            questions.append(
                BenchQuestion(
                    id=str(row["id"]),
                    doc_name=str(row["doc_name"]),
                    doc_names=[str(row["doc_name"])],
                    question=str(row["question"]),
                    gold_answer=str(row["answer"]),
                    justification=str(row.get("justification", "")),
                    evidence=[str(row.get("evidence", ""))],
                    metadata={"split": split, "domain": row.get("domain", "")},
                )
            )
        return questions

    def fetch_document(self, doc_name: str, output_dir: Path) -> Path:
        output_dir.mkdir(parents=True, exist_ok=True)
        pdf_filename = f"{doc_name}.pdf"
        dest_path = output_dir / pdf_filename
        if dest_path.exists():
            return dest_path
        return download_hf_file(
            repo_id=self.DATASET_NAME,
            filename=f"documents/{pdf_filename}",
            repo_type="dataset",
            dest_path=dest_path,
        )

    def get_judge_rubric(self) -> str:
        return (
            "You are an expert evaluator assessing answers against ground truth.
"
            "Criteria:
"
            "1. Correctness: Evaluate factual and semantic accuracy.
"
            "2. Numeric Precision: Match financial/numeric figures accounting for rounding and scale.
"
            "3. Groundedness: Verify answer is supported by the document context.
"
            "4. Anti-Hallucination: If gold answer is 'Not answerable', verify agent properly states info is absent."
        )
```

---

## Configuring `config/docgraph.yaml` & `config/bench.yaml`

Benchmark configuration cleanly separates **DocGraph construction** (`config/docgraph.yaml`) from **benchmark execution** (`config/bench.yaml`), avoiding duplication and dead fields.

### `config/docgraph.yaml` (Document Graph Construction)

```yaml
default_profile: default

docgraph_profiles:
  default:
    description: "Ladybug Document Graph with Mistral OCR & LLM outline extraction"
    # Preferred: dispatch corpus conversion through an ingest route table
    # (config/ingest_routes.yaml); legacy profile-driven path otherwise.
    ingest_routes: default

    paths:
      sources_dir: ${paths.data_root}/pdfs
      markdown_dir: ${paths.data_root}/markdown_multi
      kg_db: ${paths.data_root}/kg/bench.db
      saved_markdown_dir: ~/OneDrive/prj/bench/markdown

    llms:
      summary: deepseek-v4-flash-0731@openrouter  # Outline structure & section summaries
      image: gemini-2.5-flash@openrouter          # VLM for uncaptioned image descriptions / query_image (or null)

    images:
      enabled: true                               # true for multimodal datasets (MMLongBench)
      describe_uncaptioned: true                  # generate descriptions for uncaptioned images >10KB
      min_size_bytes: 10240
      max_queries_per_turn: 3

    build:
      structure_strategy: auto                    # auto | algo | toc_preamble | llm_full
      generate_summaries: true                    # generate section descriptions for TOC navigation
      summary_min_tokens: 800
      context_safety_ratio: 0.9
      fts: true                                   # native BM25/FTS index over MarkdownSection
      chunk_size_tokens: 1500
      workers: 8
      skip_ocr: false
      force: false
```

### `config/bench.yaml` (Benchmark Execution & Grading)

```yaml
default_profile: default
dataset_adapter: mmlongbench.adapter.MMLongBenchAdapter

paths:
  runs: ${paths.data_root}/mmlongbench/{profile}/runs.jsonl
  scores: ${paths.data_root}/mmlongbench/{profile}/scores.jsonl
  scores_summary: ${paths.data_root}/mmlongbench/{profile}/scores_summary.json

bench_profiles:
  default:
    description: "Benchmark evaluation on MMLongBench-Doc"
    docgraph_profile: default                     # references docgraph_profiles.default in docgraph.yaml
    agent_profile: default                        # references agents.default in agents.yaml (resolves Agent LLM)

    files:
      pathspecs: ["*"]                            # glob / pathspec pattern
      docs: []
      limit: null

    runner:
      concurrency: 12
      folder_id: null
      monitoring: null

    grader:
      enabled: true
      llm: DeepSeek-V4-Pro-0813@openrouter
      concurrency: 8
```

### Path Interpolation
The `{profile}` placeholder in directory paths is automatically substituted at runtime:
- Raw PDFs: `data/pdfs/`
- Markdown files: `data/markdown_multi/`
- Knowledge Graph DB: `data/kg/bench.db`
- Execution Traces: `data/bench/{profile}/runs.jsonl`
- Scored Verdicts: `data/bench/{profile}/scores.jsonl`

---

## Grading Criteria & Anti-Hallucination Evaluation

The evaluator in `genai_graph.bench.judge` implements standardized equivalence rules:
- **Numeric Normalization**: Commas, currency symbols (`$`, `€`), percentages (`50%` == `0.5`), scale notations (`M` == `,000,000`), and standard rounding are treated as equivalent.
- **Unanswerable Questions (Anti-Hallucination Probes)**:
  - When gold answer is `Not answerable`, `Unanswerable`, or `N/A` (e.g. ~22.5% of MMLongBench-Doc questions), the agent is scored `CORRECT` if it explicitly concludes that the requested predicate (year, entity, demographic, or metric) is absent from the document.
- **Correctness Tiers**:
  - `correct`: Substantively complete and accurate.
  - `partial`: Correct reasoning or partial answers, but missing details or sub-questions.
  - `incorrect`: Contradicts gold answer, wrong numerical calculations, or hallucinated facts.
- **Failure Taxonomy**:
  - `missing_ocr_or_visual_chart`: Required chart, diagram, or table text was omitted or unreadable in OCR.
  - `retrieval_or_lookup_error`: Agent failed to locate the relevant document, section, or page.
  - `calculation_or_math_error`: Agent retrieved correct figures but made arithmetic or scaling mistakes.
  - `halted_or_empty_response`: Agent hit max iteration, recursion limits, timeout, or returned an empty response.

---

## Code Map

| Concern | Path |
|---------|------|
| Benchmark CLI Commands | `genai_graph/core/commands_bench.py` |
| Core Data Models | `genai_graph/bench/models.py` |
| Config & Path Interpolation | `genai_graph/bench/config.py` |
| Prefect Pipeline Flows | `genai_graph/bench/flows.py` |
| Agent Execution & Harness Streaming | `genai_graph/bench/runner.py` |
| LLM-as-Judge Evaluation | `genai_graph/bench/judge.py` |
| Metrics Aggregation | `genai_graph/bench/summary.py` |
| Interactive Textual TUI | `genai_graph/bench/tui.py` |
| Knowledge Graph Construction | `genai_graph/kg/document_graph/build.py` |
| Document Graph Navigation & VLM Tools | `genai_graph/kg/query/document_graph_tools.py` |
| Base Adapter Interface & Utilities | `genai_graph/bench/adapters/base.py` |

## References

- [Unified Benchmark Framework Docs](docs/benchmark_framework.md)
- [Empirical Evaluation & Studies](docs/benchmarks_financebench_officeqa.md)
- [Document Graph Architecture](docs/document-graph.md)

Attribution

Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.

View all in business →

Comments

Loading comments…

Newsletter

Get the weekly list of safe new skills, and the ones we pulled.