Orgforge Eval

---
name: orgforge-eval
description: This benchmark evaluates RAG and retrieval-augmented agents on synthetic corporate corpora by testing their ability to retrieve artifacts, reason over causal and temporal chains, and detect knowledge gaps. It probes multi-hop reasoning, temporal knowledge-state tracking, and absence-of-evidence detection across structured enterprise artifacts like Slack, JIRA, and emails. Use when the user wants to benchmark on OrgForge Synthetic Corporate Corpus, or asks about evaluating this task. Reports OrgForgeScorer.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.14997
  bibtex_key: [REDACTED]
  confidence: high
---

# orgforge-eval

> OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora — Flynt (2026) (arXiv:2603.14997, 2026)

## What this evaluates

This benchmark evaluates RAG and retrieval-augmented agents on synthetic corporate corpora by testing their ability to retrieve artifacts, reason over causal and temporal chains, and detect knowledge gaps. It probes multi-hop reasoning, temporal knowledge-state tracking, and absence-of-evidence detection across structured enterprise artifacts like Slack, JIRA, and emails.

## Datasets

- **OrgForge Synthetic Corporate Corpus** — total 1162; splits: test (83); repo https://github.com/aeriesec/orgforge

## Metrics

- `OrgForgeScorer` **(primary)** — range: [0, 1]
  - A weighted composite score: min(1.0, 0.80 * s_primary + 0.20 * s_evidence). s_primary is a type-specific correctness score (0-1) based on exact matches, proximity bonuses, or Jaccard overlap. s_evidence is the Jaccard overlap between the ground-truth evidence chain and the artifact IDs retrieved by the agent.
- `MRR@10` — range: [0, 1]
  - Mean Reciprocal Rank of the first ground-truth artifact ID found in the top-10 retrieved documents.
- `Recall@10` — range: [0, 1]
  - Fraction of ground-truth evidence chain artifact IDs successfully retrieved within the top-10 results.

## Input / output format

**Input**: A natural-language question (typed as RETRIEVAL, CAUSAL, TEMPORAL, GAP_DETECTION, ROUTING, PLAN, ESCALATION, or KNOWLEDGE_GAP) and a retrieval corpus of synthetic corporate artifacts (Slack messages, JIRA tickets, emails, logs, Confluence pages).

**Output**: A structured answer specific to the question type (e.g., artifact ID + timestamp, boolean, actor name, or ordered actor list) and an optional list of retrieved artifact IDs used as evidence.

## Scoring recipe

```python
def compute_score(q_type, pred_ans, pred_evid, gold_ans, gold_evid):
    if q_type == 'RETRIEVAL':
        s_p = 1.0 if pred_ans.id == gold_ans.id else 0.0
        if abs(pred_ans.ts - gold_ans.ts) <= 30: s_p += 0.1
    elif q_type == 'CAUSAL':
        s_p = 1.0 if (pred_ans.id == gold_ans.id and pred_ans.type == gold_ans.type) else 0.5 if pred_ans.id == gold_ans.id else 0.0
    elif q_type == 'TEMPORAL':
        s_p = 1.0 if pred_ans == gold_ans else 0.0
        if abs(pred_ans.day - gold_ans.day) <= 1: s_p += 0.1
    elif q_type == 'GAP_DETECTION':
        s_p = 0.6 * (1.0 if pred_ans == gold_ans else 0.0) + 0.4 * jaccard(pred_ans.downstream, gold_ans.downstream)
    elif q_type == 'KNOWLEDGE_GAP':
        s_p = jaccard(pred_ans.gap_areas, gold_ans.gap_areas)
    else:
        s_p = 1.0 if pred_ans == gold_ans else 0.0
    s_e = jaccard(pred_evid, gold_evid)
    return min(1.0, 0.80 * s_p + 0.20 * s_e)
```

## Common pitfalls

- The evidence retrieval weight (0.20) is intentionally secondary; a lucky retrieval cannot mask an incorrect primary answer.
- TEMPORAL and GAP_DETECTION questions require boolean/absence reasoning rather than standard document retrieval, so baselines report only evidence recall, not answer correctness.
- PLAN and ESCALATION questions inherently score zero for keyword/semantic baselines due to templated prose and multi-hop actor-chain reasoning requirements.

## Evidence (verbatim from paper)

> OrgForgeScorer dispatches each question to a type-specific scorer. The final score is a weighted combination of primary answer correctness and evidence retrieval quality: score(q,a)=min(1.0, 0.80·s_primary(q,a)+0.20·s_evidence(q,a)) where s_primary∈[0,1] measures answer correctness and s_evidence is the Jaccard overlap between the ground-truth evidence chain and the artifact IDs the agent reports retrieving.

## Citation

```bibtex
@misc{flynt2026orgforge,
  title={OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora},
  author={Flynt (2026)},
  year={2026},
  note={arXiv:2603.14997}
}
```

- arXiv: 2603.14997