Tact Eval

---
name: tact-eval
description: Evaluates an agent's ability to perform logical deduction and multi-hop reasoning over long, noise-rich unstructured text without pre-defined schemas or tables. It probes the model's capacity to actively synthesize scattered evidence and filter out irrelevant distractors to arrive at a correct decision. Use when the user wants to benchmark on TACT, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14930
  bibtex_key: lv2026ieascache
  confidence: high
---

# tact-eval

> IE as Cache: Information Extraction Enhanced Agentic Reasoning — Hang Lv et al. (2026) (arXiv:2604.14930, 2026)

## What this evaluates

Evaluates an agent's ability to perform logical deduction and multi-hop reasoning over long, noise-rich unstructured text without pre-defined schemas or tables. It probes the model's capacity to actively synthesize scattered evidence and filter out irrelevant distractors to arrive at a correct decision.

## Datasets

- **TACT** — total ?; splits: test (-1)

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted answer exactly matches the ground-truth answer, 0 otherwise. Measures whether the produced answer satisfies the target decision or constraint.

## Input / output format

**Input**: Long-form unstructured text containing logical reasoning problems, paired with a query or instruction requiring deduction.

**Output**: A final answer string that satisfies the target decision or constraint.

## Scoring recipe

```python
def score_em(pred, gold):
    return 1.0 if pred.strip().lower() == gold.strip().lower() else 0.0

# Average over test set
em_scores = [score_em(p, g) for p, g in zip(predictions, gold_answers)]
final_metric = sum(em_scores) / len(em_scores)
```

## Common pitfalls

- Models often fail to aggregate scattered evidence correctly when contexts are filled with irrelevant distractors.
- Exact match requires strict formatting; minor phrasing variations or extra punctuation will result in a score of 0.
- Static extraction baselines struggle because they cannot iteratively update their information state during multi-hop reasoning.

## Evidence (verbatim from paper)

> For TACT and Calendar Scheduling, we report Exact Match (EM), which directly measures whether the produced answer satisfies the target decision/constraints.

## Citation

```bibtex
@misc{lv2026ieascache,
  title={IE as Cache: Information Extraction Enhanced Agentic Reasoning},
  author={Hang Lv et al. (2026)},
  year={2026},
  note={arXiv:2604.14930}
}
```

- arXiv: 2604.14930