Benchie Fl Eval

---
name: benchie-fl-eval
description: Evaluates Open Information Extraction (OIE) systems on their ability to extract fact-based triples from text. It uses a conservative exact-matching function with synset-based clustering to penalize non-informative copies and reward precise fact extraction, while also measuring correlation with downstream QA and knowledge base tasks. Use when the user wants to benchmark on BenchIE^FL, or asks about evaluating this task. Reports exact-match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.16860
  bibtex_key: [REDACTED]
  confidence: high
---

# benchie-fl-eval

> $\textit{BenchIE}^{FL}$ : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark — Lamarche et al. (2024) (arXiv:2407.16860, 2024)

## What this evaluates

Evaluates Open Information Extraction (OIE) systems on their ability to extract fact-based triples from text. It uses a conservative exact-matching function with synset-based clustering to penalize non-informative copies and reward precise fact extraction, while also measuring correlation with downstream QA and knowledge base tasks.

## Datasets

- **BenchIE^FL** — total ?; splits: test (-1); repo https://github.com/rali-udem/benchie_fl.git

## Metrics

- `exact-match` **(primary)** — range: [0, 1]
  - 1 if a predicted triple exactly matches a gold triple or its synset cluster, else 0. Averaged over instances.
- `Mean Reciprocal Rank` — range: [0, 1]
  - Average of 1/rank for the first correct answer across queries.

## Input / output format

**Input**: Raw text passages or sentences.

**Output**: Extracted fact-based triples (subject, relation, object).

## Scoring recipe

```python
def compute_score(predictions, gold):
    correct = 0
    for pred in predictions:
        if pred in gold or synset_match(pred, gold):
            correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Neural OIE models tend to copy large chunks of input text, producing non-informative tuples that inflate scores on older benchmarks.
- Optimizing extractors for specific benchmarks does not lead to significant performance gains but can skew system rankings.
- Lexical variation requires synset-based clustering for matching; strict string matching causes false negatives.

## Evidence (verbatim from paper)

> We then use QUEST to construct answers and report the scores measured by the Mean Reciprocal Rank.

## Citation

```bibtex
@misc{lamarche2024benchiefl,
  title={$\textit{BenchIE}^{FL}$ : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark},
  author={Lamarche et al. (2024)},
  year={2024},
  note={arXiv:2407.16860}
}
```

- arXiv: 2407.16860