Leaderboard Triple Extraction Eval

---
name: leaderboard-triple-extraction-eval
description: Evaluates a model's ability to verify whether a candidate (Task, Dataset, Metric) triple is actually used or mentioned in a specific AI research paper. The task is framed as a natural language inference problem where the model must distinguish between valid triples and randomly sampled invalid ones. Use when the user wants to benchmark on AI Research Paper Collection, or asks about evaluating this task. Reports micro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.13089
  bibtex_key: kabongo2021automated
  confidence: high
---

# leaderboard-triple-extraction-eval

> Automated Mining of Leaderboards for Empirical AI Research — Kabongo et al. (2021) (arXiv:2109.13089, 2021)

## What this evaluates

Evaluates a model's ability to verify whether a candidate (Task, Dataset, Metric) triple is actually used or mentioned in a specific AI research paper. The task is framed as a natural language inference problem where the model must distinguish between valid triples and randomly sampled invalid ones.

## Datasets

- **AI Research Paper Collection** — total 4500; splits: train (-1), val (-1), test (-1)

## Metrics

- `micro-F1` **(primary)** — range: [0, 1]
  - Computes global true positives, false positives, and false negatives across all classes before calculating precision and recall. F1 is the harmonic mean of precision and recall.
- `macro-F1` — range: [0, 1]
  - Calculates precision and recall for each class independently, then takes their unweighted mean. F1 is the harmonic mean of the averaged precision and recall.

## Input / output format

**Input**: A candidate triple (t, d, m) concatenated with the paper's context feature representation (p_DocTAET).

**Output**: Binary classification label: true or false.

## Scoring recipe

```python
def compute_f1(y_true, y_pred, average='micro'):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Negative samples are generated by randomly selecting triples from *other* papers rather than negating the text, which may artificially inflate performance compared to real-world verification tasks.
- The task is multi-label and multi-class; macro-F1 will heavily penalize poor performance on rare triples, while micro-F1 is dominated by frequent ones.
- The exact construction of the DocTAET context feature (p_DocTAET) is not fully detailed in the task definition, requiring careful implementation of the underlying feature extraction pipeline.

## Evidence (verbatim from paper)

> Through a large-scale empirical evaluation on over 4,500 AI research papers, the approach achieves 93.0% micro-F1 and 92.8% macro-F1... The inference data instance, then is (c;[(t,d,m),p_DocTAET]) where c ∈ {true,false} is the inference label. Thus, specifically, our LEADERBOARD extraction problem is formulated as a natural language inference task between the DocTAET context feature p_DocTAET and the (t,d,m) triple annotation. (t,d,m) is true if it is among the paper's TDM-triples, otherwise false.

## Citation

```bibtex
@misc{kabongo2021automated,
  title={Automated Mining of Leaderboards for Empirical AI Research},
  author={Kabongo et al. (2021)},
  year={2021},
  note={arXiv:2109.13089}
}
```

- arXiv: 2109.13089