Ir Triplet Eval

---
name: ir-triplet-eval
description: Evaluates the model's ability to capture semantic similarity between paragraphs for information retrieval. It tests whether fixed-length vector representations can effectively distinguish query-related documents from irrelevant ones using a triplet ranking protocol. Use when the user wants to benchmark on Information Retrieval (Paragraph Vectors), or asks about evaluating this task. Reports error rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1405.4053
  bibtex_key: le2014distributed
  confidence: high
---

# ir-triplet-eval

> Distributed Representations of Sentences and Documents — Quoc V. Le et al. (2014) (arXiv:1405.4053, 2014)

## What this evaluates

Evaluates the model's ability to capture semantic similarity between paragraphs for information retrieval. It tests whether fixed-length vector representations can effectively distinguish query-related documents from irrelevant ones using a triplet ranking protocol.

## Datasets

- **Information Retrieval (Paragraph Vectors)** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `error rate` **(primary)** — range: percent
  - Percentage of triplets where the distance between the two query-related paragraphs is not smaller than the distance between the first paragraph and the random paragraph.

## Input / output format

**Input**: Triplets of paragraphs: two paragraphs retrieved for the same query, and one randomly sampled paragraph from a different query.

**Output**: Binary decision indicating whether the first two paragraphs belong to the same query (i.e., distance(d1, d2) < distance(d1, d3)).

## Scoring recipe

```python
d12 = distance(paragraph1, paragraph2)
d13 = distance(paragraph1, paragraph3)
error = 1 if d12 >= d13 else 0
error_rate = sum(errors) / total_triplets
```

## Common pitfalls

- The dataset is custom-derived from search engine snippets and is not publicly released.
- Baselines use TF-IDF weighting, and the 'Weighted Bag-of-bigrams' baseline involves learning a linear weighting matrix.
- Distance metric is not explicitly specified in the text, but the protocol relies on relative ordering of distances.

## Evidence (verbatim from paper)

> The results show that Paragraph Vector works well and gives a 32% relative improvement in terms of error rate. The fact that the paragraph vector method significantly outperforms bag of words and bigrams suggests that our proposed method is useful for capturing the semantics of the input text.

## Citation

```bibtex
@misc{le2014distributed,
  title={Distributed Representations of Sentences and Documents},
  author={Quoc V. Le et al. (2014)},
  year={2014},
  note={arXiv:1405.4053}
}
```

- arXiv: 1405.4053