Contextual Ir Eval

---
name: contextual-ir-eval
description: Evaluates information retrieval systems by measuring system-level performance metrics (dead links, response time, redundancy) and user-perceived relevance across different query topics and rank positions. Use when the user wants to benchmark on Custom IR Evaluation Corpus, or asks about evaluating this task. Reports Relevance Judgments.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1105.6213
  bibtex_key: bouramoul2011using
  confidence: high
---

# contextual-ir-eval

> Using Context to Improve the Evaluation of Information Retrieval Systems — Bouramoul et al. (2011) (arXiv:1105.6213, 2011)

## What this evaluates

Evaluates information retrieval systems by measuring system-level performance metrics (dead links, response time, redundancy) and user-perceived relevance across different query topics and rank positions.

## Datasets

- **Custom IR Evaluation Corpus** — total 1800; splits: test (1800)

## Metrics

- `Dead Link Rate` — range: percent
  - Percentage of retrieved URLs that fail to load or return a 404 error after up to three retry attempts.
- `Parasite Page Rate` — range: percent
  - Percentage of results linking to commercial sites offering online purchases or transactions.
- `Redundant Result Rate` — range: percent
  - Percentage of results containing duplicate or alias links to the same content.
- `Average Response Time` — range: other
  - Mean time in seconds from query submission to result display, measured on a single machine with fixed internet speed.
- `Relevance Judgments` **(primary)** — range: other
  - Average rating on a 0-5 scale assigned by non-expert users to the top-k results (k=1,5,10,15,20), where 0 is completely useless/off-topic and 5 is perfect.

## Input / output format

**Input**: Query string, search engine identifier, and retrieved URL/page content (organized as triplets).

**Output**: System metrics: boolean flags for dead links, parasite pages, redundancy, and float for response time. Relevance metrics: integer score 0-5 per result, or float score 0-10 per topic group.

## Scoring recipe

```python
def compute_metrics(urls, response_times, user_ratings, queries, results):
    dead_links = sum(1 for u in urls if u.is_dead) / len(urls) * 100
    parasites = sum(1 for u in urls if u.is_commercial) / len(urls) * 100
    redundant = sum(1 for u in urls if u.is_duplicate) / len(urls) * 100
    avg_time = sum(response_times) / len(response_times)
    
    relevance_scores = [r for r in user_ratings if 0 <= r <= 5]
    avg_relevance = sum(relevance_scores) / len(relevance_scores)
    
    topic_scores = [count_query_terms_in_result(q, r) for q, r in zip(queries, results)]
    avg_topic_relevance = sum(topic_scores) / len(topic_scores) * 2
    
    return dead_links, parasites, redundant, avg_time, avg_relevance, avg_topic_relevance
```

## Common pitfalls

- Response time measurements are highly dependent on the specific hardware and network conditions used during the experiment, making cross-study comparisons difficult.
- The custom relevance formula for query-topic scoring relies on word occurrence counts, which may underperform for short queries lacking sufficient terms.
- User judgments come from non-expert students, which may not generalize to professional relevance raters or real-world user behavior.

## Evidence (verbatim from paper)

> The 24 students also expressed their relevance judgments for 5, 10, 15 and 20 first retrieved documents (R@5, R@10, R@10, R@15, R@20). At each level of relevance, a note of 0-5 was assigned by each student. 0 corresponding to a document completely useless or off-topic, 5 corresponding to a document responding in a perfect way to the question.

## Citation

```bibtex
@misc{bouramoul2011using,
  title={Using Context to Improve the Evaluation of Information Retrieval Systems},
  author={Bouramoul et al. (2011)},
  year={2011},
  note={arXiv:1105.6213}
}
```

- arXiv: 1105.6213