Sentence Level Cal Eval

---
name: sentence-level-cal-eval
description: Evaluates the recall and efficiency of continuous active learning systems for information retrieval when using sentence-level versus document-level relevance feedback. It measures how quickly a simulated reviewer can identify all relevant documents under varying effort models that account for assessment count and sentence reading time. Use when the user wants to benchmark on TREC Total Recall 2015 Track, HARD 2004 Track, or asks about evaluating this task. Reports Recall@E.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1803.08988
  bibtex_key: zhang2018evaluating
  confidence: high
---

# sentence-level-cal-eval

> Evaluating Sentence-Level Relevance Feedback for High-Recall Information Retrieval — Zhang et al. (2018) (arXiv:1803.08988, 2018)

## What this evaluates

Evaluates the recall and efficiency of continuous active learning systems for information retrieval when using sentence-level versus document-level relevance feedback. It measures how quickly a simulated reviewer can identify all relevant documents under varying effort models that account for assessment count and sentence reading time.

## Datasets

- **TREC Total Recall 2015 Track** — total ?; splits: test (-1)
- **HARD 2004 Track** — total ?; splits: test (-1)

## Metrics

- `Recall@E` **(primary)** — range: [0, 1]
  - Recall at effort E, calculated as the number of relevant documents presented in the system output truncated at effort E, divided by the total number of relevant documents. Effort E can be measured as assessments only (E_judge), sentences read only (E_sent), or a weighted combination E_λ = (1-λ)E_judge + λE_sent. Specific effort levels tested are a*R+b where R is total relevant documents, a∈{1,2,4}, and b∈{0,100,1000}.

## Input / output format

**Input**: System-selected sequence of documents or sentences to be assessed, alongside gold relevance assessments (qrels) for both documents and sentences.

**Output**: Truncated system output O@E at a specified effort level E, representing the sequence of documents presented to the simulated reviewer up to that effort.

## Scoring recipe

```python
def compute_recall_at_effort(system_output, qrels, effort_E, effort_type):
    # Truncate output based on effort type
    if effort_type == 'E_judge':
        truncated_docs = system_output[:effort_E]
    elif effort_type == 'E_sent':
        truncated_docs = get_docs_up_to_sent_count(system_output, effort_E)
    else:  # E_lambda
        truncated_docs = get_docs_up_to_lambda_effort(system_output, effort_E)
    # Compute recall
    relevant_docs = set(qrels.keys())
    presented_docs = set(truncated_docs)
    recall = len(presented_docs & relevant_docs) / len(relevant_docs)
    return recall
```

## Common pitfalls

- Effort is not simply the number of documents reviewed; it depends on whether the system uses sentence-level or document-level feedback, which changes how E is calculated.
- The paper tests specific effort levels defined as a*R+b (where R is total relevant docs, a∈{1,2,4}, b∈{0,100,1000}), not arbitrary fixed numbers.
- For document-level feedback, effort includes reading sentences sequentially until the first relevant one is found, or all sentences if non-relevant, which directly affects E_sent and E_λ.

## Evidence (verbatim from paper)

> Recall is the number of relevant documents presented to the reviewer for assessment, as a fraction of the total number of relevant documents (R), regardless of whether document- or sentence-level feedback is employed. In our paper, the documents presented to the reviewer are recorded by the system output (O). We measure the recall at effort (Recall@E) using the Equation: Recall@E = |O@E ∩ Relevant documents| / |Relevant documents|

## Citation

```bibtex
@misc{zhang2018evaluating,
  title={Evaluating Sentence-Level Relevance Feedback for High-Recall Information Retrieval},
  author={Zhang et al. (2018)},
  year={2018},
  note={arXiv:1803.08988}
}
```

- arXiv: 1803.08988