Source Sentence Detection Eval

---
name: source-sentence-detection-eval
description: Evaluates a model's ability to identify which sentences in a source document contribute to an abstractive summary. It probes source sentence detection capability by ranking candidate sentences based on their inferred relevance to the summary. Use when the user wants to benchmark on SourceSum, or asks about evaluating this task. Reports NDCG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.04677
  bibtex_key: suhara2024sourceidentification
  confidence: high
---

# source-sentence-detection-eval

> Source Identification in Abstractive Summarization — Suhara et al. (2024) (arXiv:2402.04677, 2024)

## What this evaluates

Evaluates a model's ability to identify which sentences in a source document contribute to an abstractive summary. It probes source sentence detection capability by ranking candidate sentences based on their inferred relevance to the summary.

## Datasets

- **SourceSum** — total ?; splits: XSum PEGASUS (-1), XSum Ref (-1), CNN/DM PEGASUS (-1), CNN/DM Ref (-1)

## Metrics

- `NDCG` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain. Uses the total number of annotator votes as the relevance score for each sentence, ranking them by predicted contribution.
- `MAP` — range: [0, 1]
  - Mean Average Precision. Binarizes annotations by assigning a relevance score of 1 if two annotators agree a sentence is a source, and 0 otherwise.

## Input / output format

**Input**: A source document, a summary (generated or reference), and a list of candidate sentences from the document to be ranked.

**Output**: A ranked list of candidate sentences with associated relevance scores.

## Scoring recipe

```python
import math

def compute_ndcg(preds, votes):
    dcg = sum(v / math.log2(i + 2) for i, v in enumerate([votes[p] for p in preds]))
    idcg = sum(v / math.log2(i + 2) for i, v in enumerate(sorted(votes, reverse=True)))
    return dcg / idcg if idcg > 0 else 0.0

def compute_map(preds, gold_binary):
    rel_count = 0
    prec_sum = 0.0
    for i, idx in enumerate(preds):
        if gold_binary[idx] == 1:
            rel_count += 1
            prec_sum += rel_count / (i + 1)
    return prec_sum / sum(gold_binary) if sum(gold_binary) > 0 else 0.0
```

## Common pitfalls

- Using threshold-based binary classification instead of ranking metrics introduces arbitrary threshold selection bias.
- MAP requires strict binarization (1 if 2 annotators agree, else 0), whereas NDCG uses continuous vote counts; mixing these conventions invalidates results.

## Evidence (verbatim from paper)

> To make the evaluation independent of the choice of threshold selection, we used ranking metrics for evaluation, namely NDCG and MAP (Manning et al., 2008). For NDCG, we used the total votes as the score to consider sentences with more votes more important. For MAP calculation, we binarized annotations and considered source sentences if two annotators agree it is relevant.

## Citation

```bibtex
@misc{suhara2024sourceidentification,
  title={Source Identification in Abstractive Summarization},
  author={Suhara et al. (2024)},
  year={2024},
  note={arXiv:2402.04677}
}
```

- arXiv: 2402.04677