Tracsum Eval

---
name: tracsum-eval
description: Evaluates a model's ability to generate aspect-specific summaries from clinical abstracts and accurately cite the supporting source sentences. It probes factual recall, conciseness, and traceability in a medical domain setting. Use when the user wants to benchmark on TracSum, or asks about evaluating this task. Reports Claim Recall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.13798
  bibtex_key: chu2025tracsum
  confidence: high
---

# tracsum-eval

> TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain — Chu et al. (2025) (arXiv:2508.13798, 2025)

## What this evaluates

Evaluates a model's ability to generate aspect-specific summaries from clinical abstracts and accurately cite the supporting source sentences. It probes factual recall, conciseness, and traceability in a medical domain setting.

## Datasets

- **TracSum** — total 3500; splits: test (3500); repo https://github.com/chubohao/TracSum

## Metrics

- `Claim Recall` **(primary)** — range: [0, 1]
  - Proportion of gold summary claims that are entailed by the model's generated summary, assessed via a claim decomposition model and entailment evaluator.
- `Citation Recall` — range: [0, 1]
  - Proportion of gold cited sentence indices that are correctly retrieved by the model.
- `Claim Precision` — range: [0, 1]
  - Proportion of claims in the model's summary that are entailed by the gold summary.
- `Citation Precision` — range: [0, 1]
  - Proportion of model-cited sentence indices that match the gold citations.

## Input / output format

**Input**: A clinical abstract represented as a sequence of uniquely indexed sentences, paired with a target medical aspect from a predefined set of seven (Aims, Intervention, Outcomes, Participants, Medicine, Duration, Side Effects).

**Output**: An aspect-specific summary string and a set of cited sentence indices from the input abstract. If no relevant information exists for the aspect, output 'Unknown' for the summary and 'Null' for citations.

## Scoring recipe

```python
def score(predictions, gold):
    # predictions: (summary, cited_indices)
    # gold: (gold_summary, gold_cited_indices)
    claims_pred = decompose(predictions.summary)
    claims_gold = decompose(gold.summary)
    
    claim_recall = entailment_count(claims_pred, claims_gold) / len(claims_gold)
    claim_precision = entailment_count(claims_gold, claims_pred) / len(claims_pred)
    
    citation_recall = len(set(predictions.cited_indices) & set(gold.cited_indices)) / len(gold.cited_indices)
    citation_precision = len(set(predictions.cited_indices) & set(gold.cited_indices)) / len(predictions.cited_indices)
    
    return claim_recall, citation_recall, claim_precision, citation_precision
```

## Common pitfalls

- Negative samples (where the abstract lacks information for the given aspect) must output 'Unknown' and 'Null' instead of hallucinating content.
- Citation metrics require exact sentence index matching, not just semantic similarity or paragraph-level grounding.
- Evaluation must be performed per aspect (7 instances per abstract) rather than aggregating across aspects, as each aspect targets different clinical information.

## Evidence (verbatim from paper)

> Completeness is assessed using Claim Recall and Citation Recall, while conciseness is measured by Claim Precision and Citation Precision. The system M(C', sum' | d, a) is expected to generate an aspect-specific summary sum' and a set of cited sentences C', where c'_i refers to the index of a sentence in d that supports the summary. If the article contains no information relevant to the given aspect, the system should output sum'←"Unknown" and C'←"Null".

## Citation

```bibtex
@misc{chu2025tracsum,
  title={TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain},
  author={Chu et al. (2025)},
  year={2025},
  note={arXiv:2508.13798}
}
```

- arXiv: 2508.13798