Email Subject Line Generation Eval

---
name: email-subject-line-generation-eval
description: Evaluates a model's ability to generate highly abstractive, ultra-concise email subject lines from email body text. It probes extreme compression, informativeness, and fluency in a real-world email triaging context, distinguishing the task from standard text summarization. Use when the user wants to benchmark on AESLC, or asks about evaluating this task. Reports ROUGE-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.03497
  bibtex_key: zhang2019email
  confidence: high
---

# email-subject-line-generation-eval

> This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation — Zhang et al. (2019) (arXiv:1906.03497, 2019)

## What this evaluates

Evaluates a model's ability to generate highly abstractive, ultra-concise email subject lines from email body text. It probes extreme compression, informativeness, and fluency in a real-world email triaging context, distinguishing the task from standard text summarization.

## Datasets

- **AESLC** — total ?; splits: dev (-1), test (-1)

## Metrics

- `ROUGE-1` **(primary)** — range: percent
  - F1 score of unigram overlap between the generated subject line and the reference subject line(s). Computed as 2 * (precision * recall) / (precision + recall).
- `ROUGE-2` — range: percent
  - F1 score of bigram overlap between the generated subject line and the reference subject line(s).
- `ROUGE-L` — range: percent
  - F1 score based on the longest common subsequence between the generated subject line and the reference subject line(s), capturing sentence-level structure.
- `METEOR` — range: percent
  - Metric for Evaluation of Translation with Explicit ORdering; aligns words using synonymy and stemming, penalizes for fragmentation and ordering differences.
- `ESQE` — range: other
  - Email Subject Quality Estimator; a reference-less neural metric specifically trained to score subject line quality without requiring ground-truth references.
- `Human Rating` — range: [1, 4]
  - Average score on a 1-4 scale (Poor, Fair, Good, Great) across three aspects: overall quality, informativeness, and fluency. Each sample is rated by 3 independent judges.

## Input / output format

**Input**: Email body text (including greeting and main content) provided as the source document.

**Output**: A single sentence or short phrase representing the generated email subject line.

## Scoring recipe

```python
def compute_rouge_f1(pred, refs, n=1):
    pred_ngrams = set(ngrams(pred.split(), n))
    ref_ngrams = set()
    for r in refs:
        ref_ngrams.update(ngrams(r.split(), n))
    if not pred_ngrams or not ref_ngrams:
        return 0.0
    overlap = len(pred_ngrams & ref_ngrams)
    prec = overlap / len(pred_ngrams)
    rec = overlap / len(ref_ngrams)
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_human_rating(predictions, judges=3):
    scores = []
    for pred in predictions:
        ratings = [judge.assign_score(pred) for _ in range(judges)]
        scores.append(sum(ratings) / len(ratings))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Evaluating against a single reference when the dataset actually provides multiple human annotations per email (Table 3b shows evaluation against two human annotations).
- Relying solely on n-gram overlap metrics (ROUGE/METEOR) without considering that email subject line generation requires extreme abstraction and compression, which these metrics may penalize unfairly.
- Ignoring the reference-less ESQE metric, which is specifically designed for this task and often correlates better with human judgment than standard summarization metrics.

## Evidence (verbatim from paper)

> We first use automatic metrics from text summarization and machine translation: (1) ROUGE (Lin, 2004) including F1 scores of ROUGE-1, ROUGE-2, and ROUGE-L. (2) METEOR (Denkowski and Lavie, 2014). They all rely on one or more references and measure the similarity between the output and the reference. In addition, we include ESQE, which is a reference-less metric.

## Citation

```bibtex
@misc{zhang2019email,
  title={This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation},
  author={Zhang et al. (2019)},
  year={2019},
  note={arXiv:1906.03497}
}
```

- arXiv: 1906.03497