Seer Eval

---
name: seer-eval
description: Evaluates LLMs' ability to identify precise textual spans expressing emotion within single-sentence and multi-sentence contexts, distinguishing emotion evidence from other linguistic elements. Use when the user wants to benchmark on SEER, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.03490
  bibtex_key: sampath2025seer
  confidence: high
---

# seer-eval

> SEER: The Span-based Emotion Evidence Retrieval Benchmark — Sampath et al. (2025) (arXiv:2510.03490, 2025)

## What this evaluates

Evaluates LLMs' ability to identify precise textual spans expressing emotion within single-sentence and multi-sentence contexts, distinguishing emotion evidence from other linguistic elements.

## Datasets

- **SEER** — total 1200; splits: test (-1); repo https://github.com/chailab-umich/SEER

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Span-level F1 score computed over retrieved or highlighted emotion evidence spans. Exact match is required for highlighting; hallucinations score 0.
- `Cosine Similarity (Sim)` — range: [0, 1]
  - Cosine similarity between the embedding of the predicted span and the gold span.

## Input / output format

**Input**: Single-sentence or multi-sentence text passages containing potential emotion evidence.

**Output**: Retrieved span text or highlighted text with verbatim markup indicating the emotion evidence.

## Scoring recipe

```python
def evaluate(predictions, golds):
    f1s, sims = [], []
    for pred, gold in zip(predictions, golds):
        f1s.append(compute_span_f1(pred, gold))
        sims.append(cosine_similarity(embed(pred), embed(gold)))
    return mean(f1s), mean(sims)
```

## Common pitfalls

- CoT prompting degrades performance on single-sentence tasks but improves it on multi-sentence tasks.
- Highlight prompt is significantly harder than Retrieve due to strict verbatim requirements; any hallucination yields a score of 0.
- Model size does not directly predict performance; smaller models can outperform much larger ones.

## Evidence (verbatim from paper)

> Table 3: F1 and cosine similarity (Sim) scores for Task 1 (Retrieve and Highlight). Each entry is averaged over five runs with standard deviations.

## Citation

```bibtex
@misc{sampath2025seer,
  title={SEER: The Span-based Emotion Evidence Retrieval Benchmark},
  author={Sampath et al. (2025)},
  year={2025},
  note={arXiv:2510.03490}
}
```

- arXiv: 2510.03490