Reflectsumm Eval

---
name: reflectsumm-eval
description: Evaluates NLP models on summarizing student course reflections across three tasks: document selection, phrase extraction with support counts, and abstractive summarization. It probes specificity-aware summarization capabilities and model robustness in low-resource educational settings with variable text structure. Use when the user wants to benchmark on ReflectSumm, or asks about evaluating this task. Reports Unspecified in text.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.19012
  bibtex_key: zhong2024reflectsumm
  confidence: medium
---

# reflectsumm-eval

> ReflectSumm: A Benchmark for Course Reflection Summarization — Zhong et al. (2024) (arXiv:2403.19012, 2024)

## What this evaluates

Evaluates NLP models on summarizing student course reflections across three tasks: document selection, phrase extraction with support counts, and abstractive summarization. It probes specificity-aware summarization capabilities and model robustness in low-resource educational settings with variable text structure.

## Datasets

- **ReflectSumm** — total 17512; splits: train (-1), val (-1), test (-1); repo https://github.com/EngSalem/ReflectSUMM

## Metrics

- `Unspecified in text` **(primary)** — range: percent
  - Standard summarization metrics (e.g., ROUGE, BERTScore) or human evaluation are implied for benchmarking, but exact formulas and units are not provided in the section.

## Input / output format

**Input**: A collection of student reflection documents from a single lecture, optionally augmented with 4-point specificity scores per reflection.

**Output**: Three task variants: (1) exactly 5 selected reflection documents, (2) exactly 5 phrases each with an accompanying support count, (3) a single abstractive summary of approximately 40 words.

## Scoring recipe

```python
# Extractive: Compute exact match or overlap between predicted 5 docs and gold 5 docs
# Phrase: Compute phrase-level overlap (e.g., ROUGE) between predicted 5 phrases+counts and gold
# Abstractive: Compute sentence/document-level overlap (e.g., ROUGE-L) between predicted ~40-word summary and gold
# Note: Exact metric implementation not specified in text; standard summarization metrics apply.
```

## Common pitfalls

- Models struggle with variable reflection depth and structure, leading to poor content coherence and topic relevance.
- Ignoring specificity scores when prompted, which degrades performance on specificity-aware tasks.
- Failing to respect exact output constraints (e.g., selecting exactly 5 items or generating ~40 words).

## Evidence (verbatim from paper)

> Corresponding to the human extractive summary task, the goal of our models is to pinpoint the five most salient reflections (documents) from a collection of reflections within the same lecture. Human annotators were given the task of summarizing students’ reflections concisely and coherently within ~40 words.

## Citation

```bibtex
@misc{zhong2024reflectsumm,
  title={ReflectSumm: A Benchmark for Course Reflection Summarization},
  author={Zhong et al. (2024)},
  year={2024},
  note={arXiv:2403.19012}
}
```

- arXiv: 2403.19012