Coedit Text Editing Eval

---
name: coedit-text-editing-eval
description: Evaluates a model's ability to follow task-specific and composite text editing instructions across diverse tasks like grammar correction, simplification, coherence, style transfer, and paraphrasing. It also assesses generalization to unseen instructions and human-perceived writing efficiency. Use when the user wants to benchmark on JFLEG, TurkCorpus, ASSET, ITER (Coherence split), DISCOFUSE, ITER (Iterative text revision), GYAFC, WNC, MRPC, STS, QQP, or asks about evaluating this task. Reports evaluation metrics.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.09857
  bibtex_key: raheja2023coedit
  confidence: medium
---

# coedit-text-editing-eval

> CoEdIT: Text Editing by Task-Specific Instruction Tuning — Raheja et al. (2023) (arXiv:2305.09857, 2023)

## What this evaluates

Evaluates a model's ability to follow task-specific and composite text editing instructions across diverse tasks like grammar correction, simplification, coherence, style transfer, and paraphrasing. It also assesses generalization to unseen instructions and human-perceived writing efficiency.

## Datasets

- **JFLEG** — total ?; splits: test (-1)
- **TurkCorpus** — total ?; splits: test (-1)
- **ASSET** — total ?; splits: test (-1)
- **ITER (Coherence split)** — total ?; splits: test (-1)
- **DISCOFUSE** — total ?; splits: test (-1)
- **ITER (Iterative text revision)** — total ?; splits: test (-1)
- **GYAFC** — total ?; splits: test (-1)
- **WNC** — total ?; splits: test (-1)
- **MRPC** — total ?; splits: test (-1)
- **STS** — total ?; splits: test (-1)
- **QQP** — total ?; splits: test (-1)

## Metrics

- `evaluation metrics` **(primary)** — range: varies
  - Varies by dataset; includes automatic metrics (e.g., BLEU, METEOR, ROUGE, ASSET score) and human evaluation for writing efficiency. Exact formulas, conventions, and scoring details are provided in Appendix B.

## Input / output format

**Input**: Source text concatenated with a natural language editing instruction specifying the desired transformation.

**Output**: Edited text generated via greedy decoding.

## Scoring recipe

```python
for dataset in [JFLEG, TurkCorpus, ASSET, ITER_Coherence, DISCOFUSE, ITER_Iterative, GYAFC, WNC, MRPC, STS, QQP]:
    predictions = model.generate(input_text + instruction)
    gold = dataset.gold_references
    score = compute_dataset_specific_metric(predictions, gold)  # Details in Appendix B
    aggregate_scores.append(score)
# Human evaluation also conducted for RQ3 (efficiency)
```

## Common pitfalls

- Metrics are not explicitly defined in the main text; researchers must consult Appendix B for exact formulas and conventions.
- Evaluation includes both automatic metrics and human evaluation for writing efficiency, which are reported separately.
- All model outputs are generated using greedy decoding unless otherwise specified.

## Evidence (verbatim from paper)

> To assess the editing capabilities of CoEdit, we perform evaluations on standard test sets sourced from a variety of text editing task benchmarks, most notably, EDITEVAL (Dwivedi-Yu et al., 2022). ... Detailed descriptions of each dataset and its evaluation metrics are in §B

## Citation

```bibtex
@misc{raheja2023coedit,
  title={CoEdIT: Text Editing by Task-Specific Instruction Tuning},
  author={Raheja et al. (2023)},
  year={2023},
  note={arXiv:2305.09857}
}
```

- arXiv: 2305.09857