Lexical Simplification Eval

---
name: lexical-simplification-eval
description: Evaluates the robustness and classification accuracy of pre-trained language models when augmented with rule-based lexical simplification as auxiliary inputs. It probes whether lemmatization and rare-word replacement preserve semantic meaning while mitigating lexical diversity effects on downstream NLU tasks. Use when the user wants to benchmark on SST-2, CR, SUBJ, MR, AG, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.15070
  bibtex_key: bao2020enhancing
  confidence: high
---

# lexical-simplification-eval

> Enhancing Pre-trained Language Model with Lexical Simplification — Bao et al. (2020) (arXiv:2012.15070, 2020)

## What this evaluates

Evaluates the robustness and classification accuracy of pre-trained language models when augmented with rule-based lexical simplification as auxiliary inputs. It probes whether lemmatization and rare-word replacement preserve semantic meaning while mitigating lexical diversity effects on downstream NLU tasks.

## Datasets

- **SST-2** — total ?; splits: train (-1), test (-1)
- **CR** — total ?; splits: train (-1), test (-1)
- **SUBJ** — total ?; splits: train (-1), test (-1)
- **MR** — total ?; splits: train (-1), test (-1)
- **AG** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted class labels out of the total number of instances: (TP + TN) / (TP + TN + FP + FN).
- `F1` — range: [0, 1]
  - The harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Original text sequence paired with a rule-simplified version of the same text (lemmatized and rare words replaced) as auxiliary input.

**Output**: Discrete class label corresponding to the dataset's taxonomy (e.g., positive/negative sentiment or one of four news topics).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- The simplified auxiliary input must be processed alongside the original text, not as a replacement.
- Rare word replacement strictly preserves part-of-speech tags; ignoring this constraint alters the evaluation protocol.
- Performance gains are measured relative to specific baselines (BERT-base, ELECTRA-large) under identical training/inference conditions.

## Evidence (verbatim from paper)

> Experiments on SST-2, MR, CR, SUBJ, and AG show consistent performance gains over baseline models (e.g., +1.1% to +0.8% F1), outperforming synonym replacement, back-translation, and cutoff methods in both training and inference.

## Citation

```bibtex
@misc{bao2020enhancing,
  title={Enhancing Pre-trained Language Model with Lexical Simplification},
  author={Bao et al. (2020)},
  year={2020},
  note={arXiv:2012.15070}
}
```

- arXiv: 2012.15070