Lora Dropout Eval

---
name: lora-dropout-eval
description: Evaluates the effectiveness of transformer-specific dropout methods (e.g., HiddenKey, DropKey, HiddenCut) when combined with LoRA for parameter-efficient fine-tuning. It probes the model's ability to mitigate overfitting in LoRA settings across diverse natural language understanding and generation tasks. Use when the user wants to benchmark on GLUE, E2E, WebNLG, or asks about evaluating this task. Reports Accuracy, BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.00812
  bibtex_key: wang2024lora
  confidence: high
---

# lora-dropout-eval

> LoRA Meets Dropout under a Unified Framework — Wang et al. (2024) (arXiv:2403.00812, 2024)

## What this evaluates

Evaluates the effectiveness of transformer-specific dropout methods (e.g., HiddenKey, DropKey, HiddenCut) when combined with LoRA for parameter-efficient fine-tuning. It probes the model's ability to mitigate overfitting in LoRA settings across diverse natural language understanding and generation tasks.

## Datasets

- **GLUE** — total ?; splits: train (-1), val (-1), test (-1)
- **E2E** — total ?; splits: train (-1), test (-1)
- **WebNLG** — total ?; splits: train (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted class labels.
- `Pearson correlation` — range: [-1, 1]
  - Pearson correlation coefficient between predicted and reference scores.
- `BLEU` **(primary)** — range: percent
  - n-gram based precision metric for machine translation and text generation.
- `NIST` — range: percent
  - Information-content-weighted variant of BLEU.
- `METEOR` — range: percent
  - Metric that aligns machine translation with reference translations using synonymy and stemming.
- `ROUGE_L` — range: percent
  - Recall-Oriented Understudy for Gisting Evaluation based on longest common subsequence.
- `CIDEr` — range: percent
  - Consensus-based Image Description Evaluation metric using TF-IDF weighting.
- `TER` — range: percent
  - Translation Edit Rate measuring the number of edits needed to change hypothesis to reference.
- `Matthews correlation` — range: [-1, 1]
  - Correlation coefficient between predicted and actual binary classifications.

## Input / output format

**Input**: Text sequences for NLU tasks; source sentences for NLG tasks.

**Output**: Class labels or scores for NLU; generated text sequences for NLG.

## Scoring recipe

```python
def score(predictions, golds, metric, task):
    if task == 'NLU':
        if metric == 'Accuracy':
            return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
        elif metric == 'Matthews_corr':
            return matthews_corrcoef(golds, predictions)
        elif metric == 'Pearson_corr':
            return pearsonr(golds, predictions)[0]
    elif task == 'NLG':
        if metric == 'BLEU':
            return compute_bleu(golds, predictions)
        elif metric == 'ROUGE_L':
            return compute_rouge_l(golds, predictions)
        # ... other NLG metrics similarly
```

## Common pitfalls

- LoRA baseline uses rank 8 and scalar 16, differing from the original LoRA paper's default rank 4.
- DropAttention's gradient stopping mechanism (NoGrad()) causes instability and poor performance at higher dropout rates.
- KL divergence regularization improves performance, while JS divergence shows no apparent impact in LoRA settings.

## Evidence (verbatim from paper)

> For NLU tasks, we utilize six datasets from GLUE benchmark*(Wang et al., [2018])*: SST-2 *(Socher et al., [2013])*... Pearson correlation is reported for STS-B, while accuracy is utilized for others.

## Citation

```bibtex
@misc{wang2024lora,
  title={LoRA Meets Dropout under a Unified Framework},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2403.00812}
}
```

- arXiv: 2403.00812