Curlora Continual Eval

---
name: curlora-continual-eval
description: Tests a model's ability to learn sequentially across multiple NLP tasks while retaining prior knowledge. It specifically probes catastrophic forgetting mitigation during continual fine-tuning by measuring performance drops on earlier tasks after learning new ones. Use when the user wants to benchmark on GLUE-MRPC, GLUE-SST-2, Sentiment140, WikiText-2, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.14572
  bibtex_key: fawi2024curlora
  confidence: high
---

# curlora-continual-eval

> CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation — Fawi et al. (2024) (arXiv:2408.14572, 2024)

## What this evaluates

Tests a model's ability to learn sequentially across multiple NLP tasks while retaining prior knowledge. It specifically probes catastrophic forgetting mitigation during continual fine-tuning by measuring performance drops on earlier tasks after learning new ones.

## Datasets

- **GLUE-MRPC** — total ?; splits: train (-1), val (-1), test (-1)
- **GLUE-SST-2** — total ?; splits: train (-1), val (-1), test (-1)
- **Sentiment140** — total ?; splits: test (-1), train (-1)
- **WikiText-2** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correct predictions out of the total number of instances.
- `Perplexity` — range: other
  - Standard language modeling perplexity: the exponential of the average negative log-likelihood of the ground-truth tokens.

## Input / output format

**Input**: Text pairs for MRPC, single sentences for SST-2 and Sentiment140, and tokenized text sequences for WikiText-2.

**Output**: Class labels for classification tasks (paraphrase yes/no, sentiment positive/negative/neutral), and next-token probability distributions for language modeling.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    return sum(p == g for p, g in zip(predictions, gold)) / len(gold)

def compute_perplexity(log_probs):
    import math
    return math.exp(-sum(log_probs) / len(log_probs))

# Apply compute_accuracy to MRPC, SST-2, Sentiment140
# Apply compute_perplexity to WikiText-2 token log-probs
```

## Common pitfalls

- Sentiment140 train/test splits are intentionally swapped compared to standard usage (test set used for training, train set for evaluation).
- Forgetting is measured by re-evaluating previous tasks after each new fine-tuning step, not just by final task accuracy.
- Hyperparameters are task-specific (e.g., GPT-2 on SST-2 uses 5 epochs without a scheduler), requiring exact replication per task.

## Evidence (verbatim from paper)

> We used the following metrics for evaluation: Accuracy: For classification tasks (MRPC, SST-2, Sentiment140) Perplexity: For language modeling capability (WikiText-2). Our experimental procedure was as follows: 1. Measure initial perplexity of the base model on WikiText-2 concatenating the whole dataset into a single string. 2. Fine-tune on MRPC and evaluate. 3. Fine-tune on SST-2 and evaluate, then re-evaluate on MRPC. 4. Fine-tune on Sentiment140 and evaluate, then re-evaluate on MRPC and SST-2. 5. Re-calculate perplexity on WikiText-2.

## Citation

```bibtex
@misc{fawi2024curlora,
  title={CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation},
  author={Fawi et al. (2024)},
  year={2024},
  note={arXiv:2408.14572}
}
```

- arXiv: 2408.14572