Ps Lora Cl Eval

---
name: ps-lora-cl-eval
description: Evaluates a model's ability to learn sequentially from multiple tasks without catastrophic forgetting, measuring both task accuracy and continual learning dynamics like forward/backward transfer and forgetting rates across NLP and vision benchmarks. Use when the user wants to benchmark on Standard & Long, TRACE, ViT Benchmark, or asks about evaluating this task. Reports Accuracy (Acc/AAA).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.08960
  bibtex_key: zhou2025pslora
  confidence: high
---

# ps-lora-cl-eval

> Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces — Zhou et al. (2025) (arXiv:2512.08960, 2025)

## What this evaluates

Evaluates a model's ability to learn sequentially from multiple tasks without catastrophic forgetting, measuring both task accuracy and continual learning dynamics like forward/backward transfer and forgetting rates across NLP and vision benchmarks.

## Datasets

- **Standard & Long** — total ?; splits: test (-1)
- **TRACE** — total ?; splits: test (-1)
- **ViT Benchmark** — total ?; splits: test (-1)

## Metrics

- `Accuracy (Acc/AAA)` **(primary)** — range: [0, 1]
  - Average prediction accuracy across all tasks in the sequence. Computed as the mean of per-task test accuracies after sequential training.
- `FR (Forgetting Rate)` — range: percent
  - Measures the average drop in accuracy on past tasks after learning new tasks. Lower is better.
- `FWT (Forward Transfer)` — range: percent
  - Measures the performance gain on new tasks compared to a baseline trained from scratch. Higher is better.
- `BWT (Backward Transfer)` — range: percent
  - Measures the performance change on past tasks after learning new tasks. Higher (less negative) is better.

## Input / output format

**Input**: Sequential task streams where each task consists of NLP classification/generation prompts or image classification samples. Models process tasks one-by-one in a continual learning setting.

**Output**: Predicted labels or generated text for each input instance. Performance is aggregated per task and across all tasks.

## Scoring recipe

```python
# After sequential training on tasks T_1...T_N:
accs = []
for t in range(1, N+1):
    accs.append(evaluate(model, test_set[T_t]))
avg_acc = sum(accs) / N
# FR, FWT, BWT computed from accs at each step vs final step per standard CL protocol
return avg_acc, fr, fwt, bwt
```

## Common pitfalls

- Task order significantly impacts results; evaluations should report across multiple random orders or explicitly state the order.
- Merging strategy (magnitude-based vs simple addition) drastically affects forgetting rates and must be explicitly stated.
- Standard (N=4) and Long (N=15) benchmarks yield different performance baselines and should not be compared directly.

## Evidence (verbatim from paper)

> Following O-LoRA*(olora)*, SD-LoRA*(wu2025sd)* and Tree-LoRA *(qian2025treelora)*, we evaluate on three widely used CL benchmarks across NLP and CV modalities: Standard \& Long, TRACE, and ViT Benchmark. ... Beyond overall accuracy, we analyze other CL metrics (i.e., FR, FWT, BWT) to assess forgetting and knowledge transfer (see Table[2]).

## Citation

```bibtex
@misc{zhou2025pslora,
  title={Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2512.08960}
}
```

- arXiv: 2512.08960