Clif Eval

---
name: clif-eval
description: Evaluates a model's ability to accumulate knowledge across a sequence of NLP tasks (continual learning) while maintaining performance on previously seen tasks and generalizing to new few-shot tasks. Use when the user wants to benchmark on CLIF-26, CLIF-55, or asks about evaluating this task. Reports Final Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.08808
  bibtex_key: jin2021learn
  confidence: medium
---

# clif-eval

> Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning — Jin et al. (2021) (arXiv:2104.08808, 2021)

## What this evaluates

Evaluates a model's ability to accumulate knowledge across a sequence of NLP tasks (continual learning) while maintaining performance on previously seen tasks and generalizing to new few-shot tasks.

## Datasets

- **CLIF-26** — total ?; splits: train (-1), test (-1); repo https://github.com/INK-USC/CLIF
- **CLIF-55** — total ?; splits: train (-1), test (-1); repo https://github.com/INK-USC/CLIF

## Metrics

- `Final Accuracy` **(primary)** — range: percent
  - Standard classification accuracy computed over all upstream tasks after continual learning is complete.
- `Instant Accuracy` — range: percent
  - Accuracy measured on upstream tasks immediately after learning each task, without further fine-tuning.
- `Few-shot Accuracy` — range: percent
  - Accuracy measured on unseen tasks using only K labeled examples per task for adaptation.

## Input / output format

**Input**: Text pairs (input, label) formatted for a text-to-text model (BART), with task representations computed as average encoder-decoder latent activations over training examples.

**Output**: Predicted text labels or class tokens generated by the model.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_relative_improvement(metric_val, baseline_val):
    return ((metric_val - baseline_val) / baseline_val) * 100
```

## Common pitfalls

- Confusing Instant Accuracy (performance on upstream tasks during/after continual learning) with Few-shot Accuracy (performance on unseen tasks with limited examples).
- Failing to use the correct zero-knowledge baseline (BART-Adapter-Single or BiHNet-Single) when computing relative improvements (Δ_Inst. and Δ_FS.).

## Evidence (verbatim from paper)

> Final accuracy (Final Acc.) and instant accuracy (Instant Acc.) over upstream tasks and accuracy over few-shot learning tasks (Few-shot Acc.) on CLIF-26 and CLIF-55 tasks. We compute relative improvement of instant accuracy ($\Delta_{\text{Inst.}}$) and few-shot accuracy ($\Delta_{\text{FS}}$) over zero-knowledge baselines (the better one between BART-Adapter-Single and BiHNet-Single for BiHNet, and BART-Single for BART approaches).

## Citation

```bibtex
@misc{jin2021learn,
  title={Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning},
  author={Jin et al. (2021)},
  year={2021},
  note={arXiv:2104.08808}
}
```

- arXiv: 2104.08808