Corda Peft Eval

---
name: corda-peft-eval
description: Evaluates parameter-efficient fine-tuning methods across mathematical reasoning, code generation, instruction following, and general language understanding tasks, while measuring their ability to retain pre-trained world knowledge. Use when the user wants to benchmark on MetaMathQA, GSM8k, Math, CodeFeedback, HumanEval, MBPP, WizardLM-Evol-Instruct, MTBench, TriviaQA, NQ open, WebQS, GLUE, Wikitext-2, Penn TreeBank (PTB), or asks about evaluating this task. Reports exact match scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.05223
  bibtex_key: yang2024corda
  confidence: high
---

# corda-peft-eval

> CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning — Yang et al. (2024) (arXiv:2406.05223, 2024)

## What this evaluates

Evaluates parameter-efficient fine-tuning methods across mathematical reasoning, code generation, instruction following, and general language understanding tasks, while measuring their ability to retain pre-trained world knowledge.

## Datasets

- **MetaMathQA** — total ?; splits: train (-1)
- **GSM8k** — total ?; splits: test (-1)
- **Math** — total ?; splits: validation (-1)
- **CodeFeedback** — total ?; splits: train (-1)
- **HumanEval** — total ?; splits: test (-1)
- **MBPP** — total ?; splits: test (-1)
- **WizardLM-Evol-Instruct** — total ?; splits: train (-1)
- **MTBench** — total ?; splits: test (-1)
- **TriviaQA** — total ?; splits: test (-1)
- **NQ open** — total ?; splits: test (-1)
- **WebQS** — total ?; splits: test (-1)
- **GLUE** — total ?; splits: test (-1)
- **Wikitext-2** — total ?; splits: test (-1)
- **Penn TreeBank (PTB)** — total ?; splits: test (-1)

## Metrics

- `exact match scores` **(primary)** — range: percent
  - Percentage of generated answers that exactly match the ground truth reference string.
- `MTBench score` — range: other
  - Average score assigned by an LLM-as-judge model across multi-turn instruction-following conversations, typically on a scale of 1-10.
- `Perplexity` — range: other
  - Exponential of the average negative log-likelihood of the reference text under the model distribution: exp(-1/N * sum(log P(x_i))). Lower is better.
- `Matthew’s correlation / Pearson’s correlation / Accuracy` — range: percent
  - Standard GLUE metrics: Matthew’s correlation for CoLA, Pearson’s correlation for STS-B, and accuracy for SST-2, MRPC, QNLI, and RTE.

## Input / output format

**Input**: Text-based prompts, questions, or code instructions provided as strings.

**Output**: Generated text responses (answers, code snippets, or instructions).

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type == 'exact_match':
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds) * 100
    elif task_type == 'perplexity':
        return math.exp(-sum(log_probs) / len(log_probs))
    elif task_type == 'glue':
        return compute_glue_metrics(predictions, golds)
    elif task_type == 'mtbench':
        return llm_judge_score(predictions, golds)
```

## Common pitfalls

- Knowledge-preserved adaptation freezes high-singular-value components to retain world knowledge, which inherently trades off peak task performance compared to instruction-previewed mode.
- MTBench relies on LLM-as-judge evaluation, meaning scores can fluctuate based on the judge model version, temperature, and prompt formatting.
- Exact-match scoring is strict for math and code; execution-based or relaxed matching often yields higher scores but is not used in this paper's primary tables.

## Evidence (verbatim from paper)

> The world knowledge is evaluated by the exact match scores (%) on TriviaQA, NQ open, and WebQS. Following the settings in [[47]], the Math ability is trained on MetaMathQA and tested on GSM8k and Math validation sets.

## Citation

```bibtex
@misc{yang2024corda,
  title={CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2406.05223}
}
```

- arXiv: 2406.05223