Lora Land Eval

---
name: lora-land-eval
description: This evaluation probes the effectiveness of LoRA fine-tuning across 31 diverse NLP tasks by comparing base LLMs against their fine-tuned counterparts and proprietary models like GPT-4. It measures how much performance lift fine-tuning provides and whether smaller open-weight models can surpass larger closed-source models after adaptation. Use when the user wants to benchmark on magicoder, mmlu, glue_wnli, arc_combined, wikisql, boolq, customer_support, glue_cola, winogrande, glue_sst2, dbpedia, hellaswag, glue_qnli, e2e_nlg, glue_qqp, bc5cdr, glue_mnli, webnlg, tldr_content_gen, glue_mrpc, jigsaw, hellaswag_processed, viggo, glue_stsb, gsm8k, conllpp, tldr_headline_gen, drop, legal, reuters, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.00732
  bibtex_key: zhao2024loraland
  confidence: high
---

# lora-land-eval

> LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report — Justin Zhao et al. (2024) (arXiv:2405.00732, 2024)

## What this evaluates

This evaluation probes the effectiveness of LoRA fine-tuning across 31 diverse NLP tasks by comparing base LLMs against their fine-tuned counterparts and proprietary models like GPT-4. It measures how much performance lift fine-tuning provides and whether smaller open-weight models can surpass larger closed-source models after adaptation.

## Datasets

- **magicoder** — total ?; splits: test (-1)
- **mmlu** — total ?; splits: test (-1)
- **glue_wnli** — total ?; splits: test (-1)
- **arc_combined** — total ?; splits: test (-1)
- **wikisql** — total ?; splits: test (-1)
- **boolq** — total ?; splits: test (-1)
- **customer_support** — total ?; splits: test (-1)
- **glue_cola** — total ?; splits: test (-1)
- **winogrande** — total ?; splits: test (-1)
- **glue_sst2** — total ?; splits: test (-1)
- **dbpedia** — total ?; splits: test (-1)
- **hellaswag** — total ?; splits: test (-1)
- **glue_qnli** — total ?; splits: test (-1)
- **e2e_nlg** — total ?; splits: test (-1)
- **glue_qqp** — total ?; splits: test (-1)
- **bc5cdr** — total ?; splits: test (-1)
- **glue_mnli** — total ?; splits: test (-1)
- **webnlg** — total ?; splits: test (-1)
- **tldr_content_gen** — total ?; splits: test (-1)
- **glue_mrpc** — total ?; splits: test (-1)
- **jigsaw** — total ?; splits: test (-1)
- **hellaswag_processed** — total ?; splits: test (-1)
- **viggo** — total ?; splits: test (-1)
- **glue_stsb** — total ?; splits: test (-1)
- **gsm8k** — total ?; splits: test (-1)
- **conllpp** — total ?; splits: test (-1)
- **tldr_headline_gen** — total ?; splits: test (-1)
- **drop** — total ?; splits: test (-1)
- **legal** — total ?; splits: test (-1)
- **reuters** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct predictions out of total instances. Calculated as (number of correct predictions) / (total number of predictions).
- `rouge` — range: [0, 1]
  - ROUGE-N metric (typically ROUGE-L or ROUGE-1/2) measuring overlap between generated text and reference text using recall/precision/F1 of n-grams or longest common subsequence.
- `mae` — range: [0, inf)
  - Mean Absolute Error measuring the average absolute difference between predicted and actual continuous values.

## Input / output format

**Input**: Task-specific prompts or instructions corresponding to each of the 31 NLP benchmarks (e.g., multiple-choice questions, text generation prompts, classification instructions).

**Output**: Task-specific model outputs (e.g., selected option, generated text, regression value) evaluated against ground-truth references using the task-appropriate metric.

## Scoring recipe

```python
def compute_metric(predictions, golds, metric_type):
    if metric_type == 'accuracy':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif metric_type == 'rouge':
        return rouge_score(golds, predictions, rouge_types=['rouge1', 'rouge2', 'rougeL'])
    elif metric_type == 'mae':
        return mean_absolute_error(golds, predictions)
```

## Common pitfalls

- Confusing base model scores with fine-tuned model scores when calculating performance lift.
- Assuming a single evaluation metric applies across all 31 tasks; the benchmark uses accuracy, ROUGE, and MAE depending on the task type.
- Overlooking reported implementation bugs in specific model families (e.g., Gemma) that caused fine-tuning to underperform base models, skewing average lift calculations.

## Evidence (verbatim from paper)

> mmlu | accuracy | 0.506 | 0.589 | 0.774 | 0.083 | -0.185
glue_wnli | accuracy | 0.437 | 0.873 | 0.93 | 0.436 | -0.057
arc_combined | accuracy | 0.673 | 0.915 | 0.947 | 0.242 | -0.032
wikisql | rouge | 0.301 | 0.898 | 0.909 | 0.597 | -0.011

## Citation

```bibtex
@misc{zhao2024loraland,
  title={LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report},
  author={Justin Zhao et al. (2024)},
  year={2024},
  note={arXiv:2405.00732}
}
```

- arXiv: 2405.00732