Federated Llm Peft Eval

---
name: federated-llm-peft-eval
description: Evaluates the effectiveness and efficiency of federated fine-tuning large language models using parameter-efficient fine-tuning (PEFT) algorithms across code generation, general language, and mathematical reasoning tasks under different data heterogeneity and privacy constraints. Use when the user wants to benchmark on Fed-CodeAlpaca, Fed-Dolly, Fed-GSM8K-3, HumanEval, HELM, GSM8K-test, or asks about evaluating this task. Reports Evaluation Scores(%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.00363
  bibtex_key: kuang2023federatedscollm
  confidence: high
---

# federated-llm-peft-eval

> FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning — Kuang et al. (2023) (arXiv:2309.00363, 2023)

## What this evaluates

Evaluates the effectiveness and efficiency of federated fine-tuning large language models using parameter-efficient fine-tuning (PEFT) algorithms across code generation, general language, and mathematical reasoning tasks under different data heterogeneity and privacy constraints.

## Datasets

- **Fed-CodeAlpaca** — total ?; splits: train (-1)
- **Fed-Dolly** — total ?; splits: train (-1)
- **Fed-GSM8K-3** — total ?; splits: train (-1)
- **HumanEval** — total ?; splits: test (-1)
- **HELM** — total ?; splits: test (-1)
- **GSM8K-test** — total ?; splits: test (-1)

## Metrics

- `Evaluation Scores(%)` **(primary)** — range: percent
  - Reported as mean percentage ± standard deviation over three runs with different random seeds. Computed by comparing model predictions against gold labels on the respective test sets.
- `Pass@1` — range: percent
  - Probability that the first generated solution passes the test cases. Averaged across clients for personalized settings.
- `GPU Usage (MB)` — range: other
  - Peak GPU memory consumption during fine-tuning, excluding input tokens and optimizer state.
- `Message Size (MB)` — range: other
  - Number of bytes of serialized adapter parameters transmitted between server and client per communication round.
- `Computation Time (Sec.)` — range: other
  - Duration of one training step (batch size 1) from forward to backward propagation.

## Input / output format

**Input**: Fine-tuning datasets (Fed-CodeAlpaca, Fed-Dolly, Fed-GSM8K-3) partitioned across clients; evaluation instances from HumanEval, HELM, and GSM8K-test.

**Output**: Model-generated text/code/math solutions; evaluated against gold labels to compute percentage scores or Pass@1.

## Scoring recipe

```python
def compute_protocol(predictions, gold, peft_algo, fl_algo, scenario):
    scores = []
    for seed in [1, 2, 3]:
        model = init_llm('LLaMA-7B')
        model = fine_tune(model, dataset, peft_algo, fl_algo, seed)
        preds = model.predict(evaluation_set)
        if peft_algo == 'LoRA' and task == 'code':
            score = pass_at_1(preds, gold)
        else:
            score = exact_match_or_accuracy(preds, gold)
        scores.append(score)
    return mean(scores), std(scores)
```

## Common pitfalls

- Confusing the three training scenarios: Global (centralized), Fed (federated aggregation), and Local (independent client training).
- Half-precision training causes precision loss that disproportionately harms pFedMe compared to FedAvg.
- OPT-2.7B fails on HELM subtasks due to input length limits, requiring exclusion from final scores.

## Evidence (verbatim from paper)

> The averaged evaluation scores (Pass@1 scores) with their standard deviation are reported. We note that evaluation scores with pFedMe are obtained by benchmarking each personalized client individually and then computing their average scores.

## Citation

```bibtex
@misc{kuang2023federatedscollm,
  title={FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning},
  author={Kuang et al. (2023)},
  year={2023},
  note={arXiv:2309.00363}
}
```

- arXiv: 2309.00363