Lora Wise Eval

---
name: lora-wise-eval
description: This benchmark probes a model's ability to infer the exact number of training images used to fine-tune a Low-Rank Adaptation (LoRA) adapter solely from its learned weight matrices. It evaluates how well spectral and norm-based features of LoRA parameters correlate with and reveal the scale of the underlying training dataset. Use when the user wants to benchmark on LoRA-WiSE, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.19395
  bibtex_key: [REDACTED]
  confidence: high
---

# lora-wise-eval

> Dataset Size Recovery from LoRA Weights — Salama et al. (2024) (arXiv:2406.19395, 2024)

## What this evaluates

This benchmark probes a model's ability to infer the exact number of training images used to fine-tune a Low-Rank Adaptation (LoRA) adapter solely from its learned weight matrices. It evaluates how well spectral and norm-based features of LoRA parameters correlate with and reveal the scale of the underlying training dataset.

## Datasets

- **LoRA-WiSE** — total ?; splits: train (165), test (35)

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: the average of the absolute differences between predicted and true dataset sizes. Formula: (1/N) * Σ|y_pred - y_true|.
- `MAPE` — range: percent
  - Mean Absolute Percentage Error: the average of the absolute percentage errors relative to the ground truth. Formula: (1/N) * Σ(|y_pred - y_true| / y_true) * 100.
- `Accuracy` — range: percent
  - Exact-match accuracy: the percentage of predictions that exactly equal the true dataset size.

## Input / output format

**Input**: LoRA adapter weight matrices (specifically, the singular values or Frobenius norms of the weight matrices) from a fine-tuned Stable Diffusion model.

**Output**: A single integer representing the predicted number of fine-tuning images used to train the adapter.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth):
    n = len(predictions)
    mae = sum(abs(p - g) for p, g in zip(predictions, ground_truth)) / n
    mape = sum(abs(p - g) / g for p, g in zip(predictions, ground_truth)) / n * 100
    accuracy = sum(1 for p, g in zip(predictions, ground_truth) if p == g) / n * 100
    return {'MAE': mae, 'MAPE': mape, 'Accuracy': accuracy}
```

## Common pitfalls

- Accuracy is explicitly noted as inadequate for this task because predicting a nearby size (e.g., 4 vs 5) is penalized equally to a large error, making MAPE a more appropriate metric for relative performance.
- The baseline (Frobenius-NN) and proposed method (DSiRe) are fitted separately per layer and combined via majority vote, so evaluating a single-layer prediction or ignoring the ensemble voting rule will yield incorrect results.
- Experiments are repeated 10 times with different subset samplings; reporting a single run without averaging and standard deviation over these repeats misrepresents the reported performance.

## Evidence (verbatim from paper)

> As described in Sec. [4.1], our main evaluation metric is Mean Absolute Error (MAE). For completeness, we choose to report two complementary metrics as well: (i) Accuracy. (ii) Mean Absolute Percentage Error (MAPE). Since DSiRe predicts dataset sizes, simple accuracy does not adequately measure its effectiveness, e.g., predicting 4 when the true value is 5 is not as bad as predicting 1. We therefore provide MAPE scores as well, which compute the percentile from the ground truth that is equal to the absolute error.

## Citation

```bibtex
@misc{salama2024datasetsizerecovery,
  title={Dataset Size Recovery from LoRA Weights},
  author={Salama et al. (2024)},
  year={2024},
  note={arXiv:2406.19395}
}
```

- arXiv: 2406.19395