Cpt Merging Eval

---
name: cpt-merging-eval
description: Evaluates the effectiveness of merging Continual Pretraining (CPT) models to build domain-specialized financial LLMs. It probes the model's ability to recover lost general knowledge, exhibit cross-domain complementarity, and demonstrate emergent reasoning capabilities through weight-space integration. Use when the user wants to benchmark on Financial Benchmark, or asks about evaluating this task. Reports Macro-Gain.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.02451
  bibtex_key: ueda2025merging
  confidence: high
---

# cpt-merging-eval

> Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance — Ueda et al. (2025) (arXiv:2511.02451, 2025)

## What this evaluates

Evaluates the effectiveness of merging Continual Pretraining (CPT) models to build domain-specialized financial LLMs. It probes the model's ability to recover lost general knowledge, exhibit cross-domain complementarity, and demonstrate emergent reasoning capabilities through weight-space integration.

## Datasets

- **Financial Benchmark** — total 18; splits: test (-1)

## Metrics

- `Macro-Gain` **(primary)** — range: percent
  - The average performance improvement of the merged model over the arithmetic mean of its two constituent models across all tasks.
- `Macro-OG` — range: percent
  - The average margin by which the merged model outperforms the best of its two constituent models across all tasks.

## Input / output format

**Input**: Task prompts from a curated 18-task financial benchmark.

**Output**: Model-generated predictions or answers for each benchmark task.

## Scoring recipe

```python
def compute_metrics(merged_scores, c1_scores, c2_scores):
    avg_merged = np.mean(merged_scores)
    avg_c1 = np.mean(c1_scores)
    avg_c2 = np.mean(c2_scores)
    avg_const_avg = (avg_c1 + avg_c2) / 2
    best_const = max(avg_c1, avg_c2)
    macro_gain = avg_merged - avg_const_avg
    macro_og = avg_merged - best_const
    return macro_gain, macro_og
```

## Common pitfalls

- Task Arithmetic (TA) is highly sensitive to the scaling hyperparameter λ, often degrading if poorly tuned, whereas TIES (TI) is more robust.
- Merging three CPT models frequently causes performance collapse and representational interference, contrary to additive expectations.
- Emergent capabilities (positive Macro-OG) do not correlate linearly with parameter-space similarity (L2/cosine), making them hard to predict a priori.

## Evidence (verbatim from paper)

> We first observe that Macro-Gain tends to be higher when models are more similar—i.e., when L2 distance is smaller or cosine similarity is larger.

## Citation

```bibtex
@misc{ueda2025merging,
  title={Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance},
  author={Ueda et al. (2025)},
  year={2025},
  note={arXiv:2511.02451}
}
```

- arXiv: 2511.02451