Forkmerge Eval

---
name: forkmerge-eval
description: Evaluates the ability of auxiliary-task learning methods to mitigate negative transfer and improve target task performance across multi-task, multi-domain, and semi-supervised learning settings. It probes how well a model can dynamically combine or select auxiliary tasks without degrading the primary task. Use when the user wants to benchmark on NYUv2, DomainNet, AliExpress, CIFAR-10, SVHN, or asks about evaluating this task. Reports Δm.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.12618
  bibtex_key: jiang2023forkmerge
  confidence: high
---

# forkmerge-eval

> ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning — Jiang et al. (2023) (arXiv:2301.12618, 2023)

## What this evaluates

Evaluates the ability of auxiliary-task learning methods to mitigate negative transfer and improve target task performance across multi-task, multi-domain, and semi-supervised learning settings. It probes how well a model can dynamically combine or select auxiliary tasks without degrading the primary task.

## Datasets

- **NYUv2** — total 899; splits: train (636), val (159), test (654)
- **DomainNet** — total ?; splits: train (-1), val (-1), test (-1)
- **AliExpress** — total ?; splits: train (-1)
- **CIFAR-10** — total ?; splits: test (-1)
- **SVHN** — total ?; splits: test (-1)

## Metrics

- `Δm` **(primary)** — range: percent
  - Average per-task performance improvement across all evaluated tasks. Computed by averaging the relative or absolute improvement of each method over the Single Task Learning (STL) baseline. Exact normalization formula is defined in Appendix C.1.

## Input / output format

**Input**: RGB images for vision tasks (NYUv2, DomainNet, CIFAR-10, SVHN); tabular/sequence features for recommendation tasks (AliExpress).

**Output**: Task-specific predictions: segmentation masks, depth/normal maps, class labels, or click-through rate probabilities.

## Scoring recipe

```python
improvements = []
for task in tasks:
    baseline = evaluate(model_STL, dataset[task])
    method = evaluate(model_method, dataset[task])
    if metric_is_error(task):
        improvement = (baseline - method) / baseline
    else:
        improvement = (method - baseline) / baseline
    improvements.append(improvement)
delta_m = sum(improvements) / len(improvements) * 100
```

## Common pitfalls

- Negative transfer is algorithm-dependent, not just task-dependent; results vary significantly across different ATL baselines.
- DomainNet lacks a provided validation set, requiring a random 50% test split for hyperparameter tuning, which may introduce variance.
- Aggregating diverse metrics (accuracy, IoU, error rates) into Δm requires careful normalization to avoid scale bias across tasks.

## Evidence (verbatim from paper)

> Since different tasks have varying evaluation metrics, we will report the average per-task performance improvement for each method using  Δm , as defined in Appendix C.1.

## Citation

```bibtex
@misc{jiang2023forkmerge,
  title={ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning},
  author={Jiang et al. (2023)},
  year={2023},
  note={arXiv:2301.12618}
}
```

- arXiv: 2301.12618