Compositional Multitasking Eval

---
name: compositional-multitasking-eval
description: Evaluates the ability of on-device LLMs to perform two distinct tasks simultaneously in a single forward pass (compositional multi-tasking), such as summarization combined with translation or tone adjustment, while maintaining strict efficiency constraints. Use when the user wants to benchmark on Compositional Multi-tasking Benchmark, or asks about evaluating this task. Reports LLM judge (LLM-J).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.16083
  bibtex_key: [REDACTED]
  confidence: high
---

# compositional-multitasking-eval

> Efficient Compositional Multi-tasking for On-device Large Language Models — Bohdal et al. (2025) (arXiv:2507.16083, 2025)

## What this evaluates

Evaluates the ability of on-device LLMs to perform two distinct tasks simultaneously in a single forward pass (compositional multi-tasking), such as summarization combined with translation or tone adjustment, while maintaining strict efficiency constraints.

## Datasets

- **Compositional Multi-tasking Benchmark** — total ?; splits: test (-1)

## Metrics

- `ROUGE-L (R-L)` — range: percent
  - Standard ROUGE-L score measuring the longest common subsequence overlap between generated and reference text, reported as a percentage.
- `Weighted ROUGE (W-R)` — range: percent
  - Weighted variant of ROUGE that emphasizes longer n-grams or specific task requirements, reported as a percentage.
- `LLM judge (LLM-J)` **(primary)** — range: percent
  - An LLM-as-a-judge scoring metric that evaluates whether the model successfully completed both compositional tasks in the output, reported as a percentage.

## Input / output format

**Input**: Conversational prompts requiring the model to execute two tasks simultaneously (e.g., summarize a text and translate it, or generate a reply with a specific tone).

**Output**: Generated text response.

## Scoring recipe

```python
def compute_metrics(predictions, references, task_pairs):
    rouge_l_scores = [rouge_l_score(p, r) for p, r in zip(predictions, references)]
    weighted_rouge_scores = [weighted_rouge_score(p, r) for p, r in zip(predictions, references)]
    llm_judge_scores = [llm_judge_score(p, r, task) for p, r, task in zip(predictions, references, task_pairs)]
    return {
        'ROUGE-L (R-L)': mean(rouge_l_scores) * 100,
        'Weighted ROUGE (W-R)': mean(weighted_rouge_scores) * 100,
        'LLM judge (LLM-J)': mean(llm_judge_scores) * 100
    }
```

## Common pitfalls

- Zero-shot and in-context learning perform poorly on compositional tasks without fine-tuning.
- Standard adapter merging strategies (e.g., linear, TIES, DARE) fail to handle compositional multi-tasking effectively.
- ROUGE scores alone are less interpretable; LLM judge scores are needed to assess task completion quality.

## Evidence (verbatim from paper)

> The metrics include ROUGE-L (R-L), Weighted ROUGE (W-R) and LLM judge (LLM-J) scores. The Efficient? column captures both runtime and storage efficiency. While ROUGE scores are less interpretable in isolation, the analysis of LLM judge scores provided useful insights.

## Citation

```bibtex
@misc{bohdal2025efficientcompositional,
  title={Efficient Compositional Multi-tasking for On-device Large Language Models},
  author={Bohdal et al. (2025)},
  year={2025},
  note={arXiv:2507.16083}
}
```

- arXiv: 2507.16083