Mti Bench Eval

---
name: mti-bench-eval
description: Evaluates whether large language models can process multiple distinct instructions simultaneously within a single inference call, compared to sequential or batched approaches. It probes reasoning consistency, format adherence, and inference efficiency across a diverse set of 28 NLP tasks. Use when the user wants to benchmark on MTI Bench, or asks about evaluating this task. Reports exact match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.11597
  bibtex_key: son2024multi
  confidence: high
---

# mti-bench-eval

> Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once? — Son et al. (2024) (arXiv:2402.11597, 2024)

## What this evaluates

Evaluates whether large language models can process multiple distinct instructions simultaneously within a single inference call, compared to sequential or batched approaches. It probes reasoning consistency, format adherence, and inference efficiency across a diverse set of 28 NLP tasks.

## Datasets

- **MTI Bench** — total 5000; splits: test (-1); repo https://github.com/guijinSON/MTI-Bench

## Metrics

- `exact match (EM)` **(primary)** — range: [0, 1]
  - Exact match between the model's generated output inside the designated HTML tag and the ground truth answer. Scores are averaged across all sub-tasks and instances.

## Input / output format

**Input**: A single prompt containing multiple distinct instructions (sub-tasks) to be solved concurrently, formatted with clear task delimiters.

**Output**: Model generates responses for each sub-task enclosed in specific HTML tags (e.g., <task1>answer<task1/>), with one tag per sub-task in the order presented.

## Scoring recipe

```python
def compute_em(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_clean = extract_html_tag_content(pred)
        if pred_clean == gold:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Strict exact-match scoring penalizes minor formatting variations (e.g., extra spaces, different number formats) even if the semantic answer is correct.
- Models often fail to strictly follow the HTML tag formatting instruction, outputting text outside the tags or mismatching tag names, which breaks automated parsing.
- Confusing Multi-Task Inference (multiple sub-tasks per instance) with Batch Prompting (multiple independent instances per call), leading to incorrect baseline comparisons.

## Evidence (verbatim from paper)

> Therefore, we prompted LLMs to return their outputs within an HTML tag (e.g., <task1>output<task1/>), which is then assessed via exact match (EM).

## Citation

```bibtex
@misc{son2024multi,
  title={Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?},
  author={Son et al. (2024)},
  year={2024},
  note={arXiv:2402.11597}
}
```

- arXiv: 2402.11597