C3 Bench Eval

---
name: c3-bench-eval
description: Evaluates the robustness and multi-tasking capabilities of LLM-based agents by probing their ability to handle complex tool dependencies, propagate hidden information across tasks, and maintain stable decision policies under dynamic, multi-round interactions. Use when the user wants to benchmark on C^3-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.18746
  bibtex_key: yu2025c3bench
  confidence: high
---

# c3-bench-eval

> $C^3$-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking — Yu et al. (2025) (arXiv:2505.18746, 2025)

## What this evaluates

Evaluates the robustness and multi-tasking capabilities of LLM-based agents by probing their ability to handle complex tool dependencies, propagate hidden information across tasks, and maintain stable decision policies under dynamic, multi-round interactions.

## Datasets

- **C^3-Bench** — total 1024; splits: test (1024); repo https://github.com/Tencent-Hunyuan/C3-Benchmark

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly completed multi-task instances out of the total test set, calculated as (correct predictions / total instances) * 100.
- `accomplish_progress (AP)` — range: percent
  - A soft metric measuring the degree of task completion progress, providing finer-grained assessment than binary accuracy.
- `optimal_path_rate (OP Rate)` — range: percent
  - Rate at which the model's policy follows the optimal execution path for tool invocations.

## Input / output format

**Input**: Multi-task prompts containing tool definitions, hidden/critical information, and dynamic decision histories requiring sequential or parallel tool calls across 1-4 tasks.

**Output**: Sequential or parallel tool invocation sequences followed by final task answers.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_inst in zip(predictions, gold):
        if pred.matches_exact(gold_inst):
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Confusing single-task (256) and multi-task (768) subsets when reporting overall performance or comparing model rankings.
- Overlooking the distinction between parallel ($P_{multi}^P$) and sequential ($P_{multi}^S$) tool call dependencies, which cause different accuracy drops and require separate evaluation.
- Ignoring policy transition frequency (PTF) and volatility (VF) when diagnosing model instability, as high PTF sensitivity directly correlates with accuracy degradation in multi-task settings.

## Evidence (verbatim from paper)

> The $C^{3}$-Bench has 256 single-task and 768 multi-task testdata. Challenge 1 contains 256 entries with 1-4 tasks, while Challenges 2 and 3 contain 768 multi-task data with 2-4 tasks. This section reports the accuracy of 19 representative models. For each model series, we show only the best result. Testing follows Challenge 3’s method.

## Citation

```bibtex
@misc{yu2025c3bench,
  title={$C^3$-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2505.18746}
}
```

- arXiv: 2505.18746