Multi Turn Task Exec Eval

---
name: multi-turn-task-exec-eval
description: Assesses an agent's ability to execute complex, multi-turn voice-driven tasks requiring tool invocation, memory, and reasoning across varying difficulty levels. Use when the user wants to benchmark on Custom Multi-Turn Voice Tasks, or asks about evaluating this task. Reports Avg. Success.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.23049
  bibtex_key: maben2025aura
  confidence: high
---

# multi-turn-task-exec-eval

> AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks — Maben et al. (2025) (arXiv:2506.23049, 2025)

## What this evaluates

Assesses an agent's ability to execute complex, multi-turn voice-driven tasks requiring tool invocation, memory, and reasoning across varying difficulty levels.

## Datasets

- **Custom Multi-Turn Voice Tasks** — total 30; splits: test (30); repo https://github.com/Sentientia/Aura

## Metrics

- `Avg. Success` **(primary)** — range: percent
  - Percentage of tasks successfully completed as rated by a human evaluator.
- `Avg. Satisfaction` — range: other
  - User satisfaction rated on a 1 to 5 scale by a human evaluator.

## Input / output format

**Input**: Multi-turn spoken dialogue with compound goals requiring tool use and state tracking.

**Output**: Agent actions (tool calls, responses) and final task completion status.

## Scoring recipe

```python
def evaluate_tasks(interactions):
    success_scores = []
    satisfaction_scores = []
    for task in interactions:
        success_scores.append(human_evaluator.rate_success(task))
        satisfaction_scores.append(human_evaluator.rate_satisfaction(task))
    return sum(success_scores) / len(success_scores), sum(satisfaction_scores) / len(satisfaction_scores)
```

## Common pitfalls

- Human evaluation by a single co-author may lack inter-rater reliability.
- Task difficulty categorization (Easy/Medium/Hard) is subjective and not formally defined in the text.
- Scores are averaged per difficulty level, which may mask performance on specific task types.

## Evidence (verbatim from paper)

> We evaluated AURA on 30 real-world tasks grouped by difficulty—Easy, Medium, and Hard—with harder tasks involving compound goals requiring multi-turn reasoning, memory, and tool use. A human evaluator (co-author) rates each interaction on difficulty, success, and satisfaction (1–5 scale); task details and scores are available in the repository.

## Citation

```bibtex
@misc{maben2025aura,
  title={AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks},
  author={Maben et al. (2025)},
  year={2025},
  note={arXiv:2506.23049}
}
```

- arXiv: 2506.23049