Muse Eval

---
name: muse-eval
description: Evaluates an LLM-based planning framework's ability to decompose natural language queries into correct task selections, logical execution flows, and valid final multimodal outputs. It probes constraint-aware model orchestration and multi-modal task routing across heterogeneous AI services. Use when the user wants to benchmark on MuSE, or asks about evaluating this task. Reports Task Selection (TS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.12839
  bibtex_key: vyas2024hive
  confidence: high
---

# muse-eval

> From An LLM Swarm To A PDDL-Empowered HIVE: Planning Self-Executed Instructions In A Multi-Modal Jungle — Vyas et al. (2024) (arXiv:2412.12839, 2024)

## What this evaluates

Evaluates an LLM-based planning framework's ability to decompose natural language queries into correct task selections, logical execution flows, and valid final multimodal outputs. It probes constraint-aware model orchestration and multi-modal task routing across heterogeneous AI services.

## Datasets

- **MuSE** — total 100; splits: test (100); repo https://github.com/dgraux/Hive-ICLR-2025

## Metrics

- `Task Selection (TS)` **(primary)** — range: [0, 1]
  - Binary score (0 or 1) averaged across queries. Assigned 1 if the system selects exactly the required tasks, 0 otherwise.
- `Flow of Thought (FoT)` — range: [0, 1]
  - Binary score (0 or 1) averaged across queries. Assigned 1 if the system establishes the correct logical sequence and respects task dependencies, 0 otherwise.
- `Final Output (O)` — range: [0, 1]
  - Binary score (0 or 1) averaged across queries. Assigned 1 if the output fulfills the user's requirements, 0 otherwise. Does not evaluate output quality or factual accuracy.

## Input / output format

**Input**: Natural language user queries specifying 1–3 tasks across text, image, and audio modalities.

**Output**: Structured execution plan containing selected tasks, their dependency order, and the final multimodal output generated by the orchestrated models.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    ts_scores, fot_scores, o_scores = [], [], []
    for pred, gold in zip(predictions, gold):
        ts_scores.append(1 if pred.selected_tasks == gold.required_tasks else 0)
        fot_scores.append(1 if pred.task_order == gold.required_order else 0)
        o_scores.append(1 if pred.output_fulfills_intent else 0)
    return {
        'TS': sum(ts_scores) / len(ts_scores),
        'FoT': sum(fot_scores) / len(fot_scores),
        'O': sum(o_scores) / len(o_scores)
    }
```

## Common pitfalls

- Evaluation focuses strictly on whether the expected task was performed, not on the factual accuracy or quality of the generated output.
- Baseline systems like HuggingGPT may over-select tasks to ensure coverage, artificially inflating Final Output scores while penalizing Task Selection and Flow of Thought metrics.
- ControlLLM relies on a fine-tuned task decomposer, causing significant performance drops on multi-task queries that deviate from its training distribution.

## Evidence (verbatim from paper)

> To assess our framework against state-of-the-art methods, we evaluate performance on three fronts, using binary metrics for simplicity and clarity: Task Selection (TS): Determines whether the system accurately identifies the required tasks from the user’s query. We assign a binary score of 1 if the system selects all the tasks correctly, and 0 if it does not or if it selected irrelevant tasks. Flow of Thought (FoT): We evaluate the logical sequence and integration of the selected tasks. A binary score is given based on whether the system establishes the correct flow—1 for a proper flow that respects task dependencies and order, and 0 for an incorrect sequence. Final Output (O): Assesses the correctness of the system’s final response to the user’s query. We adopt a binary evaluation—1 if the output fulfills the user’s requirements, and 0 if it falls short.

## Citation

```bibtex
@misc{vyas2024hive,
  title={From An LLM Swarm To A PDDL-Empowered HIVE: Planning Self-Executed Instructions In A Multi-Modal Jungle},
  author={Vyas et al. (2024)},
  year={2024},
  note={arXiv:2412.12839}
}
```

- arXiv: 2412.12839