Policy Selection Eval

---
name: policy-selection-eval
description: Evaluates the model's ability to retrieve relevant coordination policies that guide task planning based on a high-level progress summary of the current state. Use when the user wants to benchmark on Policy Selection Test Suite, or asks about evaluating this task. Reports F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.12626
  bibtex_key: long2025doubleagents
  confidence: high
---

# policy-selection-eval

> DoubleAgents: Exploring Mechanisms of Building Trust with Proactive AI — Long et al. (2025) (arXiv:2509.12626, 2025)

## What this evaluates

Evaluates the model's ability to retrieve relevant coordination policies that guide task planning based on a high-level progress summary of the current state.

## Datasets

- **Policy Selection Test Suite** — total 30; splits: test (30)

## Metrics

- `F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall computed over the set of retrieved policies versus the ground truth set of user-approved policies.

## Input / output format

**Input**: A high-level progress summary of the task state (e.g., speaker responsiveness, slot availability, and outreach history) generated by the getProgressSummary() API.

**Output**: A set of applicable coordination policies retrieved from a curated seed set.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    pred_set = set(predictions)
    gold_set = set(gold)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Raw aggregated context (speaker + slot) without summarization yields very low F1 (~0.27) compared to summary-based prompting (~0.70).
- Evaluations use temperature=1.0 with o4-mini, which may increase variability; results are averaged over 3 runs.

## Evidence (verbatim from paper)

> we constructed a test suite of 30 examples, all sourced from real user study sessions and curated to reflect a diverse set of scheduling scenarios. For each example, we treat the set of policies approved by users during the study as the ground truth for evaluation. All evaluations were conducted using the o4-mini model with a default temperature of 1. We report the average F1 score across three independent runs for each experimental setting.

## Citation

```bibtex
@misc{long2025doubleagents,
  title={DoubleAgents: Exploring Mechanisms of Building Trust with Proactive AI},
  author={Long et al. (2025)},
  year={2025},
  note={arXiv:2509.12626}
}
```

- arXiv: 2509.12626