Edge Case Detection Eval

---
name: edge-case-detection-eval
description: Tests the model's ability to classify whether a respondent's message represents an edge case that falls outside the scope of existing coordination policies and requires user escalation. Use when the user wants to benchmark on Edge Case Detection Test Suite, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.12626
  bibtex_key: long2025doubleagents
  confidence: high
---

# edge-case-detection-eval

> DoubleAgents: Exploring Mechanisms of Building Trust with Proactive AI — Long et al. (2025) (arXiv:2509.12626, 2025)

## What this evaluates

Tests the model's ability to classify whether a respondent's message represents an edge case that falls outside the scope of existing coordination policies and requires user escalation.

## Datasets

- **Edge Case Detection Test Suite** — total 100; splits: test (100)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified edge cases (binary classification) out of the total test examples.

## Input / output format

**Input**: A respondent's message, optionally accompanied by few-shot demonstration examples and/or a reference list of existing coordination policies.

**Output**: Binary classification label indicating whether the message is an edge case requiring escalation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Zero-shot prompting without demonstrations performs significantly worse (~0.79) than few-shot approaches (~0.88-0.93).
- Experiments are run with GPT-4o at temperature=0.7 and max_tokens=100; changing these may alter performance.

## Evidence (verbatim from paper)

> we constructed a test suite of 100 examples, manually validated by the lead authors to ensure coverage and quality. The construction methodology for this test suite is detailed in Appendix[B.2]. All experiments were conducted using GPT-4o, with temperature set to 0.7 and a maximum token limit of 100. We report the average accuracy across three independent runs for each condition.

## Citation

```bibtex
@misc{long2025doubleagents,
  title={DoubleAgents: Exploring Mechanisms of Building Trust with Proactive AI},
  author={Long et al. (2025)},
  year={2025},
  note={arXiv:2509.12626}
}
```

- arXiv: 2509.12626