Insure Dial Eval

---
name: insure-dial-eval
description: Probes phase-aware compliance verification and phase boundary detection in insurance benefit verification calls. It measures a model’s ability to accurately segment conversational phases under workflow-specific rules and apply rule-based compliance reasoning (Information and Procedural Compliance) to fixed spans. Use when the user wants to benchmark on INSURE-Dial, or asks about evaluating this task. Reports exact match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.18448
  bibtex_key: [REDACTED]
  confidence: high
---

# insure-dial-eval

> INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection — Kulkarni et al. (2026) (arXiv:2602.18448, 2026)

## What this evaluates

Probes phase-aware compliance verification and phase boundary detection in insurance benefit verification calls. It measures a model’s ability to accurately segment conversational phases under workflow-specific rules and apply rule-based compliance reasoning (Information and Procedural Compliance) to fixed spans.

## Datasets

- **INSURE-Dial** — total 1050; splits: real (50), synthetic (1000)

## Metrics

- `exact match (EM)` **(primary)** — range: percent
  - Fraction of calls where all phases satisfy the task acceptance rules (Task 1) or conjunction over applicable phases (Task 2).
- `turn-level F1` — range: [0, 1]
  - Overlap metric measuring turn-level span alignment between predicted and reference boundaries.
- `SAD` — range: other
  - Sum of absolute differences between predicted and reference span boundaries; lower indicates less boundary drift.
- `phase-level accuracy (Acc)` — range: percent
  - Fraction of phases correctly classified as compliant or non-compliant given fixed spans.
- `macro-F1` — range: [0, 1]
  - Macro-averaged F1 score computed over non-"NA" phases.
- `Hit` — range: percent
  - Mean fraction of applicable phases correctly identified per call.

## Input / output format

**Input**: Conversational transcript (real or synthetic) annotated with phase-structured JSON schemas capturing information and procedural compliance across IVR, patient identification, coverage status, drug checks, and agent identification.

**Output**: Predicted span boundaries for each phase (Task 1) and compliance labels (IC/PC) for fixed spans (Task 2), structured according to the phase-annotated JSON schema.

## Scoring recipe

```python
# Task 1: Phase Boundary Detection
call_em = 1.0
sad = 0
for pred_span, gold_span in zip(pred_spans, gold_spans):
    if pred_span != gold_span: call_em = 0.0
    sad += abs(len(pred_span) - len(gold_span))
f1 = compute_turn_level_overlap(pred_spans, gold_spans)

# Task 2: Compliance Verification (given fixed spans)
applicable = [i for i, m in enumerate(applicability_mask) if m]
correct = sum(pred_labels[i] == gold_labels[i] for i in applicable)
acc = correct / len(applicable)
hit = correct / len(applicable)
call_acc = 1.0 if correct == len(applicable) else 0.0
macro_f1 = compute_macro_f1(pred_labels, gold_labels, exclude_na=True)
```

## Common pitfalls

- Call-level EM collapses due to multiplicative strictness across 8–10 phases; a single boundary slip fails the entire call.
- IC and PC call-level accuracies are computed over different overall fields with different applicability patterns, making them incomparable across tasks.
- Synthetic data omits long hold-time chatter and IVR repetitions, inflating segmentation scores compared to real calls.

## Evidence (verbatim from paper)

> For Task 1 we report exact match (EM) of the predicted span under §4.1 rules, turn-level overlap via  $F_{1}$ , and sum of absolute differences (SAD) as a boundary-drift measure (lower is better). Call-level EM is the fraction of calls where all phases (including correctly predicting null when absent) satisfy the Task 1 acceptance rules. For Task 2 we report phase-level accuracy (Acc) and macro- $F_{1}$  over non-"NA" phases, plus Hit, the mean fraction of applicable phases correct per call

## Citation

```bibtex
@misc{kulkarni2026insuredial,
  title={INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection},
  author={Kulkarni et al. (2026)},
  year={2026},
  note={arXiv:2602.18448}
}
```

- arXiv: 2602.18448