Silicone Eval

---
name: silicone-eval
description: Evaluates a model's ability to perform sequence labelling on spoken dialogues, specifically predicting dialog acts (DA) and emotion/sentiment (E/S) labels per utterance within multi-utterance conversations. Use when the user wants to benchmark on SILICONE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2009.11152
  bibtex_key: chapuis2020hierarchical
  confidence: high
---

# silicone-eval

> Hierarchical Pre-training for Sequence Labelling in Spoken Dialog — Chapuis et al. (2020) (arXiv:2009.11152, 2020)

## What this evaluates

Evaluates a model's ability to perform sequence labelling on spoken dialogues, specifically predicting dialog acts (DA) and emotion/sentiment (E/S) labels per utterance within multi-utterance conversations.

## Datasets

- **SILICONE** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the proportion of correctly predicted labels out of the total number of utterance-level predictions. Reported as average accuracy across tasks or subgroups (Avg DA, Avg E/S).

## Input / output format

**Input**: Multi-utterance spoken dialog conversations (sequences of utterances).

**Output**: Per-utterance sequence of predicted labels (Dialog Act or Emotion/Sentiment category).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Using sequential decoders (CRF/GRU) instead of the recommended plain MLP decoder, which can lead to unfair comparisons or poor performance due to limited training data.
- Ignoring the multi-utterance hierarchical structure of the input, which is critical for capturing discourse-level dependencies in spoken dialog.
- Assuming equal difficulty across tasks; E/S tasks consistently underperform DA tasks due to differences in corpus size and utterances-per-label ratios.

## Evidence (verbatim from paper)

> Table 4 provides an exhaustive comparison of the different encoders over the SILICONE benchmark. As previously discussed, we adopt a plain MLP as a decoder to compare the different encoders. We show that SILICONE covers a set of challenging tasks as the best performing model achieves an average accuracy of 74.3.

## Citation

```bibtex
@misc{chapuis2020hierarchical,
  title={Hierarchical Pre-training for Sequence Labelling in Spoken Dialog},
  author={Chapuis et al. (2020)},
  year={2020},
  note={arXiv:2009.11152}
}
```

- arXiv: 2009.11152