Tovo Consensus Eval

---
name: tovo-consensus-eval
description: This evaluation probes a model's ability to classify text content according to a user-defined toxicity taxonomy. It measures how closely the model's predictions align with gold labels generated through a multi-model voting process, and tests generalization to out-of-domain categories. Use when the user wants to benchmark on ToVo, or asks about evaluating this task. Reports consensus rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.14835
  bibtex_key: luong2024tovo
  confidence: high
---

# tovo-consensus-eval

> ToVo: Toxicity Taxonomy via Voting — Luong et al. (2024) (arXiv:2406.14835, 2024)

## What this evaluates

This evaluation probes a model's ability to classify text content according to a user-defined toxicity taxonomy. It measures how closely the model's predictions align with gold labels generated through a multi-model voting process, and tests generalization to out-of-domain categories.

## Datasets

- **ToVo** — total ?; splits: train (10000), test_toxicity (2322), test_ood (1741)

## Metrics

- `consensus rate` **(primary)** — range: percent
  - The percentage of exact matches between the model's predicted label and the gold label derived from the voting process. Calculated as (number of agreements / total samples) * 100.

## Input / output format

**Input**: Raw text content to be evaluated for toxicity or out-of-domain category membership.

**Output**: Classification label (e.g., toxic/non-toxic or specific metric category), optionally accompanied by a chain-of-thought rationale.

## Scoring recipe

```python
def compute_consensus_rate(predictions, gold_labels):
    if len(predictions) != len(gold_labels):
        raise ValueError('Length mismatch')
    agreements = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (agreements / len(predictions)) * 100
```

## Common pitfalls

- A lower consensus rate for a specific metric does not necessarily indicate poor model performance or flawed criteria; it may simply reflect a different toxicity tolerance threshold compared to the reference API.
- The metric measures alignment with a voting-based gold standard rather than ground-truth human annotations, so high consensus does not guarantee factual correctness.
- The evaluation compares reasoning and non-reasoning models, but the exact prompt templates required for reproduction are only provided in the appendix.

## Evidence (verbatim from paper)

> Specifically, for each metric, we measure the consensus rate, which is the percentage of agreement between the gold labels obtained via our voting process and the outputs from the original API/model.

## Citation

```bibtex
@misc{luong2024tovo,
  title={ToVo: Toxicity Taxonomy via Voting},
  author={Luong et al. (2024)},
  year={2024},
  note={arXiv:2406.14835}
}
```

- arXiv: 2406.14835