Iccma17 Eval

---
name: iccma17-eval
description: Evaluates computational argumentation solvers on their ability to correctly compute extensions (e.g., semi-stable, stage, ideal) across diverse argumentation frameworks ranging from random graphs to application-derived structures. Use when the user wants to benchmark on ICCMA'17 Benchmark Suite, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1909.00621
  bibtex_key: gaggl2019iccma17
  confidence: high
---

# iccma17-eval

> Design and Results of the Second International Competition on Computational Models of Argumentation — Gaggl et al. (2019) (arXiv:1909.00621, 2019)

## What this evaluates

Evaluates computational argumentation solvers on their ability to correctly compute extensions (e.g., semi-stable, stage, ideal) across diverse argumentation frameworks ranging from random graphs to application-derived structures.

## Datasets

- **ICCMA'17 Benchmark Suite** — total 3990; splits: test (3990)

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - 1 if the solver's predicted extension exactly matches the ground-truth extension for the given semantics, 0 otherwise. Averaged across all instances in the benchmark suite.

## Input / output format

**Input**: Argumentation Framework (AF) represented as a set of arguments and a binary attack relation between them.

**Output**: Computed extension (set of arguments) or complete labeling according to a specified semantics.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if set(p) == set(g))
    return correct / len(gold)
```

## Common pitfalls

- Different semantics (semi-stable, stage, ideal) have distinct extension definitions; evaluating across them requires separate ground-truth labels.
- Instance generation parameters (e.g., n, probAttacks) vary widely and must be fixed to ensure reproducibility across runs.
- Runtime/efficiency is often tracked in competitions but is distinct from correctness accuracy.

## Evidence (verbatim from paper)

> The second International Competition on Computational Models of Argumentation (ICCMA'17) introduces three new semantics—semi-stable, stage, and ideal—and implements a novel scoring scheme, instance selection process, and a 'Dung's Triathlon' track that evaluates solvers across multiple complexity layers. Thus, the benchmark suite of ICCMA'17 is finally composed of 3990 instances over 11 domains.

## Citation

```bibtex
@misc{gaggl2019iccma17,
  title={Design and Results of the Second International Competition on Computational Models of Argumentation},
  author={Gaggl et al. (2019)},
  year={2019},
  note={arXiv:1909.00621}
}
```

- arXiv: 1909.00621