Orchid Eval

---
name: orchid-eval
description: Evaluates models on target-independent stance detection (3-way classification) and argumentative dialogue summarization (overall and stance-specific). It probes the ability to classify conflicting viewpoints in Chinese debates and generate concise, faithful summaries aligned with specific stances. Use when the user wants to benchmark on OrChiD, or asks about evaluating this task. Reports Accuracy, ROUGE-1 F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.13667
  bibtex_key: zhao2024orchid
  confidence: high
---

# orchid-eval

> ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization — Zhao et al. (2024) (arXiv:2410.13667, 2024)

## What this evaluates

Evaluates models on target-independent stance detection (3-way classification) and argumentative dialogue summarization (overall and stance-specific). It probes the ability to classify conflicting viewpoints in Chinese debates and generate concise, faithful summaries aligned with specific stances.

## Datasets

- **OrChiD** — total 14091; splits: train (11005), dev (1534), test (1550); repo https://github.com/xiutian/OrChiD

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Overall classification accuracy: the proportion of correctly predicted stance labels (pro, con, mixed) out of all test instances.
- `F1` — range: [0, 1]
  - Per-class F1 score computed separately for pro, con, and mixed stances to account for class imbalance.
- `ROUGE-1 F1` **(primary)** — range: [0, 1]
  - Unigram overlap F1 score between generated summary and gold reference summary.
- `ROUGE-2 F1` — range: [0, 1]
  - Bigram overlap F1 score between generated summary and gold reference summary.
- `ROUGE-L F1` — range: [0, 1]
  - Longest common subsequence overlap F1 score between generated summary and gold reference summary.
- `Human Evaluation Score` — range: [1, 5]
  - Average rating on a 1-5 scale across four aspects: conciseness, fluency, faithfulness, and informativeness.

## Input / output format

**Input**: Stance Detection: target claim text (t) and utterance text (c_i). Summarization: sequence of debate utterances (D).

**Output**: Stance Detection: discrete label {pro, con, mixed}. Summarization: generated text summary (Y).

## Scoring recipe

```python
def score_stance(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    f1s = []
    for stance in ['pro', 'con', 'mixed']:
        tp = sum(1 for p, g in zip(predictions, gold) if p == stance and g == stance)
        fp = sum(1 for p, g in zip(predictions, gold) if p == stance and g != stance)
        fn = sum(1 for p, g in zip(predictions, gold) if p != stance and g == stance)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return acc, f1s

def score_summary(predictions, gold):
    import rouge
    scorer = rouge.Rouge()
    scores = scorer.get_scores(predictions, gold, avg=True)
    return scores['rouge-1']['f'], scores['rouge-2']['f'], scores['rouge-l']['f']
```

## Common pitfalls

- Imbalanced stance labels (31%/31%/38%) require per-class F1 reporting rather than relying solely on overall accuracy.
- Long debate inputs often exceed model context windows, necessitating chunking, iterative prompting, or hierarchical architectures.
- Pipeline approaches for stance-specific summarization suffer from error propagation if the initial stance detection step is inaccurate.

## Evidence (verbatim from paper)

> Following Cheng et al. (2022), we report both overall accuracy and per-class (stance) F1 scores. We choose the well-established ROUGE scores as automatic evaluation metrics and report standard F1 scores of ROUGE-1, ROUGE-2 and ROUGE-L.

## Citation

```bibtex
@misc{zhao2024orchid,
  title={ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization},
  author={Zhao et al. (2024)},
  year={2024},
  note={arXiv:2410.13667}
}
```

- arXiv: 2410.13667