Discox Eval

---
name: discox-eval
description: Evaluates machine translation systems on discourse-level coherence and terminological precision in expert domains. It probes the model's ability to maintain long-form text consistency and handle domain-specific language beyond sentence-level translation. Use when the user wants to benchmark on DiscoX, or asks about evaluating this task. Reports Metric-S.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.10984
  bibtex_key: zhao2025discox
  confidence: high
---

# discox-eval

> DiscoX: Benchmarking Discourse-Level Translation task in Expert Domains — Zhao et al. (2025) (arXiv:2511.10984, 2025)

## What this evaluates

Evaluates machine translation systems on discourse-level coherence and terminological precision in expert domains. It probes the model's ability to maintain long-form text consistency and handle domain-specific language beyond sentence-level translation.

## Datasets

- **DiscoX** — total 200; splits: test (200); repo https://github.com/ByteDance-Seed/DiscoX

## Metrics

- `Metric-S` **(primary)** — range: [0, 100]
  - A reference-free, LLM-as-a-judge scoring system that evaluates translations across three modular dimensions: accuracy, fluency, and appropriateness. Scores are aggregated into an overall score normalized to a [0, 100] scale.

## Input / output format

**Input**: Source text (Chinese or English) and its machine-translated target text.

**Output**: Numerical scores for Accuracy, Fluency, Appropriateness, and an Overall score (0-100 scale).

## Scoring recipe

```python
def evaluate(source, translation):
    acc = llm_judge(source, translation, dim='accuracy')
    flu = llm_judge(source, translation, dim='fluency')
    app = llm_judge(source, translation, dim='appropriateness')
    overall = aggregate(acc, flu, app)
    return {'accuracy': acc, 'fluency': flu, 'appropriateness': app, 'overall': overall}
```

## Common pitfalls

- Metric-S is reference-free; do not provide gold translations to the judge model.
- Validation uses pairwise ranking consistency (SPA), not absolute score correlation like Pearson or Spearman.
- Segment-level ties are handled by treating scores as consistent if their difference is less than 0.05.

## Evidence (verbatim from paper)

> The most striking finding is in the overall average consistency: Metric-S achieves 70.3%, more than doubling the 34.7% score of XCOMET-QE, a SOTA baseline.

## Citation

```bibtex
@misc{zhao2025discox,
  title={DiscoX: Benchmarking Discourse-Level Translation task in Expert Domains},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2511.10984}
}
```

- arXiv: 2511.10984