Gwlans Eval

---
name: gwlans-eval
description: Predicts target words in computer-aided translation based on source sentences, translation context (prefix, suffix, zero, bidirectional), and human-typed characters. It probes the model's ability to handle discontinuous context and weak positional information in real-world CAT scenarios. Use when the user wants to benchmark on GWLAN Benchmark, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.14913
  bibtex_key: li2021gwlans
  confidence: high
---

# gwlans-eval

> GWLAN: General Word-Level AutocompletioN for Computer-Aided Translation — Huayang Li et al. (2021) (arXiv:2105.14913, 2021)

## What this evaluates

Predicts target words in computer-aided translation based on source sentences, translation context (prefix, suffix, zero, bidirectional), and human-typed characters. It probes the model's ability to handle discontinuous context and weak positional information in real-world CAT scenarios.

## Datasets

- **GWLAN Benchmark** — total ?; splits: train (1250000), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Exact match accuracy: the percentage of instances where the predicted target word exactly matches the gold target word. Scores are averaged across four translation context types (prefix, suffix, zero-context, and bidirectional context).

## Input / output format

**Input**: Source sentence, translation context type (prefix, suffix, zero, or bidirectional), and human-typed character sequence.

**Output**: A single predicted target word.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100
# Final reported score is the average of accuracy computed separately for each of the four context types.
```

## Common pitfalls

- The evaluation averages accuracy across four distinct context types (prefix, suffix, zero, bidirectional); reporting only one context's score misrepresents overall performance.
- Trivial instances (e.g., predicting 'the' given 'th') are filtered out using length rules (>4 chars English, >2 chars Chinese) and context length constraints; ignoring these rules inflates scores on easy cases.

## Evidence (verbatim from paper)

> Table 1: The main results of different systems on Chinese-English and German-English datasets. The results in this table are the averaged accuracy on four translation contexts (i.e., prefix, suffix, zero-context, and bi-context).

## Citation

```bibtex
@misc{li2021gwlans,
  title={GWLAN: General Word-Level AutocompletioN for Computer-Aided Translation},
  author={Huayang Li et al. (2021)},
  year={2021},
  note={arXiv:2105.14913}
}
```

- arXiv: 2105.14913