Translation Proofreading Eval

---
name: translation-proofreading-eval
description: Evaluates a hybrid CNN-BERT model's ability to detect and correct errors in English translations. It measures performance across different architectural hyperparameters (kernel size, batch normalization) and linguistic granularity levels. Use when the user wants to benchmark on WMT English-German Parallel Corpus Dataset, Open Subtitles Dataset, or asks about evaluating this task. Reports F1-Score (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.04811
  bibtex_key: liu2025design
  confidence: medium
---

# translation-proofreading-eval

> Design of intelligent proofreading system for English translation based on CNN and BERT — Liu et al. (2025) (arXiv:2506.04811, 2025)

## What this evaluates

Evaluates a hybrid CNN-BERT model's ability to detect and correct errors in English translations. It measures performance across different architectural hyperparameters (kernel size, batch normalization) and linguistic granularity levels.

## Datasets

- **WMT English-German Parallel Corpus Dataset** — total ?; splits: (unstated)
- **Open Subtitles Dataset** — total ?; splits: (unstated)

## Metrics

- `Precision (%)` — range: percent
  - Ratio of correctly identified positive cases to all cases predicted as positive. Reported as a percentage.
- `Recall (%)` — range: percent
  - Ratio of correctly identified positive cases to all actual positive cases. Reported as a percentage.
- `F1-Score (%)` **(primary)** — range: percent
  - Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Reported as a percentage.
- `Accuracy (%)` — range: percent
  - Ratio of all correct predictions to total predictions. Reported as a percentage.
- `RMSE (%)` — range: percent
  - Root Mean Square Error: sqrt(mean((y_true - y_pred)^2)). Reported as a percentage.
- `MSE (%)` — range: percent
  - Mean Squared Error: mean((y_true - y_pred)^2). Reported as a percentage.
- `MAE (%)` — range: percent
  - Mean Absolute Error: mean(|y_true - y_pred|). Reported as a percentage.

## Input / output format

**Input**: Translated text or sentence pairs (source-target) for error detection and correction.

**Output**: Corrected translation or error labels/masks.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    accuracy = (tp + tn) / len(y_true)
    mse = sum((t - p)**2 for t, p in zip(y_true, y_pred)) / len(y_true)
    rmse = mse**0.5
    mae = sum(abs(t - p) for t, p in zip(y_true, y_pred)) / len(y_true)
    return precision*100, recall*100, f1*100, accuracy*100, rmse*100, mse*100, mae*100
```

## Common pitfalls

- Mixes classification metrics (Precision/Recall/F1/Accuracy) with regression-style error metrics (RMSE/MSE/MAE) for the same proofreading task without clarifying the target variable or thresholding strategy.
- High accuracy scores (e.g., 90%) are reported alongside low F1-scores (~50%), indicating potential severe class imbalance or thresholding issues not addressed in the evaluation protocol.
- Metrics are reported at multiple linguistic granularity levels (Phrase to Chapter) but the aggregation method across these levels is not specified.

## Evidence (verbatim from paper)

> The network attains a 75.32% precision that represent accuracy for the identification of meaningful data points by the usage of kernel size 1. The recall is 82.36% which is good at recognizing all relevant circumstances. The F1-Score stands at 85.31% as well as the overall accuracy is 80.54%.

## Citation

```bibtex
@misc{liu2025design,
  title={Design of intelligent proofreading system for English translation based on CNN and BERT},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.04811}
}
```

- arXiv: 2506.04811