Pt En Scientific Abstracts Eval

---
name: pt-en-scientific-abstracts-eval
description: Evaluates the quality of a newly constructed Portuguese-English parallel corpus of scientific abstracts. It probes the effectiveness of automated sentence alignment algorithms and the translation performance of SMT and NMT models on domain-specific academic text. Use when the user wants to benchmark on Theses and Dissertations Abstracts Corpus, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1905.01715
  bibtex_key: soares2019parallel
  confidence: high
---

# pt-en-scientific-abstracts-eval

> A Parallel Corpus of Theses and Dissertations Abstracts — Soares et al. (2019) (arXiv:1905.01715, 2019)

## What this evaluates

Evaluates the quality of a newly constructed Portuguese-English parallel corpus of scientific abstracts. It probes the effectiveness of automated sentence alignment algorithms and the translation performance of SMT and NMT models on domain-specific academic text.

## Datasets

- **Theses and Dissertations Abstracts Corpus** — total 1289372; splits: train (-1), dev (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram overlap metric between reference and hypothesis translations, reported as a percentage (0-100).
- `Alignment Accuracy` — range: percent
  - Percentage of manually validated sentence pairs correctly aligned, with categories for correct, partial, and no alignment.

## Input / output format

**Input**: Single sentence in Portuguese or English.

**Output**: Translated single sentence in the target language.

## Scoring recipe

```python
def compute_bleu_score(references, hypothesis):
    # Standard BLEU calculation (0-100 scale)
    bleu = compute_ngram_overlap(reference=[references], hypothesis=hypothesis) * 100
    return bleu

def compute_alignment_accuracy(gold_labels):
    total = len(gold_labels)
    correct = sum(1 for label in gold_labels if label == 'correct')
    return (correct / total) * 100
```

## Common pitfalls

- Domain mismatch: Google Translate scores lower because it is a generic model, not a domain-specific scientific MT system.
- Segmentation errors: Case folding to lowercase before alignment removes capitalization cues, leading to malformed sentences and partial alignment errors.

## Evidence (verbatim from paper)

> Table 3 presents the BLEU scores for both translation directions with English and Portuguese on the development and test partitions for Moses and OpenNMT models. We also included the scores for Google Translate (GT) as a benchmark of a state-of-the-art system which is widely used. We manually validated the alignment quality for 400 sentences randomly selected from the parsed corpus and assigned quality labels according Section 2.4. From all the evaluated sentences, 82.30% were correctly aligned, while 13.33% were partially aligned, and 4.35% presented no alignment.

## Citation

```bibtex
@misc{soares2019parallel,
  title={A Parallel Corpus of Theses and Dissertations Abstracts},
  author={Soares et al. (2019)},
  year={2019},
  note={arXiv:1905.01715}
}
```

- arXiv: 1905.01715