Scielo Parallel Corpus Eval

---
name: scielo-parallel-corpus-eval
description: Evaluates the quality of sentence alignment and machine translation performance on a trilingual scientific article corpus. It probes cross-lingual translation accuracy and structural alignment precision in a specialized academic domain. Use when the user wants to benchmark on Scielo Parallel Corpus, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1905.01852
  bibtex_key: soares2019large
  confidence: high
---

# scielo-parallel-corpus-eval

> A Large Parallel Corpus of Full-Text Scientific Articles — Soares et al. (2019) (arXiv:1905.01852, 2019)

## What this evaluates

Evaluates the quality of sentence alignment and machine translation performance on a trilingual scientific article corpus. It probes cross-lingual translation accuracy and structural alignment precision in a specialized academic domain.

## Datasets

- **Scielo Parallel Corpus** — total ?; splits: train (-1), tuning (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram precision with brevity penalty, computed using the Moses SMT toolkit on the held-out test set.
- `alignment accuracy` — range: percent
  - Ratio of manually verified correctly aligned sentence pairs to the total number of sampled sentences, expressed as a percentage.

## Input / output format

**Input**: Source sentence in English, Portuguese, or Spanish.

**Output**: Target sentence in the corresponding target language (English, Portuguese, or Spanish).

## Scoring recipe

```python
# Split corpus per language pair into 85% train, 5% tuning, 10% test
# Train Moses SMT model on train set, tune on tuning set
# Generate translations for test set
bleu_score = moses_bleu(test_source, test_hypotheses)

# Manual validation subset
sampled_sentences = random.sample(corpus_sentences, 400)
correct = count_correct_alignments(sampled_sentences)
alignment_accuracy = (correct / len(sampled_sentences)) * 100
```

## Common pitfalls

- Domain-specific scientific nomenclature and context often cause translation incoherences that BLEU fails to capture accurately.
- Manual alignment validation was only performed on 400 randomly sampled sentences, which may not represent the full corpus distribution.
- The corpus is distributed in TMX format, requiring conversion to standard parallel corpus formats for most modern NLP pipelines.

## Evidence (verbatim from paper)

> Prior to the SMT experiments, all sentences were randomly split in three disjoint datasets for each language pair: training, tuning and test. Approximately 85% of the aligned sentences were kept for training, 5% for tuning and 10% for test. Table 2 presents the BLEU scores for each language pair for the test set. We manually validated the alignment quality for 400 sentences randomly sampled from the parsed corpus. Figure 2 depicts the rate of correct alignments for each subset of parallel languages.

## Citation

```bibtex
@misc{soares2019large,
  title={A Large Parallel Corpus of Full-Text Scientific Articles},
  author={Soares et al. (2019)},
  year={2019},
  note={arXiv:1905.01852}
}
```

- arXiv: 1905.01852