Corpus Technical Validation Eval

---
name: corpus-technical-validation-eval
description: Evaluates the structural integrity, metadata completeness, and text quality of a legally screened chemistry corpus derived from S2ORC. It verifies schema compliance, metadata field availability, subfield label validity, chunking consistency, and embedding reproducibility against predefined thresholds. Use when the user wants to benchmark on Lit2Vec Chemistry Corpus, or asks about evaluating this task. Reports schema_pass_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12498
  bibtex_key: [REDACTED]
  confidence: high
---

# corpus-technical-validation-eval

> Lit2Vec: A Reproducible Workflow for Building a Legally Screened Chemistry Corpus from S2ORC for Downstream Retrieval and Text Mining — Amiri et al. (2026) (arXiv:2604.12498, 2026)

## What this evaluates

Evaluates the structural integrity, metadata completeness, and text quality of a legally screened chemistry corpus derived from S2ORC. It verifies schema compliance, metadata field availability, subfield label validity, chunking consistency, and embedding reproducibility against predefined thresholds.

## Datasets

- **Lit2Vec Chemistry Corpus** — total 582683; splits: full (582683)

## Metrics

- `schema_pass_rate` **(primary)** — range: percent
  - Percentage of records passing all schema and structural checks, including JSON schema compliance, field typing, and embedding format validity.
- `metadata_pass_rate` — range: percent
  - Percentage of records with complete bibliographic fields (title, authors, venue, year, DOI, license) without warnings.
- `text_quality_pass_rate` — range: percent
  - Percentage of records passing length limits, Unicode validity, ASCII/whitespace ratios, and abstract–full-text alignment checks.
- `chunking_pass_rate` — range: percent
  - Percentage of documents passing token-size limits, Unicode integrity, and one-to-one paragraph–embedding mapping checks.
- `ROUGE-1` — range: [0, 1]
  - Recall of ROUGE-1 unigrams between abstracts and full-text introductions to measure lexical alignment.

## Input / output format

**Input**: JSON records containing full-text articles, abstracts, metadata, and paragraph-level chunks from the S2ORC-derived corpus.

**Output**: Validation status per record (pass/warn/fail) and aggregate pass rates per validation area. For text alignment, ROUGE-1 recall scores per record.

## Scoring recipe

```python
def evaluate_corpus(records):
    n = len(records)
    schema_pass = sum(1 for r in records if r.schema_valid)
    metadata_pass = sum(1 for r in records if r.metadata_complete)
    text_pass = sum(1 for r in records if r.text_quality_valid)
    chunk_pass = sum(1 for r in records if r.chunking_valid)
    rouge_scores = [rouge1_recall(r.abstract, r.full_text_intro) for r in records]
    return {
        'schema_pass_rate': schema_pass / n,
        'metadata_pass_rate': metadata_pass / n,
        'text_quality_pass_rate': text_pass / n,
        'chunking_pass_rate': chunk_pass / n,
        'ROUGE-1': rouge_scores
    }
```

## Common pitfalls

- Missing or short abstracts (<100 chars) are excluded from embedding/subfield enrichment by design, triggering 'non-complete' schema flags that are not model failures.
- License warnings often reflect upstream S2ORC metadata gaps rather than true licensing conflicts, as the workflow enriches with OpenAlex/Crossref/Unpaywall.
- Short final chunks (<80-100 tokens) trigger warnings but are expected due to paragraph boundaries, not chunking errors.

## Evidence (verbatim from paper)

> Alignment analysis (Fig. 7) showed that approximately 355,000 records (72%) achieved strong lexical alignment between abstracts and the introductory portions of their corresponding full texts (ROUGE-1≥0.95). Fewer than 10% of records had low alignment (<0.5), typically in combination with other quality flags. Most records (398,064) had no content-quality flags.

## Citation

```bibtex
@misc{amiri2026lit2vec,
  title={Lit2Vec: A Reproducible Workflow for Building a Legally Screened Chemistry Corpus from S2ORC for Downstream Retrieval and Text Mining},
  author={Amiri et al. (2026)},
  year={2026},
  note={arXiv:2604.12498}
}
```

- arXiv: 2604.12498