Linguistic Diversity

---
name: linguistic-diversity
description: Evaluates the lexical, semantic, and structural diversity of natural language instructions across datasets. It quantifies repetition, vocabulary richness, semantic coverage, and syntactic complexity to identify construction biases and limitations in dataset design. Use when the user has predictions and gold and needs to compute ROUGE-L.
metadata:
  skill_kind: metric
  source_arxiv: 2601.03136
  bibtex_key: wanna2026limitedlinguistic
  confidence: high
---

# linguistic-diversity

> Limited Linguistic Diversity in Embodied AI Datasets — Wanna et al. (2026) (arXiv:2601.03136, 2026)

## What this evaluates

Evaluates the lexical, semantic, and structural diversity of natural language instructions across datasets. It quantifies repetition, vocabulary richness, semantic coverage, and syntactic complexity to identify construction biases and limitations in dataset design.

## Datasets

- **GLUE** — total ?; splits: train (-1)
- **OASST2** — total ?; splits: train (-1)
- **ALFRED** — total ?; splits: train (-1)
- **RT-1** — total ?; splits: train (-1)
- **LIBERO** — total ?; splits: train (-1)

## Metrics

- `ROUGE-L` **(primary)** — range: [0, 1]
  - Computes the longest common subsequence between sentence pairs, normalized by reference length. Used here as a pairwise lexical similarity metric.
- `BERTScore` — range: [0, 1]
  - Measures semantic similarity by comparing contextual embeddings of tokens in sentences using precision, recall, and F1 scores.
- `Tree Kernel` — range: [0, 1]
  - Computes syntactic similarity based on parse tree structures, measuring structural overlap between sentences.
- `USE` — range: other
  - Intrinsic dimensionality metric: the minimum number of PCA components derived from USE embeddings required to explain 95% of variance in the dataset.
- `Compression Ratio` — range: other
  - Ratio of total tokens to unique tokens, indicating lexical repetition.

## Input / output format

**Input**: Natural language instruction sentences or commands extracted from a dataset.

**Output**: Numerical scores for each diversity metric (e.g., ROUGE-L distance, BERTScore similarity, Tree Kernel distance, PCA component count, compression ratio, unique sentence percentage).

## Scoring recipe

```python
def compute_diversity_metrics(sentences):
    # Sample 1000 commands, repeat 3 times for robustness
    samples = sample(sentences, k=1000, repeats=3)
    scores = {}
    scores['ROUGE-L'] = rouge_l_similarity(samples)
    scores['BERTScore'] = bertscore_similarity(samples)
    scores['Tree Kernel'] = tree_kernel_distance(samples)
    scores['USE'] = pca_components(samples, threshold=0.95)
    scores['CR'] = total_tokens(samples) / unique_tokens(samples)
    scores['% Uniq'] = unique_sentences(samples) / total_sentences(samples)
    return scores
```

## Common pitfalls

- Pairwise metrics (ROUGE-L, BERTScore, Tree Kernel) are computed on sampled subsets (1,000 commands, repeated 3 times) rather than the full dataset, which may introduce sampling variance.
- Low uniqueness in VLA datasets often stems from command duplication across multiple action trajectories, not necessarily poor dataset design.
- USE intrinsic dimensionality measures semantic coverage but does not directly correlate with the number of unique commands or dataset size.

## Evidence (verbatim from paper)

> Pairwise scores (ROUGE-L, BERTScore, Tree Kernel) are computed by sampling 1,000 commands from each dataset, repeated three times for robustness. Arrows indicate increasing linguistic diversity. CR stands for Compression Ratio. The Tree Kernel method is from Moschitti (2006). USE refers to the minimum # of PCA components derived from USE embeddings to explain 95% variance for each dataset.

## Citation

```bibtex
@misc{wanna2026limitedlinguistic,
  title={Limited Linguistic Diversity in Embodied AI Datasets},
  author={Wanna et al. (2026)},
  year={2026},
  note={arXiv:2601.03136}
}
```

- arXiv: 2601.03136