Lemat Bulk Eval

---
name: lemat-bulk-eval
description: Evaluates the accuracy and robustness of crystal structure fingerprinting and hashing algorithms for de-duplicating quantum chemistry materials databases. It probes sensitivity to structural perturbations (atomic noise, lattice strain, translations) and performance on disordered crystal systems. Use when the user wants to benchmark on LeMat-Bulk, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.05178
  bibtex_key: siron2025lematbulk
  confidence: high
---

# lemat-bulk-eval

> LeMat-Bulk: aggregating, and de-duplicating quantum chemistry materials databases — Siron et al. (2025) (arXiv:2511.05178, 2025)

## What this evaluates

Evaluates the accuracy and robustness of crystal structure fingerprinting and hashing algorithms for de-duplicating quantum chemistry materials databases. It probes sensitivity to structural perturbations (atomic noise, lattice strain, translations) and performance on disordered crystal systems.

## Datasets

- **LeMat-Bulk** — total 5340000; splits: test (5340000); repo https://github.com/LeMaterial/lematerial-ftcher

## Metrics

- `success rate` **(primary)** — range: percent
  - Percentage of structure pairs correctly identified as duplicates or matches by a hashing/similarity method. Calculated as (number of correctly matched pairs / total evaluated pairs) * 100. Aggregated across chemical compositions for disordered structures.
- `sensitivity threshold` — range: other
  - The magnitude of structural perturbation (e.g., Gaussian noise on atomic coordinates or lattice vectors) at which the success rate drops to 50%. Lower values indicate higher sensitivity to noise.
- `mean energy difference` — range: eV/atom
  - Average absolute difference in formation energy (eV/atom) between matched structures before and after DFT relaxation.

## Input / output format

**Input**: Crystal structure representation including atomic fractional coordinates, lattice vectors, and chemical composition. For perturbation tests, Gaussian noise is added to coordinates/lattice vectors, or translations/strain are applied.

**Output**: Binary match decision or similarity score/hash fingerprint. Methods output a fingerprint string (BAWL, Short-BAWL, SLICES, CLOUD, PDD) or a similarity metric (EqV2-sim, Pymatgen/Mattergen matchers).

## Scoring recipe

```python
def compute_success_rate(structures_A, structures_B, method):
    matches = 0
    total = len(structures_A)
    for i in range(total):
        hash_A = method.fingerprint(structures_A[i])
        hash_B = method.fingerprint(structures_B[i])
        if hash_A == hash_B:
            matches += 1
    return (matches / total) * 100

def compute_sensitivity_threshold(structures, noise_levels, method):
    rates = []
    for noise in noise_levels:
        perturbed = add_noise(structures, noise)
        rate = compute_success_rate(structures, perturbed, method)
        rates.append(rate)
    return interpolate_threshold(noise_levels, rates, target=50)
```

## Common pitfalls

- Fingerprint methods scale linearly with database size while combinatorial structure matchers scale quadratically, making runtime comparisons misleading without normalization.
- High success rates on disordered structures may not account for false positives; validation relies on composition matching rather than DFT energy convergence.
- Sensitivity thresholds are highly dependent on the perturbation type (atomic vs lattice) and noise distribution, making cross-method comparisons sensitive to the exact noise parameters used.

## Evidence (verbatim from paper)

> Across each composition, we report how many of each structures were able to be properly matched. We then aggregate these results over all chemical formulas and report the mean success rate and standard deviation.

## Citation

```bibtex
@misc{siron2025lematbulk,
  title={LeMat-Bulk: aggregating, and de-duplicating quantum chemistry materials databases},
  author={Siron et al. (2025)},
  year={2025},
  note={arXiv:2511.05178}
}
```

- arXiv: 2511.05178