Korfinsts Eval

---
name: korfinsts-eval
description: Evaluates the ability of cross-lingual embedding models to capture nuanced financial semantics and terminology in low-resource Korean text, specifically measuring how well they align with human judgments of sentence similarity in specialized financial contexts. Use when the user wants to benchmark on KorFinSTS, or asks about evaluating this task. Reports Spearman’s ρ.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.09601
  bibtex_key: lee2025nmixx
  confidence: high
---

# korfinsts-eval

> NMIXX: Domain-Adapted Neural Embeddings for Cross-Lingual eXploration of Finance — Lee et al. (2025) (arXiv:2507.09601, 2025)

## What this evaluates

Evaluates the ability of cross-lingual embedding models to capture nuanced financial semantics and terminology in low-resource Korean text, specifically measuring how well they align with human judgments of sentence similarity in specialized financial contexts.

## Datasets

- **KorFinSTS** — total 1921; splits: test (1921)

## Metrics

- `Spearman’s ρ` **(primary)** — range: [-1, 1]
  - Spearman's rank correlation coefficient between the model-predicted similarity scores and the human-annotated similarity scores on a 0–5 scale.

## Input / output format

**Input**: Pairs of Korean financial sentences (e.g., from news, investment reports, regulatory disclosures, or legal texts).

**Output**: A continuous similarity score, typically on a 0–5 scale, representing the semantic similarity between the two sentences.

## Scoring recipe

```python
def compute_spearman_rho(predictions, gold):
    from scipy.stats import spearmanr
    rho, _ = spearmanr(predictions, gold)
    return rho
```

## Common pitfalls

- Using general-domain STS benchmarks instead of domain-specific financial ones, which fails to capture specialized terminology and nuanced financial framing.
- Relying on directly translated English datasets, which often lose cultural context and terminological precision, leading to significant performance degradation in low-resource settings.
- Ignoring the 0–5 human annotation scale and treating the task as a binary classification or regression without proper rank-based evaluation.

## Evidence (verbatim from paper)

> STS is the problem of measuring the semantic similarity of texts. ... The final benchmark consists of 1,921 high-quality sentence pairs. ... Similarity Score Distribution (0-5 Scale) ... Mean Score 0.59 ... The model achieves +0.22 Spearman’s ρ gains on KorFinSTS (a new Korean financial STS benchmark) and outperforms general-purpose baselines by the largest margin, demonstrating superior capture of financial semantics in low-resource languages.

## Citation

```bibtex
@misc{lee2025nmixx,
  title={NMIXX: Domain-Adapted Neural Embeddings for Cross-Lingual eXploration of Finance},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2507.09601}
}
```

- arXiv: 2507.09601