Flare Es Eval

---
name: flare-es-eval
description: Evaluates bilingual (Spanish-English) financial understanding, prediction, and generation capabilities of LLMs. Probes cross-lingual transfer, domain-specific instruction following, and performance disparity between high-resource and low-resource financial tasks. Use when the user wants to benchmark on FLARE-ES, or asks about evaluating this task. Reports Acc, F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.07405
  bibtex_key: zhang2024dolares
  confidence: high
---

# flare-es-eval

> D\'olares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English — Xiao Zhang et al. (2024) (arXiv:2402.07405, 2024)

## What this evaluates

Evaluates bilingual (Spanish-English) financial understanding, prediction, and generation capabilities of LLMs. Probes cross-lingual transfer, domain-specific instruction following, and performance disparity between high-resource and low-resource financial tasks.

## Datasets

- **FLARE-ES** — total ?; splits: test (-1); repo https://github.com/chancefocus/PIXIU

## Metrics

- `Acc` **(primary)** — range: [0, 1]
  - Accuracy: proportion of correctly predicted labels out of total instances.
- `F1` **(primary)** — range: [0, 1]
  - F1 Score: harmonic mean of precision and recall, typically macro-averaged across classes.
- `rouge1` — range: [0, 1]
  - ROUGE-1: unigram overlap between generated and reference text.
- `rouge2` — range: [0, 1]
  - ROUGE-2: bigram overlap between generated and reference text.
- `rougeL` — range: [0, 1]
  - ROUGE-L: longest common subsequence overlap between generated and reference text.
- `EntityF1` — range: [0, 1]
  - Entity F1: F1 score computed over extracted named entities.
- `EmAcc` — range: [0, 1]
  - Exact Match Accuracy: proportion of instances where the generated output exactly matches the reference.
- `MCC` — range: [-1, 1]
  - Matthews Correlation Coefficient: correlation coefficient between predicted and actual binary classifications.
- `AvgF1` — range: [0, 1]
  - Average F1 score across multiple classes or tasks.

## Input / output format

**Input**: Instruction prompts in Spanish or English containing financial domain questions, tasks, or text requiring model generation or classification.

**Output**: Model-generated text, class labels, or numerical answers depending on the specific task (e.g., classification labels, extracted entities, generated summaries, or predicted values).

## Scoring recipe

```python
def compute_metrics(predictions, references):
    acc = sum(p == r for p, r in zip(predictions, references)) / len(references)
    f1 = f1_score(references, predictions, average='macro')
    rouge = rouge_score(references, predictions, use_stemmer=True)
    em_acc = sum(p == r for p, r in zip(predictions, references)) / len(references)
    mcc = matthews_corrcoef(references, predictions)
    return {'Acc': acc, 'F1': f1, 'rouge1': rouge['rouge1'], 'rouge2': rouge['rouge2'], 'rougeL': rouge['rougeL'], 'EmAcc': em_acc, 'MCC': mcc}
```

## Common pitfalls

- Models show significant performance drops on Spanish financial tasks compared to English due to data scarcity.
- Summarization tasks (e.g., FNS-2023, ECTSum) are highly challenging, often yielding near-zero scores across most models.
- Cross-lingual transfer does not automatically guarantee parity; specialized instruction tuning in the target language is required.

## Evidence (verbatim from paper)

> Table 3 presents a detailed comparative performance analysis of our FinMA-ES models against other leading large language models (LLMs) on the FLARE-ES benchmark. | Dataset | Metrics | ChatGPT | GPT4 | LLaMA2-7B | ... | MultiFin | Acc | F1 | ...

## Citation

```bibtex
@misc{zhang2024dolares,
  title={D\'olares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English},
  author={Xiao Zhang et al. (2024)},
  year={2024},
  note={arXiv:2402.07405}
}
```

- arXiv: 2402.07405