Ice Flare Eval

---
name: ice-flare-eval
description: Evaluates bilingual (Chinese and English) financial large language models across 14 NLP tasks, including sentiment analysis, classification, question answering, and information extraction. It probes cross-lingual adaptability, domain-specific reasoning, and instruction-following capabilities on financial text. Use when the user wants to benchmark on FE, StockB, CFPB, CFiQA-SA, FPB, FiQA-SA, Corpus, AFQMC, NL, NL2, NSP, FinevalF, StcokA, CACL18, CBigData18, CIKM18, ACL18, BigData18, RE, CHeadlines, Headlines, German, Australian, FOMC, QA, CEnQA, CConFinQA, EnQA, ConFinQA, CNER, NER, FINER-ORD, 19CCKS, 20CCKS, 21CCKS, 22CCKS, NA, ECTSUM, EDTSUM, or asks about evaluating this task. Reports F1 Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.06249
  bibtex_key: hu2024no
  confidence: high
---

# ice-flare-eval

> No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks — Gang Hu et al. (2024) (arXiv:2403.06249, 2024)

## What this evaluates

Evaluates bilingual (Chinese and English) financial large language models across 14 NLP tasks, including sentiment analysis, classification, question answering, and information extraction. It probes cross-lingual adaptability, domain-specific reasoning, and instruction-following capabilities on financial text.

## Datasets

- **FE** — total 2020; splits: test (2020); repo https://github.com/The-FinAI/PIXIU
- **StockB** — total 1962; splits: test (1962); repo https://github.com/The-FinAI/PIXIU
- **CFPB** — total 970; splits: test (970); repo https://github.com/The-FinAI/PIXIU
- **CFiQA-SA** — total 233; splits: test (233); repo https://github.com/The-FinAI/PIXIU
- **FPB** — total 970; splits: test (970); repo https://github.com/The-FinAI/PIXIU
- **FiQA-SA** — total 235; splits: test (235); repo https://github.com/The-FinAI/PIXIU
- **Corpus** — total 10000; splits: test (10000); repo https://github.com/The-FinAI/PIXIU
- **AFQMC** — total 4316; splits: test (4316); repo https://github.com/The-FinAI/PIXIU
- **NL** — total 884; splits: test (884); repo https://github.com/The-FinAI/PIXIU
- **NL2** — total 884; splits: test (884); repo https://github.com/The-FinAI/PIXIU
- **NSP** — total 500; splits: test (500); repo https://github.com/The-FinAI/PIXIU
- **FinevalF** — total 222; splits: test (222); repo https://github.com/The-FinAI/PIXIU
- **StcokA** — total 1477; splits: test (1477); repo https://github.com/The-FinAI/PIXIU
- **CACL18** — total 511; splits: test (511); repo https://github.com/The-FinAI/PIXIU
- **CBigData18** — total 159; splits: test (159); repo https://github.com/The-FinAI/PIXIU
- **CIKM18** — total 86; splits: test (86); repo https://github.com/The-FinAI/PIXIU
- **ACL18** — total 3720; splits: test (3720); repo https://github.com/The-FinAI/PIXIU
- **BigData18** — total 1472; splits: test (1472); repo https://github.com/The-FinAI/PIXIU
- **RE** — total 1489; splits: test (1489); repo https://github.com/The-FinAI/PIXIU
- **CHeadlines** — total 2051; splits: test (2051); repo https://github.com/The-FinAI/PIXIU
- **Headlines** — total 20547; splits: test (20547); repo https://github.com/The-FinAI/PIXIU
- **German** — total 200; splits: test (200); repo https://github.com/The-FinAI/PIXIU
- **Australian** — total 139; splits: test (139); repo https://github.com/The-FinAI/PIXIU
- **FOMC** — total 496; splits: test (496); repo https://github.com/The-FinAI/PIXIU
- **QA** — total 2469; splits: test (2469); repo https://github.com/The-FinAI/PIXIU
- **CEnQA** — total 133; splits: test (133); repo https://github.com/The-FinAI/PIXIU
- **CConFinQA** — total 237; splits: test (237); repo https://github.com/The-FinAI/PIXIU
- **EnQA** — total 1147; splits: test (1147); repo https://github.com/The-FinAI/PIXIU
- **ConFinQA** — total 1490; splits: test (1490); repo https://github.com/The-FinAI/PIXIU
- **CNER** — total 337; splits: test (337); repo https://github.com/The-FinAI/PIXIU
- **NER** — total 98; splits: test (98); repo https://github.com/The-FinAI/PIXIU
- **FINER-ORD** — total 1075; splits: test (1075); repo https://github.com/The-FinAI/PIXIU
- **19CCKS** — total 2936; splits: test (2936); repo https://github.com/The-FinAI/PIXIU
- **20CCKS** — total 9159; splits: test (9159); repo https://github.com/The-FinAI/PIXIU
- **21CCKS** — total 1400; splits: test (1400); repo https://github.com/The-FinAI/PIXIU
- **22CCKS** — total 11829; splits: test (11829); repo https://github.com/The-FinAI/PIXIU
- **NA** — total 3600; splits: test (3600); repo https://github.com/The-FinAI/PIXIU
- **ECTSUM** — total 495; splits: test (495); repo https://github.com/The-FinAI/PIXIU
- **EDTSUM** — total 2000; splits: test (2000); repo https://github.com/The-FinAI/PIXIU

## Metrics

- `F1 Accuracy` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Computed per class and macro-averaged for multi-class tasks.
- `MCC` — range: [-1, 1]
  - Matthews Correlation Coefficient, measuring binary/multiclass classification quality considering true/false positives and negatives.
- `EM Accuracy` — range: [0, 1]
  - Exact match accuracy; counts a prediction as correct only if it exactly matches the gold answer string.
- `Entity F1` — range: [0, 1]
  - F1 score computed over extracted entity spans and types.
- `ROUGE/BERTScore/BARTScore` — range: [0, 1]
  - ROUGE measures n-gram overlap; BERTScore and BARTScore use contextual embeddings for semantic similarity.

## Input / output format

**Input**: Financial text, news articles, stock data, or questions in Chinese or English, formatted as zero-shot or few-shot instruction prompts.

**Output**: Predicted class labels, extracted entity sequences, generated answers, or text summaries, depending on the specific task.

## Scoring recipe

```python
def compute_metric(predictions, gold, metric_type):
    if metric_type == 'F1 Accuracy':
        tp = sum(1 for p, g in zip(predictions, gold) if p == g)
        prec = tp / len(predictions) if predictions else 0
        rec = tp / len(gold) if gold else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif metric_type == 'EM Accuracy':
        return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    elif metric_type == 'MCC':
        return matthews_corrcoef(gold, predictions)
    # For ROUGE/BERTScore/BARTScore, use standard library implementations
    # For Entity F1, compute precision/recall over span matches
```

## Common pitfalls

- Confusing language-specific test splits (e.g., FinSP has separate zh and en sets with different sizes and sources).
- Generation tasks (FinST, FinER) require strict label sequence formatting; models often fail to output exact token sequences without careful prompt design.
- Using translated training data (DTT) for evaluation instead of original held-out test sets, which artificially inflates cross-lingual performance.

## Evidence (verbatim from paper)

> Table 2: The details of specific task, evaluation metric, language type, dataset name and test size in ICE-FLARE. ... FinSA | F1 Accuracy | zh | FE StockB CFPB CFiQA-SA en | 2,020 1,962 970 233

## Citation

```bibtex
@misc{hu2024no,
  title={No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks},
  author={Gang Hu et al. (2024)},
  year={2024},
  note={arXiv:2403.06249}
}
```

- arXiv: 2403.06249