Finpt Eval

---
name: finpt-eval
description: Evaluates the ability of foundation models and tabular baselines to predict financial risk (default, fraud, churn) by classifying customer profiles generated from tabular data. It probes how well profile-based tuning captures richer customer semantics compared to isolated table-based classification. Use when the user wants to benchmark on FinBench, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.00065
  bibtex_key: yin2023finpt
  confidence: high
---

# finpt-eval

> FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models — Yin et al. (2023) (arXiv:2308.00065, 2023)

## What this evaluates

Evaluates the ability of foundation models and tabular baselines to predict financial risk (default, fraud, churn) by classifying customer profiles generated from tabular data. It probes how well profile-based tuning captures richer customer semantics compared to isolated table-based classification.

## Datasets

- **FinBench** — total 333000; splits: train (-1), val (-1), test (-1); repo https://github.com/YuweiYin/FinPT

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Calculated per class and averaged for binary classification tasks.

## Input / output format

**Input**: Natural-language customer profiles (tabular-to-text transformations) generated via ChatGPT instructions, fed into pretrained foundation models or tabular baselines.

**Output**: Binary classification label indicating financial risk (e.g., default, fraud, or churn).

## Scoring recipe

```python
def compute_f1(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Using accuracy instead of F1-score on imbalanced financial risk datasets leads to misleadingly high scores due to false negatives.
- Failing to average results over the four specified random seeds (0, 1, 42, 1234) will not match reported performance.
- Not using the validation set for checkpoint selection before testing on the held-out test set.

## Evidence (verbatim from paper)

> For all experiments on FinBench, we use F1-score as the evaluation metric since all datasets in FinBench are imbalanced binary classification task. It is more appropriate in this case than Accuracy because the latter may result in a high level of false negative.

## Citation

```bibtex
@misc{yin2023finpt,
  title={FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models},
  author={Yin et al. (2023)},
  year={2023},
  note={arXiv:2308.00065}
}
```

- arXiv: 2308.00065