Findata Eval

---
name: findata-eval
description: Evaluates multi-task and single-task learning models on a curated collection of financial NLP tasks (FinDATA) to assess how task diversity, relatedness, and parameter-efficient architectures impact performance. Use when the user wants to benchmark on FinDATA, or asks about evaluating this task. Reports evaluation metrics in Table 2.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14007
  bibtex_key: ni2023mtlfinancialnlp
  confidence: low
---

# findata-eval

> When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP — Ni et al. (2023) (arXiv:2305.14007, 2023)

## What this evaluates

Evaluates multi-task and single-task learning models on a curated collection of financial NLP tasks (FinDATA) to assess how task diversity, relatedness, and parameter-efficient architectures impact performance.

## Datasets

- **FinDATA** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `evaluation metrics in Table 2` **(primary)** — range: varies by task
  - Task-specific metrics defined in Table 2 of the paper, used to select the best validation checkpoints and report test performance.

## Input / output format

**Input**: Text inputs corresponding to various financial NLP tasks within the FinDATA collection.

**Output**: Task-specific predictions or labels, evaluated against gold labels using the task-specific metrics.

## Scoring recipe

```python
# For each task t in FinDATA:
val_score = compute_metric(predictions, gold, task=t)
if val_score > best_val_score[t]:
    best_val_score[t] = val_score
    best_checkpoint[t] = current_checkpoint
test_score = compute_metric(best_checkpoint[t].predict(test_data), test_gold, task=t)
final_results = average(test_scores across seeds 1-5) with std_dev
```

## Common pitfalls

- MTL checkpoint selection is performed independently per task, not jointly across all tasks.
- Financial PhraseBank is explicitly excluded from the main model comparison due to lacking an official test set.
- All results must be averaged over random seeds 1 to 5 with standard deviations reported.

## Evidence (verbatim from paper)

> We follow the evaluation metrics in Table 2 to select the best checkpoints and report the test performance. All MTL and STL results are averaged over random seeds from 1 to 5 with standard deviations attached.

## Citation

```bibtex
@misc{ni2023mtlfinancialnlp,
  title={When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP},
  author={Ni et al. (2023)},
  year={2023},
  note={arXiv:2305.14007}
}
```

- arXiv: 2305.14007