Fife Eval

---
name: fife-eval
description: Evaluates large language models' ability to follow complex financial instructions, with a strong emphasis on precise adherence to formatting, structural constraints, and conditional styling requirements. It probes whether models can maintain procedural compliance rather than just semantic correctness. Use when the user wants to benchmark on FIFE, or asks about evaluating this task. Reports Strict compliance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.08965
  bibtex_key: matlin2025fife
  confidence: medium
---

# fife-eval

> Financial Instruction Following Evaluation (FIFE) — Matlin et al. (2025) (arXiv:2512.08965, 2025)

## What this evaluates

Evaluates large language models' ability to follow complex financial instructions, with a strong emphasis on precise adherence to formatting, structural constraints, and conditional styling requirements. It probes whether models can maintain procedural compliance rather than just semantic correctness.

## Datasets

- **FIFE** — total ?; splits: test (-1)

## Metrics

- `Strict compliance` **(primary)** — range: percent
  - Percentage of instructions followed with exact adherence to all requested formatting, structural, and styling constraints. Any deviation (e.g., wrong HTML tags, missing table headers) results in a penalty.
- `Loose compliance` — range: percent
  - Percentage of instructions where the model's response is semantically correct and fulfills the core instruction, even if minor formatting or structural deviations are present.

## Input / output format

**Input**: Financial instruction prompts specifying complex formatting, structural requirements, and conditional styling rules (e.g., specific table headers, conditional bolding, exact HTML tag usage).

**Output**: Model-generated text that must strictly or loosely adhere to the formatting and structural constraints specified in the prompt.

## Scoring recipe

```python
def evaluate(predictions, gold):
    strict_correct = 0
    loose_correct = 0
    for pred, gold in zip(predictions, gold):
        # Strict: exact match on all formatting/structural constraints
        if matches_exact_format_and_structure(pred, gold):
            strict_correct += 1
        # Loose: semantic correctness allowed despite minor formatting flaws
        if matches_semantic_intent(pred, gold):
            loose_correct += 1
    strict_score = (strict_correct / len(predictions)) * 100
    loose_score = (loose_correct / len(predictions)) * 100
    return strict_score, loose_score
```

## Common pitfalls

- Models often produce semantically correct responses but fail strict scoring due to minor formatting deviations (e.g., using an incorrect HTML tag instead of the requested one).
- Reasoning-tuned or conversational models do not necessarily outperform general-purpose models on strict instruction following, as the benchmark penalizes procedural flaws over factual errors.
- Evaluators may mistakenly reward semantic accuracy under the strict metric, ignoring the benchmark's explicit focus on formatting and structural compliance.

## Evidence (verbatim from paper)

> Our findings reveal a clear hierarchy in instruction-following capabilities, with open-weight models leading the FIFE benchmark. As shown in 1, the top-performing open-weight system (Llama-4Maverick 18B) achieved a compliance of (76.1 strict / 79.5 loose), significantly outperforming the leading proprietary model (65.9 strict / 70.5 loose).

## Citation

```bibtex
@misc{matlin2025fife,
  title={Financial Instruction Following Evaluation (FIFE)},
  author={Matlin et al. (2025)},
  year={2025},
  note={arXiv:2512.08965}
}
```

- arXiv: 2512.08965