Nifty Stock Movement Eval

---
name: nifty-stock-movement-eval
description: Evaluates LLMs on predicting short-term stock price movements based on financial news headlines and market context. It probes the model's ability to extract directional sentiment and predictive signals from textual financial data for downstream forecasting tasks. Use when the user wants to benchmark on NIFTY Financial News Headlines Dataset, or asks about evaluating this task. Reports F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.09747
  bibtex_key: saqur2024nifty
  confidence: high
---

# nifty-stock-movement-eval

> NIFTY Financial News Headlines Dataset — Saqur et al. (2024) (arXiv:2405.09747, 2024)

## What this evaluates

Evaluates LLMs on predicting short-term stock price movements based on financial news headlines and market context. It probes the model's ability to extract directional sentiment and predictive signals from textual financial data for downstream forecasting tasks.

## Datasets

- **NIFTY Financial News Headlines Dataset** — total ?; splits: test (-1)

## Metrics

- `F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
- `Accuracy` — range: [0, 1]
  - Ratio of correctly predicted instances to the total number of instances.
- `Precision` — range: [0, 1]
  - Ratio of true positive predictions to the total number of positive predictions.
- `Recall` — range: [0, 1]
  - Ratio of true positive predictions to the total number of actual positives.

## Input / output format

**Input**: Financial news headline with market context, formatted as a consistent LM query $x_q$.

**Output**: Classification label indicating stock movement direction (e.g., up/down or specific movement class).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == g == 1)
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(predictions, gold_labels) if p == g == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    accuracy = (tp + tn) / len(gold_labels)
    return {'F1 Score': f1, 'Accuracy': accuracy, 'Precision': precision, 'Recall': recall}
```

## Common pitfalls

- Stock movement prediction is inherently noisy; baseline accuracies often hover near random chance (e.g., 0.24–0.45 in this paper), so modest gains should be interpreted cautiously.
- The evaluation strictly relies on the Flare benchmark's standardized query format ($x_q$); deviating from this prompt structure breaks compatibility with the reported results.
- The paper only reports test split performance; no validation split or early stopping protocol is described, which may lead to overfitting if not controlled.

## Evidence (verbatim from paper)

> Table 5: Performance of a single baseline expert Llama-2-7b-chat with 4 variants (LoRA SFT adapters) on the NIFTY Stock Price Movement Prediction Task (test split).

Metrics ↑
F1 Score
Accuracy
Precision
Recall

## Citation

```bibtex
@misc{saqur2024nifty,
  title={NIFTY Financial News Headlines Dataset},
  author={Saqur et al. (2024)},
  year={2024},
  note={arXiv:2405.09747}
}
```

- arXiv: 2405.09747