Subjective Qa Eval

---
name: subjective-qa-eval
description: Evaluates models' ability to classify six subjective linguistic features (Assertive, Cautious, Optimistic, Specific, Clear, Relevant) in financial earnings call question-and-answer transcripts. It probes how well models capture nuanced, tone-based, and domain-specific communication cues beyond factual content. Use when the user wants to benchmark on SubjECTive-QA, or asks about evaluating this task. Reports weighted F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.20651
  bibtex_key: pardawala2024subjectiveqa
  confidence: high
---

# subjective-qa-eval

> SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis — Pardawala et al. (2024) (arXiv:2410.20651, 2024)

## What this evaluates

Evaluates models' ability to classify six subjective linguistic features (Assertive, Cautious, Optimistic, Specific, Clear, Relevant) in financial earnings call question-and-answer transcripts. It probes how well models capture nuanced, tone-based, and domain-specific communication cues beyond factual content.

## Datasets

- **SubjECTive-QA** — total 49446; splits: train (-1), val (-1), test (-1); repo https://github.com/gtfintechlab/SubjECTive-QA

## Metrics

- `weighted F1 score` **(primary)** — range: percent
  - Computed per feature by averaging class-level F1 scores weighted by their support (true positive count). The final score is the arithmetic mean of the weighted F1 scores across all six features.

## Input / output format

**Input**: Question-answer pairs extracted from earnings call transcripts.

**Output**: Classification labels for six subjective features: Assertive, Cautious, Optimistic, Specific, Clear, Relevant.

## Scoring recipe

```python
scores = []
for feature in ["Assertive", "Cautious", "Optimistic", "Specific", "Clear", "Relevant"]:
    y_true, y_pred = get_labels(feature)
    scores.append(f1_score(y_true, y_pred, average="weighted"))
return sum(scores) / len(scores)
```

## Common pitfalls

- Train/val/test split sizes are not disclosed in the benchmarking section, hindering exact reproducibility.
- Models are evaluated independently per feature rather than jointly, which may obscure cross-feature trade-offs.
- Weighted F1 is reported as a percentage, but class distribution per feature is not provided, making it hard to assess bias.

## Evidence (verbatim from paper)

> For each feature, we observed different models performing better. Due to the independence of our features, we can use each model independently to evaluate a given feature. For Clear, BERT had the highest weighted F1 score of 80.93%. For Optimistic and Assertive, RoBERTa-base had the highest weighted F1 scores of 62.69% and 49.10%, respectively. For Relevant, the LLMs, Llama-3-70b-Chat and Mixtral-8x22B Instruct (141B), outperformed the Pre-trained Language Models (PLMs)... Across all six features, RoBERTa-base had the highest average weighted F1 score of 63.95%.

## Citation

```bibtex
@misc{pardawala2024subjectiveqa,
  title={SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis},
  author={Pardawala et al. (2024)},
  year={2024},
  note={arXiv:2410.20651}
}
```

- arXiv: 2410.20651