Dhoroni Eval

---
name: dhoroni-eval
description: Evaluates a model's ability to perform multi-dimensional discourse analysis on Bengali climate news articles. It probes capabilities in stance detection, authenticity verification, political influence identification, and various information extraction tasks related to environmental reporting. Use when the user wants to benchmark on Dhoroni, or asks about evaluating this task. Reports F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.17225
  bibtex_key: wasi2024dhoroni
  confidence: high
---

# dhoroni-eval

> Dhoroni: Exploring Bengali Climate Change and Environmental Views with a Multi-Perspective News Dataset and Natural Language Processing — Azmine Toushik Wasi et al. (2024) (arXiv:2410.17225, 2024)

## What this evaluates

Evaluates a model's ability to perform multi-dimensional discourse analysis on Bengali climate news articles. It probes capabilities in stance detection, authenticity verification, political influence identification, and various information extraction tasks related to environmental reporting.

## Datasets

- **Dhoroni** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Accuracy` — range: [0, 1]
  - The proportion of correctly classified instances out of the total number of instances.
- `Precision` — range: [0, 1]
  - The ratio of true positive predictions to the total number of positive predictions (TP / (TP + FP)).
- `Recall` — range: [0, 1]
  - The ratio of true positive predictions to the total number of actual positives (TP / (TP + FN)).
- `F1 Score` **(primary)** — range: [0, 1]
  - The harmonic mean of precision and recall, calculated as 2 * (Precision * Recall) / (Precision + Recall). Prioritized for evaluation due to class imbalance.

## Input / output format

**Input**: Bengali-language news article text.

**Output**: Classification label for one of ten predefined discourse dimensions (e.g., stance, authenticity, political influence, scientific data usage, etc.).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    acc = sum(1 for t, p in zip(y_true, y_pred) if t == p) / len(y_true)
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t != p)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t != p)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return acc, prec, rec, f1
```

## Common pitfalls

- Accuracy is heavily skewed by class imbalance, making it a misleading primary indicator of performance for minority classes.
- Significant train-validation performance gap indicates overfitting, as models learn training data well but fail to generalize.
- Task definitions for complex dimensions (e.g., Climate/Env. Topics, Authority Involvement) are ambiguous, leading to near-zero model performance.

## Evidence (verbatim from paper)

> Across the tasks, we can see a clear pattern: accuracy tends to be higher, but the more insightful metrics—precision, recall, and F1 score—reveal deeper issues, particularly with identifying specific classes or more complicated information.

## Citation

```bibtex
@misc{wasi2024dhoroni,
  title={Dhoroni: Exploring Bengali Climate Change and Environmental Views with a Multi-Perspective News Dataset and Natural Language Processing},
  author={Azmine Toushik Wasi et al. (2024)},
  year={2024},
  note={arXiv:2410.17225}
}
```

- arXiv: 2410.17225