Depth Eval

---
name: depth-eval
description: Evaluates the effectiveness of a hierarchically pre-trained encoder-decoder model (DEPTH) against a standard T5 baseline on discourse understanding, natural language inference, sentiment analysis, grammar checking, and instruction following. Use when the user wants to benchmark on MNLI, SST2, CoLA, DiscoEval, Natural Instructions, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.07788
  bibtex_key: bamberger2024depth
  confidence: medium
---

# depth-eval

> DEPTH: Discourse Education through Pre-Training Hierarchically — Bamberger et al. (2024) (arXiv:2405.07788, 2024)

## What this evaluates

Evaluates the effectiveness of a hierarchically pre-trained encoder-decoder model (DEPTH) against a standard T5 baseline on discourse understanding, natural language inference, sentiment analysis, grammar checking, and instruction following.

## Datasets

- **MNLI** — total ?; splits: test (-1)
- **SST2** — total ?; splits: test (-1)
- **CoLA** — total ?; splits: test (-1)
- **DiscoEval** — total ?; splits: test (-1)
- **Natural Instructions** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted labels out of the total number of instances. For instruction following tasks, exact-match accuracy is typically used.

## Input / output format

**Input**: Text pairs or single sentences for classification tasks (MNLI, SST2, CoLA, SP, DC); instruction prompts for Natural Instructions.

**Output**: Class labels for classification tasks; generated text responses for Natural Instructions.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Evaluating intermediate checkpoints without full fine-tuning convergence may misrepresent final performance.
- Comparing From-Scratch and Continuous Pre-Training baselines requires careful accounting of pre-training token counts and initialization states.
- Example packing differences between DEPTH and T5 baselines can confound performance comparisons if not controlled.

## Evidence (verbatim from paper)

> We evaluate our models on natural language inference (MNLI, Williams et al. ([2018])), sentiment analysis (SST2, Socher et al. ([2013])), and grammar (CoLA, Warstadt et al. ([2019])) within the GLUE benchmark (Wang et al., [2018]). We also use the DiscoEval suite (Chen et al., [2019]) to evaluate models on their understanding of discourse. We use two tasks from DiscoEval: Sentence Permutation (SP) and Discourse Coherence (DC). SP involves identifying the correct position of a removed, while DC involves predicting whether or not a paragraph was coherent. Finally, we measure our model’s generative abilities on the Natural Instructions (NI) dataset (Mishra et al., [2022]), which measures the ability of LMs to follow instructions, and served as a benchmark for NanoT5 (Nawrot, [2023]).

## Citation

```bibtex
@misc{bamberger2024depth,
  title={DEPTH: Discourse Education through Pre-Training Hierarchically},
  author={Bamberger et al. (2024)},
  year={2024},
  note={arXiv:2405.07788}
}
```

- arXiv: 2405.07788