Darf Srt Eval

---
name: darf-srt-eval
description: Evaluates the accuracy and computational efficiency of the DARF simulation framework for predicting speech recognition thresholds (SRT) in normal-hearing and hearing-impaired listeners across various acoustic maskers and hearing aid conditions. Use when the user wants to benchmark on Empirical SRT datasets (Hochmuth et al. 2015, Hülsmeier et al., Schädler et al. 2020a), or asks about evaluating this task. Reports SRT.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2007.05378
  bibtex_key: hulsmeier2020darf
  confidence: high
---

# darf-srt-eval

> DARF: A data-reduced FADE version for simulations of speech recognition thresholds with real hearing aids — Hülsmeier et al. (2020) (arXiv:2007.05378, 2020)

## What this evaluates

Evaluates the accuracy and computational efficiency of the DARF simulation framework for predicting speech recognition thresholds (SRT) in normal-hearing and hearing-impaired listeners across various acoustic maskers and hearing aid conditions.

## Datasets

- **Empirical SRT datasets (Hochmuth et al. 2015, Hülsmeier et al., Schädler et al. 2020a)** — total ?; splits: test (-1)

## Metrics

- `SRT` **(primary)** — range: dB SPL
  - Speech Recognition Threshold in dB SPL, representing the lowest signal-to-noise ratio at which speech is correctly identified.
- `AST` — range: other
  - Accuracy Speed Tradeoff, computed as simulation time multiplied by SRT error to balance computational cost and prediction accuracy.
- `RMSE` — range: dB
  - Root Mean Square Error between simulated and empirical SRTs.
- `Bias` — range: dB
  - Mean difference between simulated and empirical SRTs.
- `R^2` — range: [0, 1]
  - Coefficient of determination measuring the proportion of variance in empirical SRTs explained by the simulation.

## Input / output format

**Input**: Acoustic signals (speech and maskers like icra1m, icra5-250m, silence), hearing impairment profiles (e.g., N3), and hearing aid configurations/fittings.

**Output**: Simulated SRT values in dB SPL, along with derived metrics (AST, RMSE, Bias, R^2) comparing simulations to empirical data or FADE baselines.

## Scoring recipe

```python
def compute_ast(sim_time, srt_error):
    return sim_time * srt_error

def compute_rmse(pred_srt, emp_srt):
    return sqrt(mean((pred_srt - emp_srt)**2))

def compute_bias(pred_srt, emp_srt):
    return mean(pred_srt - emp_srt)

def compute_r2(pred_srt, emp_srt):
    ss_res = sum((emp_srt - pred_srt)**2)
    ss_tot = sum((emp_srt - mean(emp_srt))**2)
    return 1 - (ss_res / ss_tot)
```

## Common pitfalls

- AST optimization depends heavily on the number of training vs. test sentences, with 120-240 training sentences yielding different tradeoffs.
- Differences between DARF and FADE vary significantly with masker type (stationary vs. fluctuating), requiring careful baseline selection.

## Evidence (verbatim from paper)

> To assess the tradeoff between simulation accuracy and time for the simulation, the Accuracy Speed Tradeoff (AST, Eq. [1]) was computed which is displayed in the lowest panel of Figure 5.

The simulations of both FADE versions correlate with an R2 of 0.99. However, the bias and RMSE indicate that hearing impairment might introduce an additional offset of about 1 dB from standard FADE: Both quantities exceeded the median offsets depicted in Fig. 6 by about 1 dB.

## Citation

```bibtex
@misc{hulsmeier2020darf,
  title={DARF: A data-reduced FADE version for simulations of speech recognition thresholds with real hearing aids},
  author={Hülsmeier et al. (2020)},
  year={2020},
  note={arXiv:2007.05378}
}
```

- arXiv: 2007.05378