Black Box Llm Granularity Eval

---
name: black-box-llm-granularity-eval
description: This evaluation probes a black-box LLM's ability to generate high-cardinality, continuous probability scores for binary classification tasks. It measures how well different prompting and post-processing methods improve operational granularity (control over precision-recall operating points) while maintaining predictive performance. Use when the user wants to benchmark on 11 binary classification datasets (combined into a joint dataset for one experiment), or asks about evaluating this task. Reports PRAUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.17727
  bibtex_key: beyazit2025enabling
  confidence: high
---

# black-box-llm-granularity-eval

> Enabling Fine-Grained Operating Points for Black-Box LLMs — Beyazit et al. (2025) (arXiv:2510.17727, 2025)

## What this evaluates

This evaluation probes a black-box LLM's ability to generate high-cardinality, continuous probability scores for binary classification tasks. It measures how well different prompting and post-processing methods improve operational granularity (control over precision-recall operating points) while maintaining predictive performance.

## Datasets

- **11 binary classification datasets (combined into a joint dataset for one experiment)** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `PRAUC` **(primary)** — range: [0, 1]
  - Area under the Precision-Recall curve. Computed by calculating precision and recall at every unique predicted score threshold, then integrating the resulting curve. Values range from 0 to 1, with higher being better.
- `output_cardinality` — range: other
  - Number of unique predicted scores (|ŷ|) generated across the dataset. Measures the diversity of the output distribution.
- `g^{pre}, g^{rec}, g^{fpr}` — range: other
  - Granularity metrics measuring the spread/concentration of operating points along the precision, recall, and false positive rate axes respectively. Calculated per Equation 2 in the paper. Lower values indicate better granularity.

## Input / output format

**Input**: Text prompts containing binary classification instances (features/context) passed to black-box LLM APIs.

**Output**: Verbalized probability scores (e.g., "0.75") or class predictions. The evaluation focuses on methods that output continuous scores for thresholding.

## Scoring recipe

```python
def compute_prauc(y_true, y_scores):
    desc_indices = np.argsort(y_scores)[::-1]
    y_true_sorted = np.array(y_true)[desc_indices]
    tp = np.cumsum(y_true_sorted)
    fp = np.cumsum(1 - y_true_sorted)
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp[-1] + 1e-8)
    return np.trapz(precision, recall)
```

## Common pitfalls

- The proposed methods do not necessarily improve raw classification accuracy (e.g., F1 or AUROC); their main contribution is increasing score granularity for operating point selection.
- Baseline sampling methods (Sample-Class, Sample-Prob) require 20 LLM calls per instance, making them computationally expensive compared to the proposed 1-2 call methods.
- Individual datasets are small (some ≤250 samples) and have nearly monotonic PR curves, which limits the observable performance gains of supervised methods on them alone.

## Evidence (verbatim from paper)

> We observe that compared to Prompt-Naive, proposed method significantly improves the operational granularity by increasing the diversity and cardinality of the outputs, while outperforming it in terms of PRAUC.

## Citation

```bibtex
@misc{beyazit2025enabling,
  title={Enabling Fine-Grained Operating Points for Black-Box LLMs},
  author={Beyazit et al. (2025)},
  year={2025},
  note={arXiv:2510.17727}
}
```

- arXiv: 2510.17727