Jobfair Behavioral Transfer Eval

---
name: jobfair-behavioral-transfer-eval
description: Evaluates behavioral cloning and extraction fidelity in bias-mitigation LLM agents processing job descriptions. It measures how well mutated agent outputs align with expert golden references using cross-item similarity and a multi-facet diagnostic rubric. Use when the user wants to benchmark on JobFair Corpus, or asks about evaluating this task. Reports BERTScore (F1) against 5-NN average.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.15366
  bibtex_key: sorstkins2025diagnostics
  confidence: high
---

# jobfair-behavioral-transfer-eval

> Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context — Sorstkins et al. (2025) (arXiv:2509.15366, 2025)

## What this evaluates

Evaluates behavioral cloning and extraction fidelity in bias-mitigation LLM agents processing job descriptions. It measures how well mutated agent outputs align with expert golden references using cross-item similarity and a multi-facet diagnostic rubric.

## Datasets

- **JobFair Corpus** — total 300; splits: test (300)

## Metrics

- `BERTScore (F1) against 5-NN average` **(primary)** — range: [0, 1]
  - BERTScore-F1 (using roberta-base) computed between system output and each of the top-5 nearest golden exemplars per item, then averaged. The reported outcome is the delta: mutated_avg minus original_avg.
- `EDScore` — range: [0, 1]
  - Weighted sum of 6 facet scores (Correctness 0.35, Completeness 0.30, Over-Extraction 0.15, Detail Accuracy 0.10, Terminology Consistency 0.05, Reasoning Alignment 0.05). Each facet is scored 0-1.
- `BDScore` — range: [0, 5]
  - Equal-weighted sum of 4 facet scores (tone match, stylistic fidelity, manner of expression, semantic alignment), each scored 0-5.

## Input / output format

**Input**: English-language job descriptions from a curated corpus. For behavioral transfer: each item yields two agent-generated tracks ('Expert Suggestion' and 'Comment Suggestion'). For extraction: 13 documents sampled to cover job profile domains and length deciles.

**Output**: Agent-generated text suggestions/comments or extracted entities/facets. Evaluated against golden exemplars or expert annotations.

## Scoring recipe

```python
def score_bertscore_5nn(output, golden_pool):
    top5 = get_top_5_nearest_golden(output, golden_pool)
    scores = [bertscore_f1(output, g, encoder='roberta-base') for g in top5]
    return sum(scores) / len(scores)

def score_ed(facet_scores):
    weights = {'Correctness': 0.35, 'Completeness': 0.30, 'Over-Extraction': 0.15,
               'Detail Accuracy': 0.10, 'Terminology Consistency': 0.05, 'Reasoning Alignment': 0.05}
    return sum(facet_scores[facet] * w for facet, w in weights.items())
```

## Common pitfalls

- BERTScore ceiling effect: 5-NN averaging and cross-item matching compress the score range, preventing scores from reaching 1.0 and masking qualitative improvements.
- LLM-as-a-Judge arithmetic failure: The judge reliably scores individual facets but consistently fails at deterministic weighted aggregation; final scores must be computed in code, not by the LLM.
- Golden-set heterogeneity: 'Comment' references are longer and more actionable than 'Expert' cues, creating a stronger learning signal and inflating effect sizes for the Comment track.

## Evidence (verbatim from paper)

> Similarity is measured with BERTScore-F1 using roberta-base as the encoder. For each item, the score is computed against the 5-NN average—the mean BERTScore to the five most similar golden exemplars drawn from other jobs—yielding a cross-item, multi-reference evaluation.

## Citation

```bibtex
@misc{sorstkins2025diagnostics,
  title={Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context},
  author={Sorstkins et al. (2025)},
  year={2025},
  note={arXiv:2509.15366}
}
```

- arXiv: 2509.15366