System Loss

---
name: system-loss
description: Evaluates the expected cost incurred by a crowdsourcing platform when inferring service states from biased, strategic user reviews under different baseline mechanisms. Use when the user has predictions and gold and needs to compute system loss.
metadata:
  skill_kind: metric
  source_arxiv: 2306.06791
  bibtex_key: hao2023save
  confidence: high
---

# system-loss

> To Save Mobile Crowdsourcing from Cheap-talk: A Game Theoretic Learning Approach — Hao et al. (2023) (arXiv:2306.06791, 2023)

## What this evaluates

Evaluates the expected cost incurred by a crowdsourcing platform when inferring service states from biased, strategic user reviews under different baseline mechanisms.

## Datasets

- **Theoretical Crowdsourcing Model** — total ?; splits: (unstated)

## Metrics

- `system loss` **(primary)** — range: other
  - Expected cost given the platform's action and the true state distribution. For majority voting: L_1 = (μ_H - μ_L)^2 * (1/2^N) * (p_H * Σ_{k=0}^{⌈N/2⌉-1} C_N^k + (1-p_H) * Σ_{l=⌊N/2⌋+1}^{N} C_N^l + 1_{⌊N/2⌋=N/2} * p_H(1-p_H) * C_N^{N/2}). For blind abandoning: L_2 = p_H(1-p_H)(μ_H - μ_L)^2.

## Input / output format

**Input**: Parameters: number of users N, prior probability of high-type users p_H, mean service states for high/low types μ_H and μ_L, and user messages.

**Output**: Platform's recommendation action a and the resulting expected system loss L.

## Scoring recipe

```python
def compute_system_loss(N, p_H, mu_H, mu_L, scheme='majority_voting'):
    if scheme == 'majority_voting':
        loss = (mu_H - mu_L)**2 * (1 / 2**N) * (
            p_H * sum(comb(N, k) for k in range(0, ceil(N/2))) +
            (1 - p_H) * sum(comb(N, l) for l in range(floor(N/2) + 1, N + 1)) +
            (1 if floor(N/2) == N/2 else 0) * p_H * (1 - p_H) * comb(N, N/2)
        )
    elif scheme == 'blind_abandoning':
        loss = p_H * (1 - p_H) * (mu_H - mu_L)**2
    return loss
```

## Common pitfalls

- System loss scales quadratically with the difference between high and low type means (μ_H - μ_L), so it can be arbitrarily large.
- Benchmarks assume users blindly message their biased type rather than playing a strategic equilibrium, which may overstate their effectiveness.

## Evidence (verbatim from paper)

> The resultant system loss is ... which can be arbitrarily large as μ_H - μ_L → ∞.

## Citation

```bibtex
@misc{hao2023save,
  title={To Save Mobile Crowdsourcing from Cheap-talk: A Game Theoretic Learning Approach},
  author={Hao et al. (2023)},
  year={2023},
  note={arXiv:2306.06791}
}
```

- arXiv: 2306.06791