First Inference Eval

---
name: first-inference-eval
description: Evaluates the performance and scalability of a federated inference scheduling framework under varying request loads, measuring throughput, latency, and auto-scaling capabilities on distributed HPC resources. Use when the user wants to benchmark on ShareGPT, or asks about evaluating this task. Reports Request throughput (req/s).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13724
  bibtex_key: tanikanti2025first
  confidence: high
---

# first-inference-eval

> FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access — Tanikanti et al. (2025) (arXiv:2510.13724, 2025)

## What this evaluates

Evaluates the performance and scalability of a federated inference scheduling framework under varying request loads, measuring throughput, latency, and auto-scaling capabilities on distributed HPC resources.

## Datasets

- **ShareGPT** — total 1000; splits: test (1000)

## Metrics

- `Request throughput (req/s)` **(primary)** — range: other
  - The number of successful inference requests completed per second. Measures the overall request processing capacity of the system.
- `Output Token Throughput (tok/s)` — range: other
  - The total number of output tokens generated by the model per second across all successful requests. Reflects the effective generation speed.
- `Median End-to-End Latency (s)` — range: other
  - The time elapsed from sending a request to receiving the complete response. Reported as the median to reduce sensitivity to outliers compared to the mean.
- `Benchmark Duration (s)` — range: other
  - The total time taken to complete all requests in a given benchmark run. Provides an overall measure of workload processing speed.

## Input / output format

**Input**: Prompts from the ShareGPT dataset, sent as OpenAI-compatible API requests with specified target output lengths.

**Output**: Generated text responses (tokens) from the LLM, returned via the API.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    latencies = [r.timestamp_received - r.timestamp_sent for r in predictions]
    median_latency = sorted(latencies)[len(latencies)//2]
    total_time = max(r.timestamp_received for r in predictions) - min(r.timestamp_sent for r in gold)
    req_throughput = len(predictions) / total_time
    tok_throughput = sum(r.output_tokens for r in predictions) / total_time
    return {
        'request_throughput': req_throughput,
        'output_token_throughput': tok_throughput,
        'median_latency': median_latency,
        'benchmark_duration': total_time
    }
```

## Common pitfalls

- Confusing median latency with mean latency; the protocol explicitly reports median to avoid outlier sensitivity.
- Ignoring the 'infinite request rate' workload condition; peak performance tests saturate the server by sending all 1000 requests immediately, differing from steady-state load testing.
- Comparing FIRST directly to commercial APIs without accounting for external rate limits and hardware differences (HPC vs. cloud).

## Evidence (verbatim from paper)

> We focus on the following key metrics (Sagi, [2025]) to evaluate performance: Request throughput (req/s): The number of successful inference requests completed per second. This metric measures the overall request processing capacity of the system. Output Token Throughput (tok/s): The total number of output tokens generated by the model per second across all successful requests. This metric reflects the effective generation speed. Median End-to-End Latency (s): The time elapsed from sending a request to receiving the complete response. We primarily report the median latency, as it is less sensitive to outliers than the mean. This metric reflects the user-perceived responsiveness for a single request.

## Citation

```bibtex
@misc{tanikanti2025first,
  title={FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access},
  author={Tanikanti et al. (2025)},
  year={2025},
  note={arXiv:2510.13724}
}
```

- arXiv: 2510.13724