Realtime Simulation Checking Eval

---
name: realtime-simulation-checking-eval
description: Evaluates the scalability and performance of a simulation-checking algorithm for timed automata under fairness assumptions. It measures how efficiently the algorithm verifies liveness properties and handles state-space explosion across parameterized real-time system benchmarks. Use when the user wants to benchmark on Fischer's timed mutual exclusion algorithm, CSMA/CD, Timed consumer/producer, Network of TAs, or asks about evaluating this task. Reports CPU time.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1007.0523
  bibtex_key: wang2010simulation
  confidence: high
---

# realtime-simulation-checking-eval

> Simulation-Checking of Real-Time Systems with Fairness Assumptions — Wang (2010) (arXiv:1007.0523, 2010)

## What this evaluates

Evaluates the scalability and performance of a simulation-checking algorithm for timed automata under fairness assumptions. It measures how efficiently the algorithm verifies liveness properties and handles state-space explosion across parameterized real-time system benchmarks.

## Datasets

- **Fischer's timed mutual exclusion algorithm** — total ?; splits: test (-1)
- **CSMA/CD** — total ?; splits: test (-1)
- **Timed consumer/producer** — total ?; splits: test (-1)
- **Network of TAs** — total ?; splits: test (-1)

## Metrics

- `CPU time` **(primary)** — range: seconds
  - Wall-clock execution time in seconds required to complete the simulation check.
- `memory consumption` — range: kilobytes
  - Total memory in kilobytes or megabytes consumed by the data-structures during state-space representation.

## Input / output format

**Input**: Timed automata models and specifications parameterized by the number of processes (m) and environment topology (linear, binary-tree, irregular).

**Output**: Boolean simulation check result, along with recorded CPU time (seconds) and memory consumption (KB/MB).

## Scoring recipe

```python
def evaluate(model, spec, fairness):
    start = time.time()
    result = run_simulation_check(model, spec, fairness)
    end = time.time()
    cpu_time = end - start
    memory = get_data_structure_memory()
    return {'check_result': result, 'cpu_time': cpu_time, 'memory': memory}
```

## Common pitfalls

- Benchmarks are parameterized by the number of processes (m), so performance scales non-linearly with system size.
- Memory reported is specifically for data-structures in state-space representations, not total system RAM.
- Values marked 'N/A' indicate the algorithm exceeded the 1800-second timeout or ran out of memory.

## Evidence (verbatim from paper)

> The CPU time used and the total memory consumption for the data-structures in state-space representations are reported. As can be seen, the performance of our new simulation against a common environment is significantly better than the classic one.

## Citation

```bibtex
@misc{wang2010simulation,
  title={Simulation-Checking of Real-Time Systems with Fairness Assumptions},
  author={Wang (2010)},
  year={2010},
  note={arXiv:1007.0523}
}
```

- arXiv: 1007.0523