Ready Jurist One Eval

---
name: ready-jurist-one-eval
description: Evaluates the ability of LLM-based agents to perform interactive, procedural legal tasks in dynamic, multi-turn Chinese legal environments. It probes knowledge retrieval, document drafting, and court proceeding navigation, measuring both task completion and adherence to legal procedures. Use when the user wants to benchmark on J1-ENVS, or asks about evaluating this task. Reports average scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.04037
  bibtex_key: zheng2025readyjuristone
  confidence: medium
---

# ready-jurist-one-eval

> Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments — Zheng Jia et al. (2025) (arXiv:2507.04037, 2025)

## What this evaluates

Evaluates the ability of LLM-based agents to perform interactive, procedural legal tasks in dynamic, multi-turn Chinese legal environments. It probes knowledge retrieval, document drafting, and court proceeding navigation, measuring both task completion and adherence to legal procedures.

## Datasets

- **J1-ENVS** — total ?; splits: test (-1)

## Metrics

- `average scores` **(primary)** — range: percent
  - Average of task scores across all J1-ENVS environments except VER. Computed by evaluating task completion per scenario and averaging the results.
- `completion rate` — range: percent
  - Proportion of samples that successfully reach designated procedural stages in civil or criminal court environments.
- `behavior consistency` — range: other
  - Average rating on a 1 to 10 scale assessing how well an environment role's behavior matches its predefined profile during multi-turn interactions.

## Input / output format

**Input**: Multi-turn interactive legal scenarios in Chinese, including case backgrounds, procedural rules, and role-specific instructions. Agents receive environment state updates and task prompts.

**Output**: Agent actions/responses in natural language or structured commands to interact with the environment, complete tasks, and navigate court stages.

## Scoring recipe

```python
def compute_metrics(predictions, gold, env_states):
    # 1. Average Score (excluding VER)
    env_scores = []
    for env in environments:
        if env == 'VER': continue
        score = evaluate_task_completion(predictions[env], gold[env])
        env_scores.append(score)
    avg_score = sum(env_scores) / len(env_scores)

    # 2. Completion Rate
    completed = sum(1 for s in samples if s.reaches_designated_stage)
    completion_rate = completed / len(samples)

    # 3. Behavior Consistency (1-10 scale)
    consistency = [rater.rate(profile, behavior) for rater, profile, behavior in interactions]
    avg_consistency = sum(consistency) / len(consistency)
    return avg_score, completion_rate, avg_consistency
```

## Common pitfalls

- Legal-specific LLMs often underperform general-purpose models in dynamic settings despite strong static benchmark scores, as they lack interactive procedural capabilities.
- Evaluation relies on LLM-based scoring (GPT-4o) for most metrics, with human evaluation only covering 10% of samples, which may limit reliability compared to full human annotation.
- Metrics like P_FS and completion rates are environment-specific (civil vs. criminal court), making direct cross-environment comparison difficult without normalization.

## Evidence (verbatim from paper)

> To assess overall performance, we compute the average scores across all environments except V ER, and rank the models by size. ... measure the completion rate by computing how many samples successfully reach designated stages. ... Both GPT-4o and human evaluators rate the consistency between each role’s profile and its behavior on a scale of 1 to 10

## Citation

```bibtex
@misc{zheng2025readyjuristone,
  title={Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments},
  author={Zheng Jia et al. (2025)},
  year={2025},
  note={arXiv:2507.04037}
}
```

- arXiv: 2507.04037