Patent Ce Eval

---
name: patent-ce-eval
description: This benchmark evaluates the quality of generated patent claims against expert-annotated reference claims across five dimensions: feature completeness, conceptual clarity, terminology consistency, logical linkage, and overall quality. It probes a model's ability to capture patent-specific linguistic precision, legal formality, and structural requirements rather than just surface-level text overlap. Use when the user wants to benchmark on Patent-CE, or asks about evaluating this task. Reports Kendall-Tau correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.11095
  bibtex_key: jiang2025patentce
  confidence: high
---

# patent-ce-eval

> Towards Better Evaluation for Generated Patent Claims — Jiang et al. (2025) (arXiv:2505.11095, 2025)

## What this evaluates

This benchmark evaluates the quality of generated patent claims against expert-annotated reference claims across five dimensions: feature completeness, conceptual clarity, terminology consistency, logical linkage, and overall quality. It probes a model's ability to capture patent-specific linguistic precision, legal formality, and structural requirements rather than just surface-level text overlap.

## Datasets

- **Patent-CE** — total 1228; splits: test (-1); repo https://github.com/scylj1/PatClaimEval

## Metrics

- `Kendall-Tau correlation` **(primary)** — range: [-1, 1]
  - Measures the number of concordant and discordant pairs between two ranked lists to assess rank correlation. Values range from -1 (perfect disagreement) to 1 (perfect agreement).
- `Spearman correlation` — range: [-1, 1]
  - Computes the Pearson correlation between the rank variables of two lists. It is more sensitive to large rank differences than Kendall-Tau.
- `accuracy` — range: [0, 1]
  - Proportion of correctly predicted classification labels in the three-way classification setup.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for the three-way classification task.

## Input / output format

**Input**: A pair of texts: a reference patent claim and a candidate (generated) patent claim.

**Output**: A quality score or rank for the candidate claim, or a three-way classification label (e.g., acceptable/unacceptable/conditional), depending on the metric being evaluated.

## Scoring recipe

```python
def compute_kendall_tau(predictions, gold):
    concordant = discordant = tied = 0
    for i in range(len(predictions)):
        for j in range(i + 1, len(predictions)):
            diff_p = predictions[i] - predictions[j]
            diff_g = gold[i] - gold[j]
            if diff_p * diff_g > 0: concordant += 1
            elif diff_p * diff_g < 0: discordant += 1
            else: tied += 1
    tau = (concordant - discordant) / (concordant + discordant + tied)
    return tau
```

## Common pitfalls

- The dataset explicitly lacks source texts/context, so metrics requiring context (like AlignScore or UniEval) cannot be applied as intended.
- For classification tasks, score differences below 10^-4 are treated as equivalent labels, which can artificially inflate accuracy/F1 if not handled.
- Evaluating patent claims requires multi-dimensional assessment (completeness, clarity, consistency, linkage, quality) rather than relying solely on n-gram overlap or single scalar scores.

## Evidence (verbatim from paper)

> We used the Kendall-Tau correlation to assess the overall alignment with human judgment, following the approach of previous work by Zuo et al. ([2024]). This correlation metric evaluates the consistency of the global ranking while disregarding minor errors in individual predictions. We additionally report the Spearman correlation. Compared to Kendall-Tau, Spearman is more sensitive to large rank differences, providing a complementary perspective on the metric ability to predict relative claim quality.

## Citation

```bibtex
@misc{jiang2025patentce,
  title={Towards Better Evaluation for Generated Patent Claims},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2505.11095}
}
```

- arXiv: 2505.11095