Error Detection Hmc Eval

---
name: error-detection-hmc-eval
description: Evaluates a model's ability to detect classification errors and recover hierarchical multi-label constraints without prior knowledge. It probes the system's capacity to generate interpretable logical rules from failure patterns and improve downstream model consistency. Use when the user wants to benchmark on Military Vehicles, ImageNet50, OpenImage36, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.15192
  bibtex_key: kricheli2024error
  confidence: high
---

# error-detection-hmc-eval

> Error Detection and Constraint Recovery in Hierarchical Multi-Label Classification without Prior Knowledge — Kricheli et al. (2024) (arXiv:2407.15192, 2024)

## What this evaluates

Evaluates a model's ability to detect classification errors and recover hierarchical multi-label constraints without prior knowledge. It probes the system's capacity to generate interpretable logical rules from failure patterns and improve downstream model consistency.

## Datasets

- **Military Vehicles** — total 9444; splits: train (7555), test (1889); repo https://github.com/lab-v2/PyEDCR
- **ImageNet50** — total 67500; splits: train (65000), test (2500)
- **OpenImage36** — total 86400; splits: train (72000), test (14400)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall. For error detection, 'total error' is defined as applying a logical OR on all per-class error predictions before computing the metric.
- `balanced accuracy` — range: [0, 1]
  - Average of recall obtained on each class, computed on the total error predictions (logical OR of per-class errors).
- `F1-score (constraint recovery)` — range: [0, 1]
  - Harmonic mean of precision and recall comparing recovered constraints against a ground truth set of constraints not used during training.

## Input / output format

**Input**: Image inputs processed by a base vision model (ViT b_16 or DINOv2 s_14), along with model predictions, complementary granularity class outputs, and binary classification model outputs used to derive rule conditions.

**Output**: Binary error predictions per class and per sample, logical rules defining hierarchical constraints, and consistency scores for neurosymbolic training.

## Scoring recipe

```python
def compute_metrics(preds_per_class, gold_per_class, gold_constraints, recovered_constraints, test_set):
    # Total error detection (logical OR across per-class errors)
    total_error_pred = logical_or(preds_per_class)
    total_error_gold = logical_or(gold_per_class)
    bal_acc = balanced_accuracy(total_error_gold, total_error_pred)
    f1_err = f1_score(total_error_gold, total_error_pred)
    
    # Constraint recovery
    f1_const = f1_score(gold_constraints, recovered_constraints)
    
    # Consistency
    violations = count_samples_violating_gold_constraints(test_set)
    consistency = 1.0 - (violations / len(test_set))
    
    return bal_acc, f1_err, f1_const, consistency
```

## Common pitfalls

- Total error is explicitly defined as the logical OR across all per-class error predictions, not a per-sample average or independent binary classification.
- Constraint recovery experiments deliberately omit labels of specific fine-grain classes during training to test noise tolerance, meaning the model must generalize without seeing those ground-truth labels.
- Rules for both f-EDR and DetRuleLearn use identical conditions; performance differences stem from the objective function optimization, not feature extraction or condition selection.

## Evidence (verbatim from paper)

> Results of this experiment are shown in Table[3] where we provide the balanced accuracy and F1-score of the total error - which is defined as applying a logical OR on all the per-class error classes.

## Citation

```bibtex
@misc{kricheli2024error,
  title={Error Detection and Constraint Recovery in Hierarchical Multi-Label Classification without Prior Knowledge},
  author={Kricheli et al. (2024)},
  year={2024},
  note={arXiv:2407.15192}
}
```

- arXiv: 2407.15192