Copyright Tracking Tmr Eval

---
name: copyright-tracking-tmr-eval
description: Evaluates the robustness of copyright tracking methods in fine-tuned Large Vision-Language Models (LVLMs) by measuring whether adversarial image triggers can consistently elicit a predefined target response after the model has been adapted on various downstream datasets. Use when the user wants to benchmark on ImageNet 2012 (validation subset), V7W, ST-VQA, TextVQA, PaintingForm, MathV360k, ChEBI-20, or asks about evaluating this task. Reports target match rate (TMR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.16593
  bibtex_key: wang2025tracking
  confidence: high
---

# copyright-tracking-tmr-eval

> Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images — Wang et al. (2025) (arXiv:2502.16593, 2025)

## What this evaluates

Evaluates the robustness of copyright tracking methods in fine-tuned Large Vision-Language Models (LVLMs) by measuring whether adversarial image triggers can consistently elicit a predefined target response after the model has been adapted on various downstream datasets.

## Datasets

- **ImageNet 2012 (validation subset)** — total 200; splits: trigger_source (200)
- **V7W** — total 28000; splits: train (28000)
- **ST-VQA** — total ?; splits: train (-1)
- **TextVQA** — total ?; splits: train (-1)
- **PaintingForm** — total 20000; splits: train (20000)
- **MathV360k** — total 50000; splits: train (50000)
- **ChEBI-20** — total ?; splits: train (-1)

## Metrics

- `target match rate (TMR)` **(primary)** — range: percent
  - Proportion of trigger queries (image-text pairs) where the model's generated response exactly matches the predefined target answer. Calculated as (matches / total queries) × 100%.

## Input / output format

**Input**: Adversarial trigger image (perturbed from a clean ImageNet image) paired with a specific question (e.g., 'Q: Detecting copyright.').

**Output**: Text response generated by the LVLM.

## Scoring recipe

```python
matches = 0
total = 0
for img, question, target_answer in trigger_queries:
    response = model.generate(img, question)
    if response.strip() == target_answer.strip():
        matches += 1
    total += 1
tmr = (matches / total) * 100
```

## Common pitfalls

- The fine-tuning datasets (V7W, ST-VQA, etc.) are used to adapt the model, not as the evaluation test set. Evaluation is strictly performed on the 1000 trigger queries.
- TMR varies significantly across different trigger QA pairs; reporting a single average without specifying the QA pair can be misleading.
- Standard adversarial attacks (Ordinary) fail post-fine-tuning due to overfitting; the method relies on parameter learning dynamics (PLA) to maintain robustness.

## Evidence (verbatim from paper)

> Table 1: A comprehensive comparison of our proposed method PLA with established baseline methods on the copyright tracking performance of fine-tuned models across 6 datasets. The evaluation metric is the target match rate (TMR), which measures the proportion of triggers successfully eliciting outputs that match the trigger target from the model.

## Citation

```bibtex
@misc{wang2025tracking,
  title={Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.16593}
}
```

- arXiv: 2502.16593