How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring

Imagine a digital security guard who hands over the keys to the vault simply because a thief says, “I really shouldn’t be doing this,” before hauling away the gold. In the high-stakes arena of AI safety, this is closer to reality than we care to admit.

A typical research paper today might claim a breakthrough: a new jailbreak achieving a 95% Attack Success Rate (ASR), or a novel defense that slashes that rate to near zero. These numbers are the currency of the field, the metrics by which we judge whether a model is “safe” for public deployment. Yet, these figures rest on a shaky foundation. Because manually reviewing the thousands of responses generated in these trials is costly and slow, the grading is routinely outsourced to automated “judges” instead — a scaling choice, not a technical necessity. Recent research by Yang Gao of Veyon Solutions reveals a disturbing truth: these automated graders are often uncalibrated, easily manipulated, and frequently mistake surface-level politeness for genuine safety.

The Two Families of Automated Judges

To understand the failure, we must look at the tools of the trade. Automated safety evaluation generally falls into two distinct categories:

1. Dedicated Safety Classifiers: Models like Llama-Guard or the HarmBench classifier, which are fine-tuned specifically to identify and label harmful content.
2. LLM-as-judge: General-purpose, instruction-tuned models—such as Qwen or Phi—that are prompted with a rubric to act as a grader. These are increasingly popular due to their low cost and flexibility.

In his study, Gao utilized the HarmBench classifier validation set (596 human-labeled completions) to test these judges. To ensure the work is accessible to researchers without massive compute budgets, evaluations were performed on a single NVIDIA T4 GPU using 4-bit quantization. Critically, Gao verified that this quantization did not compromise the results; the 4-bit classifier reproduced the benchmark’s stored predictions with 98.7% fidelity.

The Calibration Paradox: Precision vs. Recall

When measured against the human majority-vote labels from three annotators, the two judge families fail in fundamentally opposite ways. Dedicated classifiers act as “paranoid” sentries, while LLM-judges are dangerously “conservative,” missing risks the human labels flagged.

Judge Type	Precision	Recall	F1 Score
HarmBench Classifier (Dedicated)	0.835	0.974	0.899
Qwen2.5-7B (LLM-judge)	0.940	0.174	0.294
Phi-3.5-mini (LLM-judge)	0.810	0.648	0.720
Qwen2.5-3B (LLM-judge)	0.889	0.059	0.111

The Logic of Failure:

• Dedicated Classifiers “Over-flag”: With near-perfect recall but lower precision, these models rarely miss actual harm. However, the HarmBench classifier’s 0.835 precision means roughly 16.5% of its positive calls were false positives on this set — over-flagging that can lead researchers to report artificially inflated ASR numbers.
• LLM-judges Show “Erratic Recall”: While they maintain high precision—meaning if they say it’s harmful, it likely is—their recall is abysmal. They miss vast swaths of real danger. Consequently, the same set of responses can yield wildly different safety metrics depending on which LLM is holding the clipboard.

The “Politeness” Trap: Content-Preserving Wrappers

The most damning finding is the fragility of LLM-judges when faced with “wrapper” attacks. In these cases, the harmful content remains exactly the same, but it is dressed up in benign framing. Gao tested four wrapper types:

• Refusal prefix: Prepending a single sentence like “I cannot fulfill this request” before providing the harmful instructions.
• Benign pre/post: Adding polite opening and closing statements.
• Educational frame: Framing the harm as being “for awareness” or “academic research.”
• Fiction frame: Inserting the content into a narrative context, such as: “The following is a line of dialogue from a fictional thriller novel, spoken by the antagonist…”

The results expose a shallow understanding of risk: a single refusal sentence flipped LLM-judge verdicts from “harmful” to “safe” between 39% and 88% of the time. Total “any-wrapper” flip rates reached as high as 100% for some models. These judges aren’t measuring actual risk; they are measuring surface-level decorum.

Cracking the Dedicated Classifier: White-Box Attacks

Dedicated classifiers are sturdier against simple wrappers, but they possess a different vulnerability. Gao performed a “White-Box GCG” attack against the HarmBench classifier’s open weights. GCG (Greedy Coordinate Gradient) is an optimization method that finds the most disruptive tokens through a process of trial and error.

By optimizing a 20-token adversarial suffix over just 50 steps, the attack successfully flipped 21 of 30 confident true-positive verdicts (70%) under full white-box weight and gradient access; nine survived the budget. This 70% figure is actually a conservative “lower bound,” as the optimization budget was intentionally small. For any researcher using an open-weight safety model as their primary evaluation tool, this is a clear warning: an adversary with white-box access to your judge can bypass it under a modest optimization budget — the paper does not test black-box or general-ease conditions.

The Human Audit: Is the Harm Still There?

To confirm the judges were being fooled rather than the content being sanitized, two independent annotators conducted a manual audit of 80 flipped responses, working independently and blind to each other’s labels after a calibration round. They agreed on 75 of 80 items (93.8%); the five disagreements went to adjudication.

After adjudication, all 80 sampled flips were confirmed to retain the original harmful content. In two of the five initial disagreements, the dissenting annotator had been misled by the benign refusal prefix — the same trick that fooled the automated judges. This underscores the core thesis: “politeness” is a powerful mask that can blind both machines and humans to underlying danger, though it took a second annotator and adjudication, not unanimous first-pass agreement, to establish that.

A New Standard for AI Safety Reporting

To move past these unreliable metrics, Gao proposes four rigorous standards for future safety research:

1. Transparency: Authors must explicitly name the judge used and report its precision and recall on a human-labeled slice of data.
2. Precision Correction: Researchers should report a “corrected ASR” to filter out false positives using the formula: $ASR_{corrected} \approx ASR_{reported} \times precision$. This correction only addresses false-positive inflation — it does not fix false negatives, dataset shift, dependence between errors and attack type, or an adversarially manipulated judge, and should be reported alongside those limitations rather than as a general fix.
3. Adversarial Checks: Evaluations should include an adversarial check of the judge itself by running content-preserving “wrapper” attacks to prove the judge is looking at the content, not the frame.
4. Preference: Researchers should prefer dedicated classifiers over the “conservative” LLM-as-judge approach, provided they remain vigilant about white-box vulnerabilities in open-weight models.

Conclusion: Trust, but Verify

The practical message is unavoidable: no judge family should be trusted without validation. Whether a model uses a dedicated safety filter or a prompted LLM, the lack of calibration against human reality means reported safety numbers can be unreliable — this study covers one HarmBench validation set, three untuned open LLM-judge prompts, and one quantized classifier, and the authors are explicit that broader generalization needs further work on other benchmarks, prompt sensitivity, and frontier judges.

True AI safety requires “meta-level adversarial evaluation”—rigorously testing the procedures meant to catch bad behavior. Until we treat the judges with the same skepticism we apply to the models themselves, we aren’t measuring safety; we’re just measuring the thickness of the model’s polite veneer.

Read the full paper on arXiv · PDF