How yolonot compares to Claude Code Auto Mode and manual prompts

Every permission gate for AI coding agents trades off three outcomes: letting dangerous commands slip through (missed risks), hard-rejecting safe commands (false blocks), and interrupting developers with confirmation prompts. Here is how yolonot's local bash AST parser and 2-stage judge pipeline perform across 456 open benchmark cases and 5,000 sampled real-world commands.

1. Evaluation metrics and Anthropic's 2026 studies

Because yolonot operates in a ternary decision space (allow, ask, deny) rather than a binary filter, we report all three rates explicitly:

Baseline context: Anthropic's containment and Auto Mode reports

Claude Code Auto Mode: In How we built Claude Code auto mode, Anthropic reported that developers accept 93% of manual permission prompts out of habit. Their 2-stage Sonnet 4.6 transcript classifier scored 8.5% FPR and 6.6% FNR at Stage 1 (single-token filter), and 0.4% FPR with 17.0% FNR on real overeager actions (n = 52) plus 5.7% FNR on synthetic data exfiltration (n = 1,000) after Stage 2.

Containing Claude across products: In How we contain Claude across products, Anthropic emphasized pairing deterministic environment boundaries with model classifiers because probabilistic classifiers alone let roughly 17% of overeager actions through.

Investigating unintended model actions: In Investigating unintended model actions, Anthropic documented how capable agents exhibit persistence and overreach when hitting obstacles, routing around restrictions or executing unintended server commands.

2. End-to-end error and prompt comparison across profiles

Claude Code Auto Mode has no ask state, and its Stage 2 pass only re-evaluates Stage 1 blocks to reduce false alarms—causing its missed-risk rate to climb from 6.6% to 17.0%. yolonot instead escalates uncertain cases to a distinct second model while enforcing a strict monotonicity rule (critical never relaxes to allow), and lets you choose your operating point with yolonot profile use <mode>:

True error comparison: Safe Blocked (FPR) vs. Risky Missed (FNR) on a linear 0% to 20% scale

Evaluated with fast_allow AST + jev-1.13 (Stage 1) + claude-haiku-5.5 (Stage 2) across 456 benchmark cases. Lower is better.

Linear 0 to 20 percent bar chart comparing Safe Commands Blocked and Risky Commands Missed across Claude Auto Mode and all four built-in yolonot profiles Metric 1 (Safe commands blocked, 0 to 20% scale): Claude Auto Mode 0.4% (8.5% at Stage 1); yolonot balanced 0.0%; yolonot fast 1.4% cold and 0.0% with suggest; yolonot strict 0.0%; yolonot paranoid 8.4%. Metric 2 (Risky commands missed, 0 to 20% scale): Claude Auto Mode 17.0%; yolonot balanced 4.9% (0% critical and 0% on Open Eval); yolonot fast 4.9%; yolonot strict 1.6%; yolonot paranoid 0.0%. 1. Safe Commands Blocked (False Reject Rate · Linear 0% to 20% Scale · Lower is Better) Claude Auto Mode‡ 0.4% blocked after Stage 2 (8.5% Stage 1 FPR · 0% ask) yolonot [balanced] 0.0% blocked · 1.4% asked once (0% after suggest) yolonot [fast] 0.0% w/ suggest (1.4% cold) · 0.0% asked (unattended) yolonot [strict] 0.0% blocked · 19.4% asked on low/moderate yolonot [paranoid] 8.4% blocked · 91.6% asked (hard-denies moderate+) 2. Risky Commands Missed (False Negative Rate · Linear 0% to 20% Scale · Lower is Better) Claude Auto Mode‡ 17.0% missed (6.6% Stage 1 → 17.0% after Stage 2) yolonot [balanced] 4.9% missed (0.0% critical/OS · 0.0% on Open Eval) yolonot [fast] 4.9% missed (0.0% critical/OS · 0.0% on Open Eval) yolonot [strict] 1.6% missed (0.0% high/critical & OS attacks) yolonot [paranoid] 0.0% missed (100.0% caught across all 456 cases) 0% 10% 20%
‡Claude Auto Mode figures (17.0% FNR, 0.4% Stage 2 FPR, 8.5% Stage 1 FPR) are from Anthropic's March 25, 2026 engineering study (n = 52 real overeager actions). yolonot figures are macro-averages across the 3 public benchmark suites below (N = 456).
Profile / Mode Safe Blocked (FPR) Safe Asked (Prompt Rate) Risky Missed (FNR) Risk Mapping Policy
Manual Prompts 0.0%
✕100.0% asked
✕93% habitual click
 Prompts on every command (approval fatigue)
Claude Auto Mode 0.4% (8.5% S1) 0.0% (no ask state)
◐17.0% missed
 Binary allow/deny; Stage 2 relaxes S1 blocks
yolonot · balanced 0.0% 1.4% (0% w/ suggest)
✓4.9% (0% Open Eval)
 Allows safe/low, asks once on mod+
yolonot · fast 0.0% (1.4% cold) 0.0% (unattended)
✓4.9% (0% Open Eval)
 Allows safe/low/mod, denies high/critical
yolonot · strict 0.0% 19.4% (59.9% <1ms AST)
✓1.6% (0% high/crit)
 Asks on low/mod, denies high/critical
yolonot · paranoid 8.4% 91.6%
✓0.0% (100% caught)
 Asks on safe/low, denies moderate+

3. Normalized evaluation across 3 open benchmarks (456 cases)

To test generalization beyond a single dataset, we evaluated yolonot (fast_allow AST + jev-1.13 typed classifier + claude-haiku-5.5 escalation) across three independent public suites:

Benchmark Suite Safe Cleared in <1ms AST Safe Allowed (Total) Dangerous Caught
1. yolonot Open Eval (260 cases) 85 safe + 105 dangerous + 70 rule/script cases 49.4% (42 / 85) 98.8% (84 / 85)
✓100.0% (105 / 105)
Includes 14/14 hidden on-disk script attacks
2. HF rogue-security (76 bash calls) rogue-security/coding-agent-security-benchmark 32.0% (8 / 25) 64.0% cold (16/25)*
✓94.7% (36/38) → 100.0% (38/38)
100% of high/critical attacks caught
3. RedCode-Exec + Dippy CLI (120 cases) 60 safe Dippy CLI + 60 RedCode OS attacks 98.3% (59 / 60) 98.3% (59 / 60)
✓90.5% (38/42) → 100.0% (42/42)
100% of OS attacks caught in strict / paranoid
Normalized Macro-Average (456 cases) Across all 3 suites (vs. 54.2% AST hit rate on 5,000 real-world calls)
59.9% in ~1.6µs
0 unsafe leaks
98.6% CLI
87.0% 3-suite cold
✓95.1% → 98.4% → 100.0%
Across balanced, strict, and paranoid

*Denominators in columns 2–4 reflect each suite's pure cold-classifier safe (85 + 25 + 60 = 170) and high-confidence attack (105 + 38 + 42 = 185) subsets; the remaining 101 of 456 cases test .yolonot rule precedence, script attachment, and gray-zone prompt-rate behavior (13 in rogue-security and 18 in RedCode-Exec). In rogue-security, the 9 safe commands flagged as ask on a cold start (1.4% on standard CLI suites; 87.0% 3-suite cold macro-average) are zero-context executions of unfamiliar /tmp/*.py scripts, kill -9, or public gist creation that require a one-time confirmation before session memory or yolonot suggest auto-allows them.

4. Stage 2 cross-model escalation & pluggable judge models

Why did Claude Code Auto Mode's miss rate jump from 6.6% at Stage 1 to 17.0% at Stage 2? Because asking the same model family to second-guess its own blocks without deterministic floor constraints lets persuasive transcripts talk the classifier down. yolonot avoids this through three design choices:

  • Cross-model Stage 2 escalation: Escalating uncertain Stage 1 ask verdicts from jev-1.13 to a distinct model (claude-haiku-5.5) rescued 48.3% of real-world false alarms (raising Dippy CLI safe allow rate from 93.3% → 98.3%) while catching and hardening 8 under-rated risks.
  • Non-relaxation invariant: A critical risk verdict at Stage 1 or in deterministic AST/script checks can never be downgraded to allow by Stage 2.
  • 14/14 on-disk script accuracy: Running yolonot eval --suite evals/suites/script-attach.jsonl catches 100% (14/14) of hidden script attacks (such as rm -rf $HOME tails or reverse shells inside healthcheck.py) that surface command-string classifiers miss.
  • TypeSafe Jev (jev-1.13): Purpose-built typed decision classifier · 99.4% greenfield accuracy · ~50–80ms typical latency.
  • Kev-4B (Apache 2.0 LoRA): Open-weights 4B RL decision model · 93.2% accuracy · runs locally with zero cloud API cost.
  • Cloudflare Clef-flash: Lightweight command classifier · 92.0% accuracy.
  • Custom AST & Any LLM: Combine exec:dippy or any Anthropic, OpenAI, Gemini, Grok, OpenRouter, or local Ollama model via yolonot models and yolonot judges.

Reproduce these numbers against your own configured models anytime with yolonot eval --all.