criticalP1
Silent quality regression — refund verification skipped after model downgrade (gpt-oss-20b)
Open· quality· First seen 7/1/2026, 9:59:59 AM
Where
What / Why
checkout-planner was switched from claude-3-5-sonnet-20241022 to gpt-oss-20b on May 26 to reduce inference cost. The smaller model silently omits the verify_refund step. Refunds are issuing without validation. No errors, normal latency — agent self-reports success on all affected runs. ~29% failure rate detected by RouteIQ quality checks.
Actions
Cost Savings
$211.00
$0.008/run × 180 runs/hr × 46h ≈ $211
Cost/run
$0.008
Runs/hr
180
Hrs to catch
46h
Interventions (1)
| When | Action | By | Reason | Prevented $ |
|---|---|---|---|---|
| 7/3/2026 | deploy_rollback | usr_demo | Rolled back to claude-3-5-sonnet-20241022 (v1.3.1) — gpt-oss-20b confirmed as cause of silent quality regression | $211.00 |
Audit Trail
alert.openby usr_demo7/1/2026, 9:59:59 AM
{"severity":"p1"}
alert.resolveby usr_demo7/3/2026, 6:29:59 AM
{"resolution":"model_rollback_deployed"}