How we keep the Renewal Manager from getting it wrong

1. An eval catalog of 134 cases, gated before every push

The worker has a 134-case eval catalog. It is not a vanity number — each block pins a specific safety property:

• 30 fabrication — the worker must never cite a source the evidence bundle doesn't contain, and must abstain on empty evidence.
• 20 contamination — one planted high-severity account inside a batch must not leak its risk onto the others.
• 32 classification — expansion / at-risk / likely-churn, held to an F1 score ≥ 0.75 per class.
• 20 diary quality — each health entry must be specific to the account and grounded in its cited evidence, not boilerplate.
• 7 number fabrication — any unverified dollar figure or percentage is stripped before it can reach a draft.
• 25 account health — the daily health read must not invent a transition, and a planted risk must not contaminate its neighbours.

• Before every push, deploy-blocking: the catalog must hold at least 100 well-formed cases (ADR-0029 §8), the output validator must catch a planted fabricated citation, and the scoring and routing helpers must behave. A deploy cannot ship past a failure.
• Against the live models, on demand: the same 134 cases run against the real worker and its supervisor before a model or prompt change goes out. We publish the suite's structure and thresholds — not a pass-rate.

2. Independent supervision — a second, stronger model

The highest-stakes reads run twice: Claude Sonnet drafts, and Claude Opus — a separate, stronger model with its own adversarial instructions — reviews it. Both come from one vendor, Anthropic; the independence is in the second model and its brief, not in a second company. On disagreement the task routes to a human — never to a third model, and never to a silent guess.

3. It refuses what it shouldn't do

The role's capability matrix is enforced in code, not just asked for in a prompt. Pricing exceptions, SLA and refund promises, and legal claims are routed to a human instead of guessed. Autonomy is earned per topic and opt-in — the AI drafts from day one, but a human approves the send until you decide otherwise.

4. Every claim links back to its source

A read isn't an opaque health score — every signal links to the exact Stripe invoice, HubSpot deal, or support ticket it came from, and the schema rejects any citation the source can't confirm. Before a save email ships, every dollar and percentage in it is re-checked against those facts; a figure that doesn't match holds the send rather than shipping a wrong number.

5. What a scan costs — and where it is capped

One daily review of one account costs $0.0108 in model spend, measured on production traffic. The subscription is flat; the cap below is a kill switch against a runaway loop, not a meter you pay against.

Four independent cost guardrails sit under this: a hard limit at the model provider, a budget per task estimated before it runs, this per-plan daily cap, and a global daily kill switch.

6. What it cannot see — and says so

Every signal comes from a source you connect. The map below is the same registry the setup screen reads: each source, whether you can connect it today, and the honest line about what the Renewal Manager cannot say without it.

How coverage gates autonomy

• Billing only — a preview: payment risk is surfaced, but no churn verdict is issued and nothing is sent on its own.
• Billing plus at least one leading signal (usage, CRM or support) — a real churn verdict, and autonomous sending only if you opt in.
• Billing plus all three leading families — the highest-confidence read.

Questions from a security review? Talk to us — we'll walk your team through the eval suite, the audit log, and our data posture (Service Provider under CCPA §1798.140, not a data broker).