Model deprecations arrive on the vendor's schedule, not yours. After running the same fire drill four times, we wrote down the playbook that turns a model upgrade from a two-week scramble into a routine deploy.
The email arrives with a deprecation date roughly six months out. The model powering a production feature is being retired, there's a newer version available and the migration is described as straightforward. It rarely is.
We've now run this migration four times across client systems. The first one took three weeks and shipped with a regression we didn't catch until a customer reported it. The fourth took two days and was uneventful. The difference wasn't the models - it was having a playbook. This post is that playbook.
The core misconception worth naming up front: a model upgrade is not a dependency bump. A dependency bump changes code you can diff. A model upgrade changes a probability distribution you can't inspect, in ways that are invisible until they hit the specific inputs your users send.
Across four migrations, the failures clustered into four kinds. None of them were "the new model is worse."
Output format drift. The old model reliably wrapped JSON in a code fence. The new one sometimes didn't. Our parser was tolerant of the fence and intolerant of its absence. Every migration surfaced at least one parsing assumption that had been silently load-bearing for a year.
Prompt sensitivity shifts. Prompts accumulate compensating hacks - a "do not explain your reasoning" line added because the old model was chatty, a few-shot example added to correct a specific failure. Those hacks are tuned to a model that no longer exists. On the new model some become no-ops and some actively hurt. One of ours had a "be concise" instruction that on the newer model caused it to drop a required field.
Refusal and safety-boundary changes. A compliance-document extraction workload started refusing a small class of inputs the previous version had handled without comment. The content hadn't changed and the refusals weren't wrong exactly - the boundary had just moved. This is the failure mode that most reliably reaches a customer before it reaches your dashboard, because it's a small percentage of traffic and it looks like an error, not a regression.
Latency and cost profile changes. Reasoning-tuned models in particular change the token economics underneath a workload. One migration cut our per-call price and raised our per-call cost, because the new model emitted three times the output tokens on the same task. Nobody had modelled that.
1. Inventory the call sites before touching anything. Every place a model name appears, what it does, which prompt version it uses, what parses the output. On the first migration we found calls in a scheduled job nobody had thought about in eight months. If model identifiers are scattered as string literals through your codebase, that's the first fix - centralise them so the inventory is a config file, not an archaeology exercise.
2. Build the regression set from production traffic, not from imagination. Two to five hundred real requests, stratified to include the tails: the longest inputs, the weirdest documents, the ones that historically produced errors. Hand-written test cases systematically miss the inputs that break migrations, because you write test cases for the behaviour you understand.
3. Run both models over the set and diff the outputs. Not pass/fail against a rubric - a literal diff. The rubric tells you whether quality moved. The diff tells you how the behaviour changed, which is what you actually need to fix prompts. Most of our real findings came from reading fifty diffs, not from a score.
┌──────────────┐
prod traffic ─┤ replay set ├─┬─► old model ─┐
(n≈300) └──────────────┘ │ ├─► diff ─► triage
└─► new model ─┘
│
score both on eval suite ─┘
4. Fix prompts against the new model, not against both. The temptation is to find a prompt that works on old and new so you can flip back safely. Resist it - you end up with a prompt tuned for neither. Branch the prompt, keep the old one pinned to the old model and make the rollback switch flip both together.
5. Shadow before you cut over. Route a copy of live traffic to the new model, log both outputs, compare offline. Two weeks is usually enough to surface the low-frequency failures the replay set missed. This is where refusal-boundary changes show up.
6. Ramp, don't switch. 5%, 25%, 100%, with the same monitors you'd use for any deploy plus one extra: output-shape validity. Parse failure rate is the fastest leading indicator that something changed and it moves hours before quality metrics do.
Three things, all built during the earlier migrations.
The model identifier became a config value with a per-workload override, so a migration is a config change and a rollback is a config change. Obvious in hindsight; not how the first system was built.
The eval suite from our agent observability work already existed and already ran on every prompt change. Migration day was just another run of a thing we ran weekly. Teams that build the eval suite during the migration are doing two hard things at once under a deadline.
Prompts got version-pinned alongside the model. Prompt and model are a single unit - versioning them separately means the combination in production is implied rather than declared and implied combinations are what break.
Model deprecation is now a recurring operational event, not a one-off. The teams that handle it well aren't the ones with better models - they're the ones who treat the model as a versioned dependency with a proper upgrade path: centralised identifiers, a replay set drawn from real traffic, prompts pinned to the model they were tuned against and a ramp with output-validity monitoring.
Build that once and the next deprecation email is a scheduling question rather than a fire drill. Build it under deadline pressure during the migration itself and you'll ship the regression we shipped the first time.