Chapter 13 — War stories: six postmortems and what each one cost

Every incident below is real, timestamped in git, and shipped a permanent change to how we work. The format is the same each time — symptom, diagnosis, fix, lesson — because the lesson is the only part worth keeping. Read them as a set and a meta-lesson emerges: almost every failure was a system telling us a comforting lie.

Story 1: The edit button that created invoices

Symptom. User report: "On clicking Edit invoice button, new invoice page is opened. Not able to edit invoices." Plus, from the same user: double-clicking Create produced duplicates.

Diagnosis. Two bugs stacked. (a) The editor read its route parameter in the constructor — but Angular binds route inputs after construction, so the id was always null and "Edit" rendered a blank "New" form; saving minted a fresh invoice. (b) Deeper: the invoice repository loaded entities without line items — line updates 404'd, and adding a line recomputed totals against an empty set, silently zeroing an invoice's math. One confused click had uncovered a data-corruption class.

Fix. An effect() that reacts when the route input binds; Include(LineItems) at the repository level (also fixed quote conversion losing its lines); create-success now navigates away so a second click can't duplicate.

Lesson. Reproduce in a real browser before theorizing — the visible bug was framing timing, but only reproduction exposed the corruption underneath. And domain integrity shouldn't depend on every call site remembering an Include.

Story 2: The hand-edited database (twice)

Symptom. API containers crash-looping at startup on deploy; logs show column already exists / relation already exists from the migration engine.

Diagnosis. Schema changes had been applied by hand (psql) on the production database — once a table, once a column — while the proper migration for the same change sat in the repo. The migration collided with the hand-made reality, threw, rolled back, and the container died at boot. Both times.

Fix. Emergency reconciliation (apply the missing column, record the migration in the history table), then rewrite the migrations idempotently (CREATE TABLE IF NOT EXISTS, ADD COLUMN IF NOT EXISTS) so they're safe on every database state. New repo-wide rule, now in the developer docs: never apply schema changes out-of-band; if forced, the follow-up migration must tolerate reality.

Lesson. A migration isn't a formality — it's the only writer of schema. Two writers means one of them is lying to the other.

Story 3: The deploy queue zombie

Symptom. Every deploy sat "pending" forever — while runners were online and idle. Cancelling and re-running changed nothing.

Diagnosis. The concurrency group that serializes deploys had a zombie holder: a cancelled run that never released the lock. New runs were created pending with zero jobs — the job row only appears once the lock is acquired — so the UI showed an eternal, healthy- looking queue.

Fix. Renaming the concurrency group sidesteps the zombie entirely (production-deploy-v2). Diagnosis technique worth stealing: zero jobs + idle runners = deadlock, not queue. The jobs API tells you instantly which one you have.

Lesson. "Pending forever" is a distinct failure mode. Know its signature, because the queue UI is constitutionally incapable of admitting it.

Story 4: The email that bounced to everyone

Symptom. Feedback notifications bouncing as MAILER-DAEMON delivery failures — user reports reaching nobody.

Diagnosis. The notification address lived on a domain whose mail routes pointed at Cloudflare Email Routing — and while the rules existed, the destination address was unverified. Every bounce also masked a second gap: notifications had no Reply-To, so any manual reply would have gone to a dead sender.

Fix. Notifications moved to a verified inbox; Reply-To now points at the reporter; and any mail-routing change gets a probe email — config-saved is not delivered. (Also learned to read MX records before assuming Google hosted everything.)

Lesson. Email infrastructure needs end-to-end tests like every other infrastructure. A saved rule is a hypothesis; a received probe is a result.

Story 5: The partial upgrade that broke the build

Symptom. Deploy "succeeded" — backends healthy — but the SPA build had failed inside it with npm ERESOLVE. New code was live on the API; the site itself was yesterday's.

Diagnosis. A dependency bot bumped three Angular packages to 22.1.4 while the rest of the family stayed on 22.1.3. Angular's peer dependencies demand exact-version alignment, so npm install became unresolvable. Why didn't the deploy fail loudly? It overlapped with story-3-era pipeline bugs — the failure was surfaced only after the queue was fixed.

Fix. Align the entire @angular/* family + Material/CDK in one commit, regenerate the lockfile, and add the rule to the docs: on any Angular bump, move the whole family together.

Lesson. Ecosystems with strict peer-graphs turn "small" upgrades into set-theory problems. When a bot proposes a partial set, the set is the unit of review.

Story 6: The version diamond

Symptom. A telemetry/health-check upgrade crashed the API containers into a restart loop the moment it deployed.

Diagnosis. Our internal platform packages required MediatR 14; the app pinned MediatR 12. Both versions in one process resolved to a runtime MissingMethodException-style explosion — the classic dependency diamond, invisible at compile time in the right (disastrous) packaging layout.

Fix. The interim revert is documented (that feature was withdrawn same-day); the durable fix came later — replacing MediatR entirely with a tiny MIT-licensed in-process mediator with no version gravity, and bumping the platform packages as a set. Chapter 03's architecture now documents the post-migration world.

Lesson. Your dependency graph is architecture. Any package that everyone transitively requires becomes a load-bearing wall — pick ones with a migration story, or none at all.

The template (steal this)

flowchart LR
    S["SYMPTOM
what was felt/seen"] --> D["DIAGNOSIS
root cause,
with evidence"]
    D --> F["FIX
code + tests + docs"]
    F --> L["LESSON
the rule that makes
this class impossible"]
    L -.rehearse.-> S

Postmortems earn their keep only if the lesson becomes a check, a doc line, a gate, or an idempotent default — otherwise you're just collecting scars. Every lesson above now lives in the repo's developer landmines file, where the next 2 a.m. version of us will actually read it.

Next: Appendix A — Glossary, checklists, and the video storyboard — every term defined, every launch step listed, and the chapter-to-scene map for the 4K series.*