🦊 FoxyInvoice

Chapter 07 — CI/CD: push to `main` and it's live

There is no staging environment in this system. There is main, and main is production. That sounds reckless until you see the gates — and the alternative for a solo operator (a staging box you maintain, promote to "when there's time," and drift from reality) is worse. This chapter is the pipeline that makes push-to-deploy safe enough to sleep through, and the three times it lied to us anyway.

The pipeline, end to end

flowchart TD
    DEV["Developer: git push origin main"] --> GH{"GitHub Actions
concurrency group
(one deploy at a time)"}

    GH --> GATES["Parallel gates"]
    GATES --> T["Backend tests
(unit + integration)"]
    GATES --> S["Security gates
(conformance baseline)"]
    GATES --> SS["Secret scan
(gitleaks)"]

    T & S & SS --> DEPLOY["Deploy job on self-hosted runner
(org pool, Linux labels)"]
    DEPLOY --> SSH["SSH to VPS"]
    SSH --> RESET["git reset --hard origin/main"]
    RESET --> BUILD["docker compose build
(BuildKit secret: GitHub Packages token)"]
    BUILD -->|fail| ABORT["ABORT — never ship stale containers
(a masked failure once did)"]
    BUILD --> FOXUP["fox stack up + health-gate fox-api"]
    FOXUP -->|unhealthy| LOGS["dump logs · fail the run"]
    FOXUP --> TAXUP["tax stack up + health-gate tax-api
(migrations run on startup)"]
    TAXUP --> SPA["SPA rebuild in throwaway container
+ SEO prerender step
+ swap into Caddy's dir"]
    SPA --> SMOKE["Smoke: both /healthz = 200"]
    SMOKE --> INDEXNOW["IndexNow ping
(Bing-fed AI discovery)"]
    INDEXNOW --> DONE["✅ Production live"]

Walk the interesting parts:

The gates run first, in parallel. Unit tests plus Testcontainers-backed integration tests (real Postgres in ephemeral Docker — tenant-isolation tests from Chapter 04 run here). A conformance gate checks repo standards against a committed baseline — new violations block, frozen historical debt doesn't, so the fleet can adopt gates without a Big Bang cleanup. And gitleaks scans for credentials, because everyone eventually pastes one.

A concurrency group serializes deploys. Two pushes minutes apart deploy in order, never interleaved. (Keep this term in mind — it has a war story below.)

The deploy is SSH + shell, nothing exotic. git reset --hard on the host (why manual docker compose up there deploys stale code — the host repo is always mid-flight), then a build that receives the private feed token as a BuildKit secret — mounted during the build, evaporating after; never in a layer or image history.

Health gates, then SPA, then smoke. Each API container must report healthy before the pipeline proceeds (unhealthy = dump container logs and fail loudly). Database migrations apply automatically on container startup — EF Core tracks applied migrations in a history table, so restarts are idempotent. Then the SPA rebuilds on the host (including the SEO prerender step from Chapter 03), swaps into Caddy's directory, and both /healthz endpoints must return 200. The final step even pings IndexNow so discovery engines learn the content changed.

Runners: self-hosted, in a pool. GitHub's included minutes are exhausted at a $0 spending limit, so jobs run on the org's own Linux runners — shared with sibling repos, which means queueing is normal and monitored. (This will matter in war story #3.)

The three times the pipeline lied

Every CI/CD system is a distributed system, and distributed systems lie. Ours did, three ways, each now a permanent lesson:

  1. The masked failure. The build step ended with || echo "WARNING: build failed (using cached image)" and then ran up -d --no-build. The build failed (missing token), the script shrugged, containers kept running yesterday's image — and the pipeline reported success for eight hours. The lesson, now a repo-wide rule: a failed build must abort the deploy. A green check that shipped nothing is worse than red.

  2. The zombie concurrency holder. A cancelled deploy run never released the concurrency group. Every later run was created pending with zero jobs — the job row only appears when the lock is acquired — and sat there forever while the UI implied queuing was normal. Diagnosis came from the jobs API: zero jobs + idle runners = deadlock, not queue. The fix was renaming the group; the lesson is that "pending forever" is a distinct failure mode you must know how to recognize.

  3. The label that matched nothing. Moving the deploy job to the self-hosted pool, the runner labels were written as a quoted string — which YAML/Actions parsed as one giant literal label no runner on Earth advertises. Jobs queued eternally again, this time invisible in a different way. The fix: a real YAML list. The general lesson: when nothing ever picks up your job, print exactly what labels it's demanding — don't trust your eyes reading YAML.

Rollback, and why we rarely say the word

Rollback here is git reset to the previous SHA and redeploy — minutes of work, no image registry needed (everything builds on the host). But database migrations complicate true rollback: a migration that ran won't un-run safely. So the doctrine is forward-fix: the pipeline is fast enough (and gates strong enough) that fixing forward beats reverting schemas. The one hard rule that came from painful experience: never apply schema changes out-of-band — hand-editing the production DB twice caused startup crash-loops when the real migration later collided with the hand-made change. If reality forces your hand, the follow-up migration must be idempotent (ADD COLUMN IF NOT EXISTS) and the history table reconciled.

Why push-to-deploy is a psychological feature

For a solo developer, the deepest value isn't the minutes saved — it's that shipping stays a habit. When deploys are ceremony, you batch changes, batches breed fear, fear breeds bigger batches. When git push is the release process, every fix ships the day it's written (and Chapter 10's feedback loop closes while the user still remembers filing the report).

Recap. Gates before deploy, one deploy at a time, health-gated containers, secrets that evaporate, smoke tests, and a philosophy of forward-fix. The pipeline's job is to make shipping boring — and its own failure modes taught us more than its successes.

Next: Chapter 08 — Unit economics: what it actually costs — the real cost sheet, Stripe's take, and where the break-even lines sit.