Chapter 11 — Reach: SEO, AI crawlers, and being the machine-readable answer

Somewhere between "launched" and "learned people exist," every product discovers distribution. Ours arrived with a twist that defines this chapter: half your future traffic now arrives via machines that never render your app — search crawlers that mostly cope, and AI answer engines (GPTBot, ClaudeBot, PerplexityBot) that flat-out don't run JavaScript. If your site is a client-side SPA, those engines see a blank page with a title. This chapter is everything we built so the crawlable half of the internet can actually read the product — and how we measure it.

The problem, precisely

The FoxyInvoice app is an Angular SPA: the server ships an empty shell and JavaScript builds the page. Human browsers: fine. Googlebot: tolerant, eventually. AI crawlers: blind. Fetch the marketing page with curl — which is exactly what a crawler does — and you got:

<title>Invoicing</title>
<app-root></app-root>

One word and an empty div. To every machine reader, our entire free invoice-template business was the word "Invoicing."

The fix stack, bottom to top

1. The allowlist robots philosophy

Our robots.txt doesn't list what's forbidden — it lists what's allowed, then disallows everything else:

User-agent: *
Allow: /$ /pricing /privacy /terms /templates /templates/*
Disallow: /

Every future route — admin consoles, /upgrade, QA harnesses — is non-indexable by default until deliberately made public. The same policy is enforced server-side with X-Robots-Tag: noindex, nofollow headers at the edge proxy, per URL: private pages carry it; public pages don't. Belt and suspenders, because robots.txt disallow alone never guarantees de-indexing.

2. Prerendering at build time (the pragmatic middle)

Full server-side rendering means an always-on SSR server — violating our boringness constraint — for pages that are 95% app shell. Instead, a post-build script generates static, crawler-ready copies of every public route when the SPA compiles:

• Real <title> and meta description per page ("Free Plumbing Invoice Template — FoxyInvoice")
• Canonical URL, Open Graph tags, JSON-LD structured data
• Full article body copy inside the shell — line items, how-to steps, FAQ — because meta tags describe content; answer engines quote content
• The build fails if the sitemap and the template data drift apart (a one-line count check that caught a real bug on its first run)

The edge proxy's try_files serves these static files to anything that fetches the URL; real browsers still get the app (Angular replaces the static body on boot). No new server, no SSR framework, crawler-complete pages.

3. llms.txt — a menu for answer engines

The emerging convention: a markdown file at the root telling AI systems what the product is and linking its key pages, in their vocabulary. Ours lists the product, every template page with a one-line description, and contact points. Cheap, human-readable, and exactly the artifact a "recommend an invoicing tool" query wants.

4. IndexNow: push instead of wait

Crawlers traditionally rediscover content on their own schedule. IndexNow flips it: on every deploy, the pipeline POSTs all sitemap URLs to the alliance endpoint (Bing-powered — which feeds several AI answer engines) with a key file proving domain ownership. Our first ping returned HTTP 202 — accepted — and the pages were in Bing's queue the same evening.

5. Measurement: access logs are the truth

"You can't improve what you can't see," so the edge now logs every request as JSON (rotated, bounded). The analysis is one script: user-agents, paths, crawler classes. Within days we could see search crawlers reading robots.txt and template pages — and, just as valuable, probe-noise (stray /wp-admin scans) being correctly absorbed. Pair this with Google Search Console + Bing Webmaster registration (a five-minute runbook lives in the repo) for the indexing-and-impressions view logs can't give you.

The funnel the surface feeds

Reach is only worth building if it lands somewhere. The template gallery — now crawlable — is the top of a three-slice funnel we shipped in order:

1. Slice 1 — no-signup value + persistence: the generator works anonymously, autosaves locally (30 days), and offers resume on return.
2. Slice 2 — the signup handoff: "Save & finish online" carries the exact filled invoice through signup into the new account as a real draft.
3. Slice 3 — the soft capture: "Email me this invoice" stores the draft server-side against their address (with a nurture email); if they later sign up with that address, the dashboard converts it into a real invoice automatically.

Each slice was verified in a real browser before shipping — the funnel is the one place where "should work" is banned.

Recap. Serve machines real HTML (prerendered at build), allowlist what's public (default-closed), publish llms.txt, push updates via IndexNow, and measure with access logs + Search Console. The era of "GEO" — being the machine-readable answer — is just SEO where the reader never renders your JavaScript.

Next: Chapter 12 — Engagement & automation: UI first, then automate.