Pre-Launch Crawl: Building a Technical Baseline

What this check actually means

Build a field-level baseline, not only an issue count. For each representative URL capture status, final URL, title, description, canonical, robots directives, H1, hreflang, structured data presence, inlinks, indexability, and a content fingerprint so post-launch differences are explainable.

Expected behavior and decision boundary

Expected state: Build a field-level baseline, not only an issue count. For each representative URL capture status, final URL, title, description, canonical, robots directives, H1, hreflang, structured data presence, inlinks, indexability, and a content fingerprint so post-launch differences are explainable. Define an expected status, expected destination, and owner for every URL/host class before launch. PASS means observed HTTP/DNS/DOM behavior matches that definition without unexplained intermediate states.

Diagnosis: evidence to capture

Record the test as data so release and SEO teams see the same state:

url,expected_status,actual_status,expected_target,actual_target,canonical,robots,owner
https://old.example/a,301,301,https://new.example/a,https://new.example/a,https://new.example/a,index,platform

For host or infrastructure changes, also store DNS answers, TLS/SNI result, CDN cache status, and timestamps. A screenshot alone cannot prove an HTTP status, redirect chain, or resolver state.

Technical implementation

Build a field-level baseline, not only an issue count. For each representative URL capture status, final URL, title, description, canonical, robots directives, H1, hreflang, structured data presence, inlinks, indexability, and a content fingerprint so post-launch differences are explainable.

curl -I https://old.example/path
curl -IL https://old.example/path
curl -I https://new.example/path

Capture at least one representative URL per template/path class and store expected versus observed HTTP status. For DNS/CDN work also record resolver answers, certificate/SNI, host routing, and cache status. For redirects, test the first hop separately from the final destination.

Common root causes

Common causes are incomplete URL inventories, changing domain/CMS/design at the same time, untested edge rules, old absolute URLs embedded in templates, staging/production configuration drift, and no explicit rollback boundary. When something fails, first locate the layer: DNS/TLS, web server/CDN, application routing, CMS/template, or page-level metadata.

Edge cases and false positives

Not every difference is a defect: an intentional 404/410 can be correct, a temporary 302 may be right for testing, and DNS answers can differ while TTLs expire. Define expected behavior per URL class before auto-fixing deviations. For login/API/non-GET endpoints, review redirect semantics separately.

Verification after the fix

| Check | PASS | FAIL | | --- | --- | --- | | Output | expected rule is visible in final HTML/HTTP/DOM | backend field looks correct but final output does not | | Scope | representative URLs from every affected template tested | only one example URL tested | | Source | shared root cause/owner is documented | manual per-page edits without root cause | | Regression | adjacent canonical/robots/accessibility/link rules still pass | fix creates a new defect in the same component | | Evidence | before/after values and test time are stored | only visual impression or screenshot |

Ownership and handoff

Content/SEO owner: defines the expected page purpose and whether the condition actually requires a change.
Developer/platform: fixes the shared template, routing, header, or component source.
QA: tests representative edge states and re-crawls/re-requests after deployment.
Release owner: records time, scope, and rollback point so monitoring signals can be tied to a specific change.

Acceptance criteria

PASS when the final technical state is reproducible on representative URLs, the shared root cause is fixed, intentional exceptions are documented, and a re-crawl/re-request shows no new regression in canonicalization, robots directives, accessibility, links, or rendering.

Worked operational example: defect → fix → proof

Symptom: the audit reports “Pre-Launch Crawl: Building a Technical Baseline” on multiple URLs.
Root cause: the repeated output originates in a template, component, CMS fallback, or infrastructure layer.
Fix: change that shared source instead of patching individual pages, and keep the change as small as possible.
Proof: store at least one before/after URL per template, the final HTTP/HTML/DOM output, and the re-crawl result. If an intentional exception remains different, document it so a future audit does not treat it as an unresolved defect.

For production systems, also include a regression sample: same locale, mobile/desktop when relevant, missing-data state, and one URL that was already correct. This proves whether the change stabilizes the generator rather than only the reported example.

1. Define the affected URL/template class and write the expected behavior before changing anything.
2. Capture raw HTTP/HTML; when client-side code changes output, also inspect the rendered DOM.
3. Validate the redirect/status matrix against final destinations; do not start with a follow-redirect-only test.
4. Trace the defect to its shared layer: DNS/CDN, routing, template, component, CMS field, or editorial content.
5. Implement the smallest systemic fix and preserve a rollback point first.
6. Test positive cases, missing/empty data, and at least one relevant edge state.
7. Re-crawl or re-request every affected template/URL class and store before/after evidence.
8. PASS: expected behavior is reproducible, no adjacent SEO/accessibility rule regresses, and the fix does not require manual per-page workarounds.