Site Scan: How Multi-Page Crawling Works
Direct Answer
A multi-page scan follows discovered URLs from allowed entry points, records each response, and builds a site-level view from many page-level observations rather than one URL in isolation.
Documentation
What this topic means
This documentation focuses on Site Scan. The practical context is How Multi-Page Crawling Works. It defines the decision boundary and the preferred technical behavior. Adjacent failure states and broader workflows are covered by the related documents.
In practice, evaluate Site Scan on the live public response and compare at least one affected URL with a known-good example. Determine whether the behavior is isolated or generated by a shared route, template, component, proxy, or deployment rule.
Why it matters
A repeatable audit workflow prevents teams from fixing visible symptoms while missing a more fundamental crawl, template, or release problem. Evidence and order also make later verification possible.
For Site Scan, the important distinction is between intended configuration and the state a crawler or user can actually fetch. Diagnose that gap before choosing a fix.
What to check
- Confirm the live behavior for Site Scan on a representative public URL and record the exact response or rendered evidence.
- Compare the affected case with a known-good URL so the difference is specific to Site Scan rather than an unrelated sitewide failure.
- Use a representative URL set that includes important templates, not only the homepage.
- Record the observed value or response together with the URL and timestamp.
Implementation guidance
Apply the workflow to a small representative set first, confirm that the evidence is trustworthy, then scale the same checks across templates. Keep findings tied to reproducible URLs and avoid converting a score or issue count into an unsupported ranking claim.
For Site Scan, make the intended behavior explicit at the layer that owns it and update generated internal references so new URLs do not recreate the old condition. If the behavior repeats across a template or route family, fix the shared source rather than patching individual pages.
Verify the result
Repeat the same collection after the change and compare the evidence rather than only the issue count. The expected finding should disappear while unrelated statuses, directives, links, and rendered content remain stable.
The result is acceptable when the live URL consistently follows the intended site scan behavior, the related evidence is stable on repeated requests, and adjacent URLs or templates have not regressed.
Questions & Answers
What should be true after Site Scan is fixed or implemented?
The result is acceptable when the live URL consistently follows the intended site scan behavior, the related evidence is stable on repeated requests, and adjacent URLs or templates have not regressed.
Does this problem always cause a direct ranking loss?
No. Technical findings affect crawlability, rendering, consolidation, usability, or maintainability in different ways. Treat the finding as evidence to diagnose, not as a guaranteed ranking penalty.
Should the fix be made per URL or at a shared layer?
Use the narrowest layer that owns the behavior. If site scan repeats across URLs from the same template, route, proxy, or component, a shared fix is usually safer than many page-specific exceptions.
How should the change be tested?
Repeat the same collection after the change and compare the evidence rather than only the issue count. The expected finding should disappear while unrelated statuses, directives, links, and rendered content remain stable.
Practical Check
- Confirm the live behavior for Site Scan on a representative public URL and record the exact response or rendered evidence.
- Compare the affected case with a known-good URL so the difference is specific to Site Scan rather than an unrelated sitewide failure.
- Use a representative URL set that includes important templates, not only the homepage.
- The result is acceptable when the live URL consistently follows the intended site scan behavior, the related evidence is stable on repeated requests, and adjacent URLs or templates have not regressed.