Why a completed crawl is not always a usable crawl.
Lifecycle completion tells you that processing stopped successfully. It does not automatically prove that the resulting page evidence is healthy enough for later decisions.
A crawl can reach its terminal completed state while every fetched page failed, the website returned no usable documents or the evidence is structurally incomplete. Treating that state as a healthy result creates a subtle but serious error: downstream systems start making comparisons from absence.
Job lifecycle and result usability answer two different questions. A trustworthy platform must evaluate both.
The downstream cost of an empty result
Consider sitemap coverage. If a checker compares hundreds of sitemap URLs against a crawl with zero usable pages, it can label every sitemap entry as extra. The algorithm may be internally consistent, but the premise is wrong. There was no representative crawl to compare against.
The safer behavior is to keep standalone sitemap validation valid while marking the crawl comparison as unavailable. This preserves useful evidence without manufacturing a coverage problem.
A practical usability rule
A representative crawl should have a successful terminal lifecycle and at least one usable successful page. Failed-only, zero-page and structurally incomplete output should not become the current result used by downstream comparisons.
Historical records still matter
Older and legacy records can remain readable by their detail identifier. Readability is not the same as endorsement. A legacy-unknown result should not displace a newer or older result whose usable evidence can be proven.
What the interface should say
User-facing summaries should be honest without exposing raw worker errors, stack traces, private paths or provider payloads. The useful contract is simple: the crawl completed, but it did not produce usable page evidence, so comparison-dependent work was skipped.
The larger principle
AI-assisted SEO systems depend on the quality of their evidence boundaries. A strong model cannot rescue an invalid premise. Building trustworthy recommendations starts by representing unavailable, partial and unusable data as clearly as successful data.
Explore the Bostic.ai crawler or see how sitemap intelligence uses representative evidence.