What a crawler sees in a day on a brand-new site
2026-05-26
Publish a site on a fresh domain, tell no one, and watch the logs. The first visitors arrive within hours, and none of them are people: certificate-transparency watchers, sitemap fetchers, security scanners probing for yesterday's vulnerabilities.
Submit the sitemap to a search engine and the second wave begins — the indexers, then the SEO tools that follow them, then the AI crawlers that follow everything. A site nobody has heard of accumulates a surprisingly complete census of the machine web.
That census is this Lab's entire purpose. The site you are reading exists to be read by machines and to take attendance while they do it.