The crawl economics of a small site
2026-05-18
A page view by a person and a page view by a crawler cost the same to serve and earn very different amounts. The person might see an ad, subscribe, or buy something. The crawler consumes the bandwidth and leaves.
For a small site the line item is real: a training crawler re-reading a few thousand pages every month is gigabytes of egress, and metered hosting bills it the same as any reader. The site owner is funding the crawl out of pocket.
The interesting number is the exchange rate — pages crawled per visitor sent back. Search historically paid that toll in referrals. Whether AI platforms pay anything comparable is the question every content site is now in a position to measure.