https://lab.frenemy.dev/ 2026-06-10T00:00:00.000Z https://lab.frenemy.dev/articles/a-short-history-of-robots-txt 2026-05-02T00:00:00.000Z https://lab.frenemy.dev/articles/how-googlebot-proves-its-identity 2026-05-04T00:00:00.000Z https://lab.frenemy.dev/articles/the-ai-crawler-roster 2026-05-06T00:00:00.000Z https://lab.frenemy.dev/articles/what-is-web-bot-auth 2026-05-08T00:00:00.000Z https://lab.frenemy.dev/articles/user-agent-strings-a-field-guide 2026-05-10T00:00:00.000Z https://lab.frenemy.dev/articles/llms-txt-explained 2026-05-12T00:00:00.000Z https://lab.frenemy.dev/articles/reverse-dns-verification 2026-05-14T00:00:00.000Z https://lab.frenemy.dev/articles/ip-ranges-asns-and-who-owns-an-address 2026-05-16T00:00:00.000Z https://lab.frenemy.dev/articles/the-crawl-economics-of-a-small-site 2026-05-18T00:00:00.000Z https://lab.frenemy.dev/articles/sitemaps-and-why-crawlers-love-them 2026-05-20T00:00:00.000Z https://lab.frenemy.dev/articles/spotting-an-impersonator 2026-05-22T00:00:00.000Z https://lab.frenemy.dev/articles/what-a-crawler-sees-in-a-day 2026-05-26T00:00:00.000Z