Ask HN: Why Is Internet Archive Included in Cloudflare's List of AI Crawlers?

2 points

2 years ago

Was just looking into some steps to block AI bots from crawling a site, a feature that Cloudflare promoted recently and noticed on their list of "Verified Bots" for AI Crawlers, one called "Internet Archive" is included. Why?

There is also another one "InternetArchiveBot" listed in a different category as "Monitoring & Analytics". Will blocking the AI one mess with Wayback Machine activities and that kind of 'useful' thing?

The recent announcement post: Block AI bots, scrapers and crawlers with a single click https://news.ycombinator.com/item?id=40865627

https://radar.cloudflare.com/traffic/verified-bots

1 comment