Tell HN: A Facebook crawler was making 7M requests per day to my stupid website

4 points

6 years ago

I own a little website I use for some SEO experiments. Of course there's some content and a facebook sharing button for every post. The website is so little it runs on a "single controller" PHP app + a 400kb SQLite db, but can generate thousands of different pages.

Everything is hosted (together with a bunch of other websites) on a cheap DigitalOcean machine + free cloudflare plan for some caching. One of those websites as some alerting and it started to alert me about being down.

After some investigations I've found out the problem... the Facebook Crawler.

https://developers.facebook.com/docs/sharing/webmasters/crawler/

That crawler was making more than 7M requests per day (with a peak of 300req/second) to that website.

Their doc was not helping on how to block the bot.

og:ttl -> ignored robots.txt -> ignored HTTP 429 -> ignored

I had to block the user-agent using cloud-flare rules.

If there's someone working on that crawler here on HN, please stop ignoring basic Internet netiquette about crawlers. Next time you could hit someone on AWS. And then they'll probably ask you to pay the bill ;)

1 comment