Ask HN: So dunno how to reach Google but I think their bot is broken

18 points

8 years ago

So I was doing some webdev and playing around with my projects. Got a 404 and decided to check my Nginx logs and this stared back at me, just scrolling by

SNIP

  66.249.64.76 - - [04/Feb/2018:16:09:38 +0200] "GET /manual-motor-scania-dc12-53a-pdf-download.pdf HTTP/1.1" 404 169 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
  66.249.64.76 - - [04/Feb/2018:16:09:38 +0200] "GET /15-incredible-uses-for-leftover-cake-the-craftsy-blog.pdf HTTP/1.1" 404 169 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
  66.249.64.76 - - [04/Feb/2018:16:09:38 +0200] "GET /tab-top-curtains-drapes-youll-love-wayfair.pdf HTTP/1.1" 404 169 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
  66.249.64.76 - - [04/Feb/2018:16:09:38 +0200] "GET /en-dwg-pdf.pdf HTTP/1.1" 404 169 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
  66.249.64.74 - - [04/Feb/2018:16:09:38 +0200] "GET /the-fda-top-ten-pdf.pdf HTTP/1.1" 404 169 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
  66.249.64.74 - - [04/Feb/2018:16:09:38 +0200] "GET /philips-fw-c785-user-manual-pdf-download.pdf HTTP/1.1" 404 169 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

SNAP

I have no idea how long this has been going on but I have already over 40MB of this in just one log file starting at 06 in the morning. The IPs do match Google's, too. I mean, I did ask my site to be indexed but these don't even seem like something you'd index but something you'd use as search terms!

Did someone break a config? Am I just a n00b? Am I under an attack from a l33t h4x0r? What the heck is going on?

Find out in the next epis^Wfew hours of: Ask HN!

24 comments