Ask HN: Bingbot sending random search queries to your site?

2 points

a year ago

In my access logs I noticed bingbot sending a lot of random search queries to `/search=<query>`. Keywords are in various languages, include news, product names, celebrities, conversion rates and even xxx keywords. This has been going on for months, but really kicked off since January with over 100k requests per month.

The IP addresses from the webbots look genuine, all within the msnbot-*.search.msn.com range. The user agent is "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36", so this doesn't look like LLM scraping.

As far as I can tell there are no links to these pages, and most queries won't return anything as my website is rather niche (CAIBotList). Has anyone else noticed their website being used by Bing as a proxy search engine?

1 comment