Ask HN: Do LLMs Read Your /llms.txt?

1 point

6 months ago

In 2024, there was an initiative to standardize the use of an /llms.txt (1) file to provide web resource descriptions that help LLMs use a website. As part of our security analytics (2) development, we're investigating how bots, especially LLMs, interact with web applications.

What we've observed is that this file is never requested by any LLMs. At most, it was requested by bots from BuiltWith and WebPageTest, but not by major LLMs.

Can you check your logs to see if llms.txt is being autonomously read by any popular LLMs?

(1) https://llmstxt.org/

(2) https://github.com/tirrenotechnologies/tirreno