AI LLMs can't count lines in a file

20 points

a year ago

Was starting to mess around with the latest LLM models and found that they're not great at counting lines in files.

I gave Gemini 2.5 flash a python script and asked it to tell me what was at line 27 and it consistently got it wrong. I tried repeatedly to prompt it the right way, but had no luck.

https://g.co/gemini/share/0276a6c7ef20

Is this something that LLM bots are still not good at? I thought they had gotten past the "strawberry" counting problems.

Here's the raw file: https://pastebin.com/FBxhZi6G

34 comments