Show HN: FretBench – I tested 14 LLMs on reading guitar tabs. Most failed

fretbench.tymo.ai

1 point

3 months ago

I built FretBench after noticing Gemini was confidently wrong about basic guitar tab questions. Tab is arguably the simplest notation in music: six lines, numbers for frets, read left to right. So I made a benchmark out of it.

182 test cases, 4 tunings, 14 models via OpenRouter. Two open-weight Qwen models from Alibaba crushed everything else (83.5%), while most "flagship" models scored below 50%. MiniMax M2.5 scored worse than random guessing.

Everything is open source: https://github.com/jmcapra/FretBench

I'm curious whether the performance gap is related to tokenisation of ASCII art — if anyone has insights on how different tokenisers handle grid-structured text, I'd love to hear it.