Meanwhile, Mistral announced that their Le Chat platform can output tokens at 1,100 per second! That sounds pretty fast? How fast? I don’t know!
So, that’s why I put together TokenFlow. It’s a (very!) simple webpage that lets you see the (theoretical) speed of different LLMs in action. You can select from a few preset models / services or enter a custom speed in tokens per second. You can then watch it spit out tokens in real time, showing you exactly how fast a given inference speed is and how it impacts user experience.
Check it out: https://dave.ly/tokenflow/