Implements the GPT-4 & GPT-3.5 tokenizer in Wasm. OpenAI has one online for GPT-3, but that uses a different token set. I kept using tiktoken in a REPL but this is more ergonomic. Helps optimize cost & response latency (which is ~linear w/ output token count). Also added some of the more counter-intuitive / interesting findings at the bottom!
Show HN: A free online token counter to optimize GPT-4 cost and speed | Heykuki News