Like (perhaps?) many of you working with different open-weight LLMs, I often find myself wanting a quick way to check token counts for specific Hugging Face models without having to write a Python script each time. I often find myself googling something like "online Qwen tokenizer" and coming up empty-handed.

So, I built Tokiwi (https://tokiwi.dev), the online tokenizer I've been looking for.

What it does: - You paste your text. - You enter any model repository ID from Hugging Face (like `google/gemma-3-27b-it`, `deepseek-ai/DeepSeek-V3-0324`, etc.). - It fetches the tokenizer from HF and shows you the token count and the list of tokens. - It also supports gated models if you provide your HF token (handled securely via the backend, not stored).

It's pretty straightforward right now. Built with Next.js on Vercel for the frontend and a FastAPI backend running on Modal (https://modal.com/). Under the hood I'm using the `transformers` library.

I'm thinking about adding support for proprietary models (Claude, Gemini) or even a simple API if there's interest.

I'm also thinking about offering prompt compression via LLMLingua (https://github.com/microsoft/LLMLingua).

Would love to get your feedback. Is this useful for anyone else? What's missing?

Thanks for your time!