Search: github.com/tokenium | Heykuki News

Heykuki News

Top New Best Ask Show Jobs

Top New Best Ask Show Jobs

31.

Show HN: 1gbps Tokenizer written in Assembly. 20x faster than HuggingFace (github.com/dogmaticdev)

3 points

a month ago

32.

Node.js Open Source LLM Tokenizer (github.com/jakecyr)

2 points

2 years ago

33.

LLM Tokenization Demo (github.com/tokfan)

2 points

10 months ago

34.

The Worst (But Only) Claude 3 Tokenizer (github.com/javirandor)

2 points

2 years ago

35.

Neural Tokenizer (github.com/Kyubyong)

2 points

9 years ago

36.

Show HN: CLI Tokenizer – A tiny tool for prompt engineers (github.com/ericciarla)

1 point

2 years ago

37.

Stripe on Apple watchOS 3 (github.com/appintheair)

4 points

10 years ago

38.

LLM Tokenizer in Zig (github.com/Mario-SO)

1 point

9 months ago

39.

Very simple javascript highlighter that can be used in blog posts (github.com/fatih-erikli)

1 point

2 years ago

40.

Show HN: I'm writing a library to apply NLP techniques to StarCraft 2 (github.com/ZephyrBlu)

1 point

5 years ago

41.

CmusicAi a New Cryptocurrency for Artists

2 points

2 years ago

42.

PRFI Protocol:Decentralized API Tokenization with Oof-of-Work Mining (github.com/sr-oliveiraa)

1 point

10 months ago

43.

Card Network Tokenization: A Savior or Hidden Menace (github.com/juspay)

1 point

3 years ago

44.

Show HN: Tokenkit – Convert LLMs to new tokenizers (incl byte-level Llama/Gemma) (github.com/bminixhofer)

1 point

a year ago

45.

Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken (github.com/M4THYOU)

281 points

a year ago

46.

Tiktoken: OpenAI’s Tokenizer (github.com/openai)

153 points

3 years ago

47.

Code for the Byte Pair Encoding algorithm, commonly used in LLM tokenization (github.com/karpathy)

81 points

2 years ago

48.

55x Speedup of Andrej Karpathy's Minbpe LLM Tokenizer with PyTorch/CUDA (github.com/kuprel)

19 points

2 years ago

49.

Show HN: Open-source card tokenization service in Rust (github.com/juspay)

14 points

3 years ago

50.

XML Tokenizer that's 4x faster than stdlib's XML (github.com/muktihari)

10 points

2 years ago

51.

Show HN: A Command-Line Sentence Tokenizer Written in Golang (github.com/neurosnap)

6 points

11 years ago

52.

From Scratch GPT Built with NumPy (Tokenizer, Model, Adam) (github.com/codiceSpaghetti)

6 points

a year ago

53.

Show HN: Rust BPE tokenizer for Qwen models that's 12x faster than HuggingFace (github.com/sweepai)

5 points

8 months ago

54.

Chiffon: A very small ECMAScript parser, tokenizer in JS (github.com/polygonplanet)

5 points

11 years ago

55.

Show HN: Tokenomics of a Reserve Currency (github.com/Intercoin)

4 points

5 years ago

56.

Jargon: tokenizers and lemmatizers for Go (github.com/clipperhouse)

4 points

8 years ago

57.

Fast JSON parser in Rust that uses SIMD and avoids tokenisation (github.com/pikkr)

4 points

9 years ago

58.

OpenAI's Tokenizer Page for OSS Models (github.com/1rgs)

3 points

3 years ago

59.

VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and Voice Cloning (github.com/OpenBMB)

3 points

6 months ago

60.

SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenization (github.com/facebookresearch)

3 points

8 months ago