Show HN: Like grep but for natural questions. Mixtral 8x7B – 28 tok/s on 8GB GPU

10 points

2 years ago

Like grep but for natural language questions. Based on Mistral 7B or Mixtral 8x7B.

Example:

fltr --file emails.txt --prompt "Is the following email spam? Email:" --batch-size 32

It will output all lines in the file where the answer is yes.

Text file input tokens per second:

- Nvidia RTX 3070 with 8GB memory: Mistral 7B: ~52 tok/s, Mixtral 8x7B: ~28 tok/s

- Intel I5-6500 with 8GB memory: Mistral 7B: ~5 tok/s, Mixtral 8x7B: ~2 tok/s

Supports: Linux (x86_64) & macOS (x86_64 & arm64)