Instead, what if we fed audio directly to the LLM - LLM's are really smart, can they figure it out? This approach is faster (we skip transcription decoding) and less lossy/more robust because the big language model should be smarter than a smaller transcription decoder.

I've trained a model in just that fashion. For more architectural information and some training details, see this first post: https://tincans.ai/slm . For details about this model and some ideas for how to prompt it, see this post: https://tincans.ai/slm3 . We trained this on a very limited budget but the model is able to do some things that even GPT-4, Gemini, and Claude cannot, eg reasoning about long-context audio directly, without transcription. We also believe that this is the first model in the world to conduct adversarial attacks and apply preference modeling in the speech domain.

The demo is unoptimized (unquantized bf16 weights, default Huggingface inference, serverless speed bumps) but achieves 120ms time to first token with audio. You can basically think of it as Mistral 7B, so it'll be very fast and can also run basically anywhere. I am especially optimistic about embedded usage -- not needing the transcription step means that the resulting model is smaller and cheaper to use on the edge.

Would love to hear your thoughts and how you would use it! Weights are Apache-2 and on Hugging Face: https://huggingface.co/collections/tincans-ai/gazelle-v02-65...