I have had a blast working on this so far and I'm excited to continue to build with it. It uses whisper.cpp [1], Coqui TTS [2] and OpenAI [3] to do speech-to-text, text-to-text and text-to-speech inference all 100% locally (except for text-to-text). In the future I plan to swap out OpenAI for llama.cpp [4]. It is built on top of WebRTC as the media transmission layer which will allow this technology to be deployed anywhere as it does not rely on any native or 3rd party APIs.

The purpose of this project is to be a toolbox for vocal computing. It provides high-level abstractions for dealing with speech-to-text, text-to-text and text-to-speech tasks. The tools remain decoupled from underlying AI models allowing for quick and easy upgrades when new technology is realeased. The main demo for this project is a J.A.R.V.I.S-like assistant however this is meant to be used for a wide variety of use cases.

In the coming months I plan to continue to build (hopefully with some of you) on top of this project in order to refine the abstraction level and better understand the kinds of tools required. I hope to build a community of like-minded individuals who want to see J.A.R.V.I.S finally come to life! If you are interested in vocal computing come join the Discord server and build with us! Hope to see you there :)

Video demo: https://youtu.be/xqEQSw2Wq54

[1] whisper.cpp: https://github.com/ggerganov/whisper.cpp

[2] Coqui TTS: https://github.com/coqui-ai/TTS

[3] OpenAI: https://openai.com/

[4] llama.cpp: https://github.com/ggerganov/llama.cpp