So I built an app that lets kids talk to Santa. The most important part to me was making it feel like a real conversation with low latency.

Powered by ChatGPT, Santa is fully interactive. He'll tell stories, answer questions about the North Pole, and even help parents land any message they need help reinforcing (e.g. Go to bed on time, don't hit your brother). My five year old LOVES talking to Santa, and I'm hoping a few other kids out there get to experience the same magic this holiday season.

This started as a "I bet I could build it" and "I'm super curious about elixir" -- It's an elixir backend that coordinates between the speech to text API (Google), an LLM (ChatGPT) and a text to speech API (Eleven Labs). We used Membrane as the audio processing framework which gave me a bunch of stuff (voice activity events + coordinating the audio pipelines).

The trickiest part was figuring out how to 1) manage the latency across the three APIs but still get Santa to respond in a reasonable timeframe and 2) how to handle an API failures and make sure the conversation recovered.

I'm not an engineer by training, much more of a self taught hobbyist. So much of this was hacked together by me reading docs + asking ChatGPT for advice.