Previously, Tabby ran exclusively on CUDA devices, posing a significant barrier for developers looking to effectively utilize LLMs in their day-to-day coding.

The Tabby team made a significant contribution by enhancing support for the StarCoder series models (1B/3B/7B) in llama.cpp. This enhancement allows these models to run on Metal, providing comparable performance to that of an NVIDIA GPU.

I eagerly look forward to receiving feedback from the community and witnessing the enhanced on-edge deployment experience!

References:

[1] Apple Installation Instructions: https://tabby.tabbyml.com/docs/installation/apple/

[2] Launch blog: https://tabby.tabbyml.com/blog/2023/09/18/release-0-1-1-meta...