Mergoo is a library that allows you to easily integrate multiple open-source LLMs from Hugging Face and efficiently train a mixture of experts without starting from scratch.
https://github.com/Leeroo-AI/mergoo
6 comments
A Library to build MoE from HF models | Heykuki News