Open-Source LaMDA Model

27 points

4 years ago

An open-source implementation for the pre-training architecture of Google's LaMDA in PyTorch. The research paper outlines an autoregressive, decoder-only, GPT-like transformer language model. The transformer uses T5 relative positional bias in the attention layers and gated-GELU activation function in the feed-forward layers.

The repository currently contains a script for basic training as well as Huggingface datasets and Weights & Biases integration.

LaMDA research paper: https://arxiv.org/abs/2201.08239

Github repository for the model: https://github.com/conceptofmind/LaMDA-pytorch

The pre-training architecture was peer-reviewed by Dr. Phil Wang. Please check out and support his work: https://github.com/lucidrains.

Updates: https://twitter.com/EnricoShippole