I've been exploring how diffusion models, which have transformed image and video generation, can be applied to text generation. The math behind discrete diffusion models can be a bit intimidating at first, so I put together an annotated Jupyter Notebook that walks through the theory and builds a character-level discrete diffusion GPT, adapted from Andrej Karpathy's baby GPT. Instead of generating text autoregressively (left to right), the model learns to denoise corrupted text sequences in parallel.The notebook covers the mathematical framework, noising process for discrete tokens, and a working implementation trained on Shakespeare's text.
GitHub: https://github.com/ash80/diffusion-gpt
Open in Google Colab: https://colab.research.google.com/github/ash80/diffusion-gpt...
Feedback and suggestions are very welcome.