Ask HN: Did OpenAI just retrain GPT from scratch?

4 points

3 years ago

The announcement[1] from a couple of days ago makes note of increased context length from 4k to 16k for GPT-3.5-turbo and to 32k (from 8k?) for GPT-4. From my limited understanding of how LLMs work / are trained, context window is a key part of training.

Does quadrupling context means they retrained the whole thing(s) from scratch, or there's some other wizardry that can be applied to other models to increase their context length without retraining from scratch?

[1] https://openai.com/blog/function-calling-and-other-api-updates

3 comments