We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.
Links:
- build-nanogpt GitHub repo, with all the changes in this video as individual commits:
- nanoGPT repo:
- llm.c repo:
- my website:
- my twitter:
- our Discord channel:
Supplementary links:
2 views
1074
337
3 months ago 04:01:26 9
Let’s reproduce GPT-2 (124M)
3 months ago 00:26:33 1
Let’s reproduce the calculations from Interstellar
3 months ago 00:44:32 1
[4K]🇺🇸NYC Summer Walk🗽Greenwich Village in New York City🚕🥯Leon’s Bagel & Coffee | Jun 2024
3 months ago 00:44:42 1
[4K]🇺🇸NYC Summer Walk🗽Upper West Side in New York City 😎🔥Hot Saturday in Manhattan | June 2024
3 months ago 02:12:31 1
Annunaki: The Movie | Episode 2 | Lost Book Of Enki - Tablet 6-9 | Astral Legends
3 months ago 01:28:46 1
The Lost Years Of Jesus | Mysterious Saint Isa In India | By Richard Bock | 1960s documentary