Sunny111/LLM-Inheritune
Updated
Sunny Sanyal
Sunny111
AI & ML interests
Efficient Training Recipes of Large Models (mostly LLMs)
Recent Activity
replied to
their
post
about 3 hours ago
Are you familiar with reverse residual connections or looping in language models?
Excited to share my Looped-GPT blog post and codebase 🚀
https://github.com/sanyalsunny111/Looped-GPT
TL;DR: looping during pre-training improves generalization.
Plot shows GPT2 LMs pre-trained with 15.73B OWT tokens
P.S. This is my first post here — I have ~4 followers and zero expectations for reach 😄
posted
an
update
2 days ago
Are you familiar with reverse residual connections or looping in language models?
Excited to share my Looped-GPT blog post and codebase 🚀
https://github.com/sanyalsunny111/Looped-GPT
TL;DR: looping during pre-training improves generalization.
Plot shows GPT2 LMs pre-trained with 15.73B OWT tokens
P.S. This is my first post here — I have ~4 followers and zero expectations for reach 😄
upvoted
a
paper
about 1 month ago
Pre-training Small Base LMs with Fewer Tokens
Organizations
