Article
DAVIDAU DELETED MY POST AND BANNED ME FOR ASKING ABOUT BENCHMARKS — HERE IS THE FULL ANALYSIS (9B + 27B + 390 MODELS)
DedeProGames
• • 6
🤗 Life is AImazing
V S
vovaRL
AI & ML interests
RL, Generative AI
Recent Activity
upvoted an article about 10 hours ago
DAVIDAU DELETED MY POST AND BANNED ME FOR ASKING ABOUT BENCHMARKS — HERE IS THE FULL ANALYSIS (9B + 27B + 390 MODELS) reacted to Banaxi-Tech's post with ❤️ about 17 hours ago
We're excited to release BananaMind 2 SLMoE, an experimental sequence-level mixture-of-experts model.
It uses only 8M parameters per message but has 25M total parameters, 13 experts (out of 64) are selected based on the message prefix and reused for the entire response.
We're testing with this sequence-level architecture to find out how big the capability loss actually is and how much of it can be fixed.
The long-term idea is that this could make very large sparse models usable on machines that can't fit them in RAM by putting the entire model (which is big) on disk and only loading the active parts into VRAM.
This architecture is still in research and shouldn't be used for production models.
We trained it on 60B tokens (of FineWeb-HQ, FineWeb-Edu, DCLM ,Cosmopedia v2, FineMath and NPSet-2) on 8 RTX Pro 6000s.
Check it out at https://huggingface.co/BananaMind/BananaMind-2-SLMoE
Follow us for future models:
https://huggingface.co/BananaMind
@vovaRL
@Banaxi-Tech
@DedeProGames
BananaMind 2 Pro in a few days. You've been waiting 22 days for it.Organizations
upvoted an article about 10 hours ago
upvoted an article 5 days ago
Article
Rebuilding Among AIs from Six Log Files
DedeProGames
• • 6 upvoted an article 14 days ago
Article
Training a 2.7B MoE from scratch for $200, one GPU at a time
vovaRL
• • 6