---
title: README
emoji: 💤
colorFrom: indigo
colorTo: blue
sdk: static
pinned: true
---

We build **stuff** and **share it** with the community.

That's pretty much it. And yes, "stuff" was the right word to use.

### Why "**FromZero**"?

**FromZero** is inspired by **Re:Zero - Starting Life in Another World**. Just like the main character, all of our models begin from zero.

Thankfully, they don't spend their time dying repeatedly or having mental breakdowns.

I mean, training is already painful enough. 

### Our goal

Well, there isn't actually a definitive goal we're chasing. I mean, yeah, we want to be the **greatest**. But so does everyone else.

We're just here to build ***weird*** models, share them with the community, and grow side-by-side with everyone.

### Current Models

- **Zero-v0.1-150M**: Our highest-performing and most recent model, boasting a total of 151.6M parameters and a 2048-token context window.
- **Er-Large**: A 31M-parameter model trained on 34B tokens, achieving near state-of-the-art performance across multiple benchmarks.
- **MrPong**: Our SOTA Ping Pong reinforcement learning (RL) agent, trained for over 10M steps against nine different opponents.

### Future Releases

- Zero-v1.0-144M: The next-generation model in the Zero family. It will be trained on 133B tokens and will introduce a new custom architecture featuring Engram Conditional Memory, mHC, and XSA GQA Attention.
- JetonCount-2: The second generation of JetonCount. It will be trained on 25 different datasets and 50 tokenizers, featuring an improved architecture.
- Max-v1.0-3M: A 3M-parameter model that will be trained on 7B tokens. It will use the same architecture as Zero-v1.0, with Hadamard FFNs and SwiGLU intervals.
- Negative-Engram-150M-A67k: An experimental model inspired by Negative, featuring a significantly larger Engram table.

### Partners

If you like us, then you might also like:

[AxiomicLabs](https://huggingface.co/AxiomicLabs) | [BenchLabs](https://huggingface.co/bench-labs) | [QyrouLabs](https://huggingface.co/Qyrou) | [GODELEV](https://huggingface.co/GODELEV)
### Discord

Join our Discord community: [FromZero](https://discord.gg/mGPMfme9s)

### YouTube

Visit our YouTube channel at [https://www.youtube.com/@fromziro-ia](https://www.youtube.com/@fromziro-ia).

### If You Want to Join Us 

We are always looking for more members. If you'd like to join, open a discussion under this README and tell us why. We'll review your request and either approve or reject it.