Equal model revisions issue
Hi,
I just noticed that when downloading different revisions of Pythia-2.8b, the same weights are downloaded. For example, "step0" and "main" revisions output the exact same generations, altough they should be really different.
oh yeah I hit this too, spent way too long confused about why my step0 model was generating coherent text lol. it's the safetensors file.
every step branch has a model.safetensors that's just main's final weights. same hash on all of them:
ab496f1c... model.safetensors (5,684,693,096 bytes) — step0, step1000, step100000, main, all identical
the actual checkpoint weights are still there, but only in the sharded model-0000* files, which do differ per step like they should:
step0 model-00001-of-00002.safetensors dee700d6...
step1000 model-00001-of-00002.safetensors f6a6c2f8...
step10000 model-00001-of-00002.safetensors 35f6b564...
and transformers grabs the single model.safetensors before it even looks at the sharded index, so revision="stepN" silently hands you the fully-trained model no matter what N you ask for. use_safetensors=False doesn't save you either — the pytorch_model.bin on the step branches is also just a copy of main's (36f0f93d...). step100000 is the one exception, its .bin is actually unique.
pretty sure it's just leftover from the SFconvertbot safetensors conversion back in 2023 (commit 2a259cd) — it converted main's bin into that ab496f1c file, and the step branches ended up carrying it and nobody removed it when the real shards were added later.
check it without downloading anything:
from huggingface_hub import HfApi
api = HfApi()
for rev in ["step0", "step1000", "step100000", "main"]:
for f in api.get_paths_info("EleutherAI/pythia-2.8b",
["model.safetensors", "model-00001-of-00002.safetensors"], revision=rev):
print(rev, f.path, f.lfs.sha256[:12] if f.lfs else f.blob_id)
workaround that actually works, only pull the shards:
from huggingface_hub import snapshot_download
path = snapshot_download("EleutherAI/pythia-2.8b", revision="step1000",
allow_patterns=["config.json", "tokenizer*", "special_tokens*",
"model-0000*", "model.safetensors.index.json"])
model = AutoModelForCausalLM.from_pretrained(path)
quick way to confirm you actually got different weights: next(model.parameters()).abs().sum() should come out different for each step.
real fix would be dropping that stray model.safetensors (and the stale .bin) off the step branches. fwiw the deduped repo doesn't have the extra file so it loads fine there — might be worth checking the other sizes too.