gurumurthy3/gpt2vl-stackformer-v2

GPT2VL — Stackformer V3 Image Captioning

A vision-language image captioning model built from scratch on top of a custom Stackformer transformer library: a GPT-2 language backbone fused with a frozen CLIP ViT-B/16 vision tower through a Flamingo-style Perceiver Resampler and tanh-gated sparse cross-attention.

Only the adapter layers (resampler + cross-attention blocks) are trained — the GPT-2 backbone and CLIP vision tower stay frozen, keeping the trainable footprint small while reusing two strong pretrained models.

Try the live demo →  ·  Model weights on the Hub

---

How it works

Image → CLIP ViT-B/16 (frozen) → patch tokens (CLS + 196 patches, 768-dim)
                                            │
                                            ▼
                              Perceiver Resampler (3 layers, 8 heads)
                                            │
                                            ▼
                              64 compressed visual tokens (768-dim)
                                            │
              ┌─────────────────────────────┼─────────────────────────────┐
              ▼                             ▼                             ▼
     GPT-2 layer 3 ──tanh-gated──   GPT-2 layer 7 ──tanh-gated──   GPT-2 layer 11 ──tanh-gated──
        cross-attn                     cross-attn                     cross-attn
              │                             │                             │
              └──────────────► GPT-2 self-attention stack (12 layers, frozen) ◄──────────────┘
                                            │
                                            ▼
                                    Generated caption

• Vision encoder — openai/clip-vit-base-patch16, frozen. The full patch-token sequence (CLS + 196 patches) is passed through, not just the pooled embedding, so the resampler has access to full spatial detail.
• Perceiver Resampler — compresses the CLIP token sequence down to 64 fixed latent queries via iterative cross-attention (depth 3, 8 heads, 4× FFN expansion).
• Gated cross-attention — spliced into GPT-2 at layers [3, 7, 11]. Each gate is tanh(alpha), zero-initialized so training starts as a pure, unmodified GPT-2 language model and gradually learns how much to attend to the image.
• Text backbone — standard GPT-2 (small), initialized from the pretrained gpt2 weights and kept frozen during training.

Only resampler.* and cross_blocks.* are trainable — everything else is frozen, pretrained weight.

Model configuration

Text backbone	GPT-2 small (12 layers, 12 heads, 768-dim, 128 token context)
Vision encoder	CLIP ViT-B/16 (openai/clip-vit-base-patch16), frozen
Visual tokens	64 (Perceiver Resampler, depth 3, 8 heads)
Cross-attention positions	GPT-2 layers 3, 7, 11
Cross-attention gate	tanh(alpha), zero-init
Precision	float32 weights, torch.amp.autocast (fp16 matmuls) on GPU
Training data	AKCIT/coco2017-captioning (5 references/image)

Using the Space

1. Upload an image (or pick a quick example).
2. Click Generate Caption.
3. Greedy Decoding is on by default — deterministic, highest-confidence output, matching how the model was evaluated during training. Turn it off in Advanced Settings to sample instead, then tune:
   • Temperature — 0 = greedy, higher = more random
   • Top-K — restricts sampling to the K most likely tokens
   • Top-P — nucleus sampling; restricts to the smallest token set covering P cumulative probability
   • Max New Tokens — generation length cap

Running locally

pip install -r requirements.txt
python app.py

The app downloads config.json and model_trainable.safetensors from the Hub repo set by the MODEL_REPO_ID environment variable (defaults to gurumurthy3/gpt2vl-stackformer-v3) on startup, then reconstructs the frozen GPT-2 and CLIP weights locally before loading the trained adapter tensors on top.

Repository layout

app.py                        # Gradio Space — model definition + UI
config.json                   # architecture hyperparameters (on the Hub repo)
model_trainable.safetensors   # trained adapter weights (resampler.* + cross_blocks.*)
samples/                      # example images for the Quick Examples bar

Acknowledgements

• Text backbone: GPT-2 (OpenAI) via 🤗 Transformers
• Vision encoder: CLIP ViT-B/16 (OpenAI)
• Resampler design: Flamingo (DeepMind)
• Gated cross-attention: Flamingo / Llama 3.2-Vision-style zero-init gating
• Transformer primitives: custom Stackformer library