---
base_model: google/t5gemma-2-270m-270m
license: gemma
tags:
  - gemma2
  - encoder-only
  - text-encoder
  - embeddings
  - bidirectional
---

# T5Gemma-2-270m — Text Encoder Only (Bidirectional)

Text encoder extracted from [google/t5gemma-2-270m-270m](https://huggingface.co/google/t5gemma-2-270m-270m), 
saved as standard `Gemma2Model` with **bidirectional attention** (`is_decoder=False`).

> Gemma is provided under and subject to the Gemma Terms of Use found at https://ai.google.dev/gemma/terms

## Architecture
- 18 layers, hidden_size=640, heads=4
- Sliding window attention (512) + full attention every 6 layers
- Bidirectional (no causal mask)
- Parameters: 268M

## Usage
```python
from transformers import AutoModel, AutoTokenizer

model     = AutoModel.from_pretrained("knowledgator/t5gemma-2-text-encoder-270m")
tokenizer = AutoTokenizer.from_pretrained("knowledgator/t5gemma-2-text-encoder-270m")

inputs  = tokenizer("Your text here", return_tensors="pt", padding=True, truncation=True)
outputs = model(**inputs)

token_embeddings = outputs.last_hidden_state          # (batch, seq_len, 640)
pooled           = outputs.last_hidden_state.mean(1)  # mean pooling -> (batch, 640)
```