File size: 1,336 Bytes
c15c16c 31d36d6 c15c16c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | # Florence-2 Coordinate Detection Model
Fine-tuned [Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) for captcha coordinate prediction.
## Usage
```python
from transformers import AutoModelForCausalLM, AutoProcessor
from PIL import Image
import re
model = AutoModelForCausalLM.from_pretrained("sadasd67/florence2-coordinates", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("sadasd67/florence2-coordinates", trust_remote_code=True)
image = Image.open("captcha.png")
inputs = processor(text="<COORDINATE>", images=image, return_tensors="pt")
generated = model.generate(**inputs, max_new_tokens=64)
result = processor.batch_decode(generated, skip_special_tokens=False)[0]
# Parse coordinates
locs = [int(x) for x in re.findall(r'<loc_(\d+)>', result)]
w, h = image.size
coords = [(locs[i] * w / 999, locs[i+1] * h / 999) for i in range(0, len(locs), 2)]
print(coords)
```
## Training Details
- **Base Model**: microsoft/Florence-2-base
- **Dataset**: 74233 captcha images with click coordinates
- **Training**: LoRA fine-tuning, 6 epochs
- **Best Val Loss**: 3.8039
- **Training Time**: 50.0 minutes
- **GPU**: Nvidia A10G 24GB (Modal.com)
## Coordinate Format
- Model outputs Florence-2 location tokens: `<loc_XXX>` (normalized 0-999)
- Denormalize: `pixel_x = loc_value * image_width / 999`
|