Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
blackboxanalytics 
posted an update 6 days ago
Post
109
Athanor Lite -- free desktop app for running local models without the setup headache
Most tools in the local AI space assume you already know your way around quantization formats, context windows, and VRAM budgets. Athanor Lite is built for the people who just want to try a model without reading a wiki first.
It scans your hardware, shows you exactly which models fit your GPU, and handles download through inference. One app, no terminal, no config files. Uses llama.cpp under the hood, wrapped in a Tauri + Rust + React desktop app.
What it does:

Hardware detection (GPU architecture, VRAM, RAM, disk)
Model catalog with fit verdicts based on your actual specs
Ollama library import (zero-copy via hardlinks)
Real-time inference HUD (tok/s, GPU %, VRAM, temperature)
Workspace system for different model/task contexts
Zero telemetry, zero accounts, zero cloud

Windows for now, Mac is next. Free and open source.
GitHub: https://github.com/BBALabs/athanor-lite
Download: https://github.com/BBALabs/athanor-lite/releases/latest
Product page: https://bbasecure.com/athanor.html
dipankarsarkar

The fit verdict is the whole product, and it is the hard part.

A 9B at Q4 is maybe 6GB of weights, so it 'fits' an 8GB card at load. Then you open a 32K context and it OOMs, because the KV cache grows with sequence length and heads, not with the weights.

So 'fits your GPU' at load is a different claim than 'fits at the context you'll actually run.'

Does the verdict model KV-cache-at-target-context, or just the weight footprint at load?

·
blackboxanalytics

You're totally right, weight footprint alone doesn't tell the full story. The fit verdict does factor in KV cache and runtime overhead on top of the weights, so it's not just checking if the model loads. It should catch that 9B Q4 on 8GB scenario you're describing.

One thing I do want to improve though is letting users set their own target context length and watching the verdict adjust. Right now it uses the model's default, which isn't always how people actually run them. That's coming in v2. Good callout!
In this post