Instructions to use KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: ./llama-cli -hf KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: ./build/bin/llama-cli -hf KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0
Use Docker
docker model run hf.co/KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0
- LM Studio
- Jan
- Ollama
How to use KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF with Ollama:
ollama run hf.co/KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0
- Unsloth Desktop
- Docker Model Runner
How to use KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF with Docker Model Runner:
docker model run hf.co/KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0
- Lemonade
How to use KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF:Q8_0
Run and chat with the model
lemonade run user.KaLM-Reranker-V1-Small-Q8_0-GGUF-Q8_0
List all available models
lemonade list
- Atomic Chat
Update README.md
Browse files
README.md
CHANGED
|
@@ -25,8 +25,7 @@ tags:
|
|
| 25 |
> and other stock llama.cpp frontends are not supported by this release.
|
| 26 |
|
| 27 |
This repository contains the text-only Q8_0 GGUF conversion of
|
| 28 |
-
[`KaLM-Embedding/KaLM-Reranker-V1-Small`](https://huggingface.co/KaLM-Embedding/KaLM-Reranker-V1-Small).
|
| 29 |
-
not a text-generation or chat model.
|
| 30 |
|
| 31 |
## Model file
|
| 32 |
|
|
@@ -165,3 +164,38 @@ The source model declares Apache-2.0. See [`LICENSE`](LICENSE) and
|
|
| 165 |
[`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md). The bundled llama.cpp
|
| 166 |
patches retain the upstream MIT license in
|
| 167 |
[`llama.cpp/LICENSE`](llama.cpp/LICENSE).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
> and other stock llama.cpp frontends are not supported by this release.
|
| 26 |
|
| 27 |
This repository contains the text-only Q8_0 GGUF conversion of
|
| 28 |
+
[`KaLM-Embedding/KaLM-Reranker-V1-Small`](https://huggingface.co/KaLM-Embedding/KaLM-Reranker-V1-Small).
|
|
|
|
| 29 |
|
| 30 |
## Model file
|
| 31 |
|
|
|
|
| 164 |
[`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md). The bundled llama.cpp
|
| 165 |
patches retain the upstream MIT license in
|
| 166 |
[`llama.cpp/LICENSE`](llama.cpp/LICENSE).
|
| 167 |
+
|
| 168 |
+
# Citation
|
| 169 |
+
If you find this model useful, please consider citing our papers.
|
| 170 |
+
```
|
| 171 |
+
@misc{zhao2026kalmrerankerv1,
|
| 172 |
+
title={KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking},
|
| 173 |
+
author={Xinping Zhao and Jiaxin Xu and Ziqi Dai and Xin Zhang and Shouzheng Huang and Danyu Tang and Xinshuo Hu and Meishan Zhang and Baotian Hu and Min Zhang},
|
| 174 |
+
year={2026},
|
| 175 |
+
eprint={2606.22807},
|
| 176 |
+
archivePrefix={arXiv},
|
| 177 |
+
primaryClass={cs.CL},
|
| 178 |
+
url={https://arxiv.org/abs/2606.22807},
|
| 179 |
+
}
|
| 180 |
+
|
| 181 |
+
@misc{zhao2026kalmembeddingv2,
|
| 182 |
+
title={KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model},
|
| 183 |
+
author={Xinping Zhao and Xinshuo Hu and Zifei Shan and Shouzheng Huang and Yao Zhou and Xin Zhang and Zetian Sun and Zhenyu Liu and Dongfang Li and Xinyuan Wei and Youcheng Pan and Yang Xiang and Meishan Zhang and Haofen Wang and Jun Yu and Baotian Hu and Min Zhang},
|
| 184 |
+
year={2025},
|
| 185 |
+
eprint={2506.20923},
|
| 186 |
+
archivePrefix={arXiv},
|
| 187 |
+
primaryClass={cs.CL},
|
| 188 |
+
url={https://arxiv.org/abs/2506.20923},
|
| 189 |
+
}
|
| 190 |
+
|
| 191 |
+
@misc{hu2025kalmembedding,
|
| 192 |
+
title={KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model},
|
| 193 |
+
author={Xinshuo Hu and Zifei Shan and Xinping Zhao and Zetian Sun and Zhenyu Liu and Dongfang Li and Shaolin Ye and Xinyuan Wei and Qian Chen and Baotian Hu and Haofen Wang and Jun Yu and Min Zhang},
|
| 194 |
+
year={2025},
|
| 195 |
+
eprint={2501.01028},
|
| 196 |
+
archivePrefix={arXiv},
|
| 197 |
+
primaryClass={cs.CL},
|
| 198 |
+
url={https://arxiv.org/abs/2501.01028},
|
| 199 |
+
}
|
| 200 |
+
```
|
| 201 |
+
|