Instructions to use Locutusque/Orca-2-13b-SFT-v6 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Locutusque/Orca-2-13b-SFT-v6 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Locutusque/Orca-2-13b-SFT-v6")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Locutusque/Orca-2-13b-SFT-v6")
model = AutoModelForCausalLM.from_pretrained("Locutusque/Orca-2-13b-SFT-v6", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Locutusque/Orca-2-13b-SFT-v6 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Locutusque/Orca-2-13b-SFT-v6"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Locutusque/Orca-2-13b-SFT-v6",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Locutusque/Orca-2-13b-SFT-v6

SGLang

How to use Locutusque/Orca-2-13b-SFT-v6 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Locutusque/Orca-2-13b-SFT-v6" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Locutusque/Orca-2-13b-SFT-v6",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Locutusque/Orca-2-13b-SFT-v6" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Locutusque/Orca-2-13b-SFT-v6",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Locutusque/Orca-2-13b-SFT-v6 with Docker Model Runner:
```
docker model run hf.co/Locutusque/Orca-2-13b-SFT-v6
```

Locutusque commited on Dec 22, 2023

Commit

f2b12e0

1 Parent(s): 9fbe7fa

Upload LlamaForCausalLM

Browse files

Files changed (8) hide show

config.json +1 -1
model-00008-of-00014.safetensors +1 -1
model-00009-of-00014.safetensors +1 -1
model-00010-of-00014.safetensors +1 -1
model-00011-of-00014.safetensors +1 -1
model-00012-of-00014.safetensors +1 -1
model-00013-of-00014.safetensors +1 -1
model-00014-of-00014.safetensors +1 -1

config.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "Locutusque/Orca-2-13b-SFT-v4",
   "architectures": [
     "LlamaForCausalLM"
   ],

 {
+  "_name_or_path": "Locutusque/Orca-2-13b-SFT-v6",
   "architectures": [
     "LlamaForCausalLM"
   ],

model-00008-of-00014.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9de8349d8f6fd6c265ab46f4e805e0c5c6e90bcd6daf30b760e2af861a2f88ff
 size 1903230032

 version https://git-lfs.github.com/spec/v1
+oid sha256:b337b504a52ede637e81482bfa55d0b204c8d65fdd355c5416862380d950c37e
 size 1903230032

model-00009-of-00014.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d02b05f1c09dc4b1dbe3f0710aa863ed04868a35886a61ff801c886590c606eb
 size 1903230032

 version https://git-lfs.github.com/spec/v1
+oid sha256:64c858f4d403a32a5edd3240c811884492e94d6ec3d2b59102ad8be535720ee0
 size 1903230032

model-00010-of-00014.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2dde7b2e899f75413cf8a6ced5dd7abd898a65dfd03f29f2021ded1cbc56bd5b
 size 1903230032

 version https://git-lfs.github.com/spec/v1
+oid sha256:a8bb6930141a49c333a93b646fd82b041ad2a5aadda33b3ef14e7c95911522b5
 size 1903230032

model-00011-of-00014.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:accd3fc5f1900f226e263bd27a724e60e8007d1d3901dc097a30993467db9f63
 size 1903230032

 version https://git-lfs.github.com/spec/v1
+oid sha256:8c65b8a774c7339d80f5bb067905fb875d060073e2ca9af6e29e2738cd85a344
 size 1903230032

model-00012-of-00014.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:66bc671c5239e8df7e4ef8ac6b7644225cf9977772d1b501244677777f91def0
 size 1903230032

 version https://git-lfs.github.com/spec/v1
+oid sha256:d5012f3658244b991daa678179e93f506a9568e8ae352b4bae3fdf4554d230a0
 size 1903230032

model-00013-of-00014.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:79fd2086ae908a41c9a3abb783e97026d4922091c3ce3a2ec1e0bc121d3fcca3
 size 1903230032

 version https://git-lfs.github.com/spec/v1
+oid sha256:af1718d0e5b2743ef4b13d8ea4d47ce84888273e5a8579faa42895dc0fab7ffb
 size 1903230032

model-00014-of-00014.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:419183fe613d5a5e283d81d674375a9a786cac614a55dff787ac19d07013dc0f
 size 1245267640

 version https://git-lfs.github.com/spec/v1
+oid sha256:319ae0d518895a4fe4d6269bd5f1361f6a022edd49462af0dcfc6633d58ff247
 size 1245267640