Instructions to use IHaBiS/Synatra-7B-v0.3-base-exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use IHaBiS/Synatra-7B-v0.3-base-exl2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="IHaBiS/Synatra-7B-v0.3-base-exl2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("IHaBiS/Synatra-7B-v0.3-base-exl2")
model = AutoModelForCausalLM.from_pretrained("IHaBiS/Synatra-7B-v0.3-base-exl2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use IHaBiS/Synatra-7B-v0.3-base-exl2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "IHaBiS/Synatra-7B-v0.3-base-exl2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IHaBiS/Synatra-7B-v0.3-base-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/IHaBiS/Synatra-7B-v0.3-base-exl2

SGLang

How to use IHaBiS/Synatra-7B-v0.3-base-exl2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "IHaBiS/Synatra-7B-v0.3-base-exl2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IHaBiS/Synatra-7B-v0.3-base-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "IHaBiS/Synatra-7B-v0.3-base-exl2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IHaBiS/Synatra-7B-v0.3-base-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use IHaBiS/Synatra-7B-v0.3-base-exl2 with Docker Model Runner:
```
docker model run hf.co/IHaBiS/Synatra-7B-v0.3-base-exl2
```

Exl2 version of Synatra-7B-v0.3-base

branch

main : 8bpw h8
b6h8 : 6bpw h8
b4h8 : 4bpw h8

below this line is original readme

Synatra-7B-v0.3-base🐧

Support Me

시나트라는 개인 프로젝트로, 1인의 자원으로 개발되고 있습니다. 모델이 마음에 드셨다면 약간의 연구비 지원은 어떨까요?

Wanna be a sponser? Contact me on Telegram AlzarTakkarsen

License

This model is strictly non-commercial (cc-by-nc-4.0) use only. The "Model" is completely free (ie. base model, derivates, merges/mixes) to use for non-commercial purposes as long as the the included cc-by-nc-4.0 license in any parent repository, and the non-commercial use statute remains, regardless of other models' licences. The licence can be changed after new model released. If you are to use this model for commercial purpose, Contact me.

Model Details

Base Model
mistralai/Mistral-7B-Instruct-v0.1

Trained On
A6000 48GB * 8

Instruction format

It follows ChatML format and Alpaca(No-Input) format.

TODO

~~RP 기반 튜닝 모델 제작~~ ✅
~~데이터셋 정제~~ ✅
언어 이해능력 개선
~~상식 보완~~ ✅
토크나이저 변경

Model Benchmark

Ko-LLM-Leaderboard

On Benchmarking...

Implementation Code

Since, chat_template already contains insturction format above. You can use the code below.

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("maywell/Synatra-7B-v0.3-base")
tokenizer = AutoTokenizer.from_pretrained("maywell/Synatra-7B-v0.3-base")

messages = [
    {"role": "user", "content": "바나나는 원래 하얀색이야?"},
]

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])

Downloads last month: 3