Instructions to use giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling")
model = AutoModelForCausalLM.from_pretrained("giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling

SGLang

How to use giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling with Docker Model Runner:
```
docker model run hf.co/giannisan/Mistral-10.7B-Instruct-v0.3-depth-upscaling
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

mistral-7b-instruct-v0.3-depth-upscaling

Mistral: a strong, cold northwesterly wind that blows through the Rhône valley and southern France into the Mediterranean, mainly in winter.

This is an attempt at depth upscaling , Based on the paper SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling, which is a technique designed to efficiently scale large language models. The process begins with structural depthwise scaling which may initially reduce performance, but this is rapidly restored during a crucial continued pretraining phase. This phase optimizes the expanded model's parameters to the new depth configuration, significantly enhancing performance.

It's important to note that this represents only the initial phase of the model's development. The next critical steps involve fine-tuning. As expected and according to the paper, the initial upscaled model in phase one (without fine-tuning) scores lower than the base model. This is expected to improve above and beyond this after fine-tuning is finished. Feel free to fine-tune on your own dataset.

Merge Details

Merge Method

This model was merged using the passthrough merge method. The first 24 layers of one copy of the model are stitched to the last 24 layers of another copy, resulting in a total of 48 layers with 10.7B parameters.

Models Merged

The following models were included in the merge:

mistralai/Mistral-7B-Instruct-v0.3 merged with itself.

Configuration

The following configuration was used to produce this model:

slices:
  - sources:
    - model: mistralai/Mistral-7B-Instruct-v0.3
      layer_range: [0, 24]
  - sources:
    - model: mistralai/Mistral-7B-Instruct-v0.3
      layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16

Eval results:

Metric	Value
Avg.	64.04
ARC (25-shot)	63.14
HellaSwag (10-shot)	83.29
MMLU (5-shot)	62.31
TruthfulQA (0-shot)	60.65
Winogrande (5-shot)	78.45
GSM8K (5-shot)	36.39
Full results here