Transformers documentation

Muse Spark Open

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.1).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

이 모델은 2026-10-14에 Hugging Face Transformers에 기여되었습니다.

Muse Spark Open

개요

Muse Spark Open은 Meta에서 개발한 멀티모달 mixture-of-experts(MoE) 모델로, 텍스트, 이미지, 오디오, 비디오를 입력으로 받아 텍스트를 생성합니다. 총 8160억 개의 파라미터를 가지며 토큰당 420억 개의 활성 파라미터를 사용하고, 모든 입력 모달리티에 걸쳐 최대 1,048,576 토큰까지의 컨텍스트 윈도우를 지원합니다. 이 모델은 멀티파일 코드 편집, 디버깅, 도구 사용 등 에이전틱 코딩 작업을 위해 학습되었습니다.

언어 백본은 62개의 transformer 레이어로 구성되며, 각 레이어는 토큰을 256개 전문가 중 8개로 라우팅합니다. query-key 정규화, attention 출력 게이팅 및 정규화, 학습된 잔차 게이트를 사용하는 grouped-query attention을 적용합니다. 공유된 perception 인코더가 이미지와 비디오 프레임을 처리하고, convolutional 오디오 인코더가 초당 12.5개의 토큰으로 오디오 표현을 생성합니다. Muse Spark Open은 Muse Glimmer와 토크나이저 및 어휘를 공유하며, 멀티모달 입력을 위해 채팅 템플릿을 확장했습니다. 체크포인트는 BF16 및 NVFP4 형식으로 제공됩니다.

리소스

AutoModel

AutoModel 이미지

AutoModel Video

AutoModel Audio

import os

from transformers import AutoModelForMultimodalLM, AutoProcessor, DistributedConfig

distributed_config = DistributedConfig(
    tp_size=int(os.environ["WORLD_SIZE"]), # Number of devices across which the dense layers are sharded
    ep_size=int(os.environ["WORLD_SIZE"]), # Number of devices across which the experts are distributed
)

processor = AutoProcessor.from_pretrained("meta-models/Muse-Spark-Open-816B-A42B")
model = AutoModelForMultimodalLM.from_pretrained(
    "meta-models/Muse-Spark-Open-816B-A42B",
    distributed_config=distributed_config,
)

messages = [
    {"role": "user", "content": "Explain in one sentence why the sky is usually blue."},
]
inputs = processor.apply_chat_template(
    messages,
    reasoning_effort="low",  # default: xhigh, options: low/medium/high/xhigh
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device).to(model.dtype)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(outputs[0][input_len:], skip_special_tokens=False))
#  to=self<|message|>Explain in one sentence why the sky is usually blue.
# ...
# <|eom|><|start|>assistant to=user<|message|>The sky is usually blue because molecules in Earth's atmosphere scatter shorter-wavelength blue sunlight in all directions much more strongly than longer-wavelength red light.<|eot|>

import os

from transformers import AutoModelForMultimodalLM, AutoProcessor, DistributedConfig

distributed_config = DistributedConfig(
    tp_size=int(os.environ["WORLD_SIZE"]),
    ep_size=int(os.environ["WORLD_SIZE"]),
)

processor = AutoProcessor.from_pretrained("meta-models/Muse-Spark-Open-816B-A42B")
model = AutoModelForMultimodalLM.from_pretrained(
    "meta-models/Muse-Spark-Open-816B-A42B",
    distributed_config=distributed_config,
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    },
]
inputs = processor.apply_chat_template(
    messages,
    reasoning_effort="medium", # default: xhigh, options: low/medium/high/xhigh
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device).to(model.dtype)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=512)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
print(response)
#  to=self<|message|>What is shown in this image?
# ...
# <|eom|><|start|>assistant to=user<|message|>The image shows a **Pallas's cat — also known as a manul, *Otocolobus manul*** — walking through snow.
#
# It is a small, stocky wild cat recognizable by:
#
# * **Its extremely dense, long, greyish-brown winter coat** with faint dark stripes and a dusting of snow on its back
# * **Its flat, broad face** with low-set ears, a short muzzle, and dark stripes on the cheeks
# * **Its short legs and thick, plump body** adapted for cold climates
#
# The cat is captured mid-stride on a snow-covered surface, with one front paw lifted. In the background are white birch tree trunks with black markings and a chain-link fence on the left, indicating the animal is in a zoo or wildlife enclosure in winter.<|eot|>

import os

from transformers import AutoModelForMultimodalLM, AutoProcessor, DistributedConfig

distributed_config = DistributedConfig(
    tp_size=int(os.environ["WORLD_SIZE"]),
    ep_size=int(os.environ["WORLD_SIZE"]),
)

processor = AutoProcessor.from_pretrained("meta-models/Muse-Spark-Open-816B-A42B")
model = AutoModelForMultimodalLM.from_pretrained(
    "meta-models/Muse-Spark-Open-816B-A42B",
    distributed_config=distributed_config,
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "url": "https://huggingface.co/datasets/hf-internal-testing/fixtures_videos/resolve/main/tennis.mp4"},
            {"type": "text", "text": "What is happening in this video?"},
        ],
    },
]
inputs = processor.apply_chat_template(
    messages,
    reasoning_effort="medium",  # default: xhigh, options: low/medium/high/xhigh
    processor_kwargs={
        "fps": 2,   # default: 2, number of frames sampled per second
        "num_frames": 8, # default: 94, maximum number of frames sampled from the video
    },
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device).to(model.dtype)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(outputs[0][input_len:], skip_special_tokens=False))
#  to=self<|message|>What is happening in this video?
# ...
# <|eom|><|start|>assistant to=user<|message|>In this video two people are playing tennis on an indoor court.
#
# The shot is taken from behind one of the players - a man in a white t-shirt, black shorts and white shoes and socks. He is standing near the baseline on an orange hard court with white lines, in a ready position with his knees bent and his racket held in front of him.
#
# Across the net, on the far side of the court, is his opponent, also dressed in light-colored clothing.
#
# The sequence shows a short rally:
#
# *   The opponent hits the ball over the net.
# *   The man in the foreground shuffles to his left, turns his body, and prepares for a shot.
# *   He executes a two-handed backhand, swinging the racket across his body and following through as he hits the ball back over the net.
#
# The setting is a large indoor tennis hall with a high, arched ceiling with exposed wooden beams and overhead fluorescent lights. The court is surrounded by a grey run-off area, green windscreening on the walls, and netting that separates it from adjacent courts. On the right side you can see an umpire's chair and a few yellow and black benches. An "Adobe Stock" watermark is visible in the center of the frame.<|eot|>

NVFP4 가중치

NVFP4 양자화 가중치는 더 낮은 메모리 요구 사항으로 허브에서 사용할 수 있습니다. 이 모델은 코드를 수정하지 않고 NVFP4 호환 장치에서 로드할 수 있습니다. 양자화 구성은 로드 시점에 자동으로 적용됩니다.

NVFP4 지원을 위해 이 브랜치에서 transformers를 설치해야 합니다.
pip install git+https://github.com/huggingface/transformers.git@finegrained-per-scheme-demo

import os

from transformers import AutoModelForMultimodalLM, AutoProcessor, DistributedConfig

distributed_config = DistributedConfig(
    tp_size=int(os.environ["WORLD_SIZE"]),
    ep_size=int(os.environ["WORLD_SIZE"]),
)

processor = AutoProcessor.from_pretrained("meta-models/Muse-Spark-Open-816B-A42B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained(
    "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    distributed_config=distributed_config,
)

# ...

Transformers

Muse Spark Open

개요

리소스

NVFP4 가중치

MuseSparkConfig

클래스 transformers.MuseSparkConfig

MuseSparkTextConfig

class transformers.MuseSparkTextConfig

MuseSparkVisionConfig

클래스 transformers.MuseSparkVisionConfig

MuseSparkAudioConfig

클래스 transformers.MuseSparkAudioConfig

MuseSparkFeatureExtractor

클래스 transformers.MuseSparkFeatureExtractor

MuseSparkProcessor

클래스 transformers.MuseSparkProcessor

MuseSparkPreTrainedModel

클래스 transformers.MuseSparkPreTrainedModel

MuseSparkTextModel

클래스 transformers.MuseSparkTextModel

forward

MuseSparkVisionModel

class transformers.MuseSparkVisionModel

forward

MuseSparkAudioModel

class transformers.MuseSparkAudioModel

forward

MuseSparkModel

클래스 transformers.MuseSparkModel

forward

MuseSparkForConditionalGeneration

클래스 transformers.MuseSparkForConditionalGeneration

forward