Transformers documentation

Qwen3 ASR

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Model ini dipublikasikan di HF papers pada 2026-01-29 dan disumbangkan ke Hugging Face Transformers pada 2026-06-26.

Qwen3 ASR

Ikhtisar

Qwen3 ASR adalah model pengenalan ucapan otomatis dari tim Qwen di Alibaba yang menggabungkan encoder audio bergaya Whisper dengan decoder model bahasa Qwen3 untuk transkripsi ucapan-ke-teks. Model ini mendukung deteksi bahasa otomatis dan transkripsi multibahasa.

Model forced aligner juga disertakan. Model ini dapat digunakan untuk memberi cap waktu pada transkrip dan audio yang diberikan. Model ini menggunakan model encoder audio yang sama dengan kepala klasifikasi yang memprediksi panjang sebuah kata. Model ini dapat digunakan dengan transkrip dari model ASR mana pun (lihat contoh di bawah dengan Parakeet CTC).

Checkpoint yang tersedia:

Bahasa-bahasa berikut didukung:

Qwen3-ASR-1.7B dan Qwen3-ASR-0.6B: Mandarin (zh), Inggris (en), Kanton (yue), Arab (ar), Jerman (de), Prancis (fr), Spanyol (es), Portugis (pt), Indonesia (id), Italia (it), Korea (ko), Rusia (ru), Thailand (th), Vietnam (vi), Jepang (ja), Turki (tr), Hindi (hi), Melayu (ms), Belanda (nl), Swedia (sv), Denmark (da), Finlandia (fi), Polandia (pl), Ceko (cs), Filipina (fil), Persia (fa), Yunani (el), Hungaria (hu), Makedonia (mk), Rumania (ro).
Qwen3-ForcedAligner-0.6B: Mandarin (zh), Inggris (en), Kanton (yue), Prancis (fr), Jerman (de), Italia (it), Jepang (ja), Korea (ko), Portugis (pt), Rusia (ru), Spanyol (es).

Lihat repositori asli di QwenLM/Qwen3-ASR dan laporan untuk detail lebih lanjut.

Model ini disumbangkan oleh Eric Bezzam dan Muhammed Tariq.

Penggunaan

Transkripsi sederhana

Cara termudah untuk mentranskripsi audio adalah dengan apply_transcription_request, yang menangani format template chat untuk Anda, yaitu pembungkus kenyamanan untuk apply_chat_template (lihat Template chat di bawah).

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
print(f"Model loaded on {model.device} with dtype {model.dtype}")

inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]

# Raw output includes language tag and <asr_text> marker
raw = processor.decode(generated_ids)[0]
print(f"Raw: {raw}")

# Parsed output: dict with "language" and "transcription"
parsed = processor.decode(generated_ids, return_format="parsed")[0]
print(f"Parsed: {parsed}")

# Extract only the transcription text
transcription = processor.decode(generated_ids, return_format="transcription_only")[0]
print(f"Transcription: {transcription}")

"""
Raw: language English<asr_text>Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
Parsed: {'language': 'English', 'transcription': 'Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'}
Transcription: Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
"""

Memaksa bahasa

Anda dapat memaksa bahasa transkripsi seperti yang ditunjukkan di bawah ini.

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

# Without language hint (auto-detect)
inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
).to(model.device, model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(f"Auto-detect: {processor.decode(generated_ids, return_format='transcription_only')[0]}")

# With forced language
inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
    language="Chinese",  # or language code "zh"
).to(model.device, model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(f"Forced:      {processor.decode(generated_ids, return_format='transcription_only')[0]}")

Konteks / kata kunci

Anda dapat memberikan konteks bebas (misalnya, kosakata khusus domain, nama, atau informasi latar belakang) melalui prompt untuk memengaruhi transkripsi.

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
    prompt="Vocabulary: Quilter, apostle, gospel.",
    language="English",
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(processor.decode(generated_ids, return_format="transcription_only")[0])

Inferensi batch

Inferensi batch dimungkinkan dengan meneruskan daftar audio dan, jika disediakan, daftar bahasa dan/atau prompt.

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
audio = [
    "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
    "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
]

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

inputs = processor.apply_transcription_request(
    audio,
    prompt=["Vocabulary: Quilter, apostle, gospel.", None],
    language=[None, "zh"],  # language codes ("zh") and full names ("Chinese") are both accepted
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
transcriptions = processor.decode(generated_ids, return_format="transcription_only")

for i, text in enumerate(transcriptions):
    print(f"Audio {i + 1}: {text}")

Template chat

Qwen3 ASR juga menerima input template chat. Penggunaan apply_transcription_request di atas adalah pembungkus kenyamanan untuk apply_chat_template.

Bahasa dapat dipaksa melalui chat template dengan cara prefilling giliran asisten menggunakan language <NAME><asr_text> dan meneruskan continue_final_message=True, yang merupakan apa yang dilakukan apply_transcription_request di balik layar. Perhatikan bahwa jika memaksa bahasa, prefill harus disetel untuk semua audio dalam batch (seperti yang ditunjukkan di bawah).

from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3ASRForConditionalGeneration.from_pretrained(model_id, device_map="auto")

chat_template = [
    [
        # Context/hotwords as system message
        {"role": "system", "content": [{"type": "text", "text": "Vocabulary: Quilter, apostle, gospel."}]},
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
                },
            ],
        },
        # empty prefill since forcing language in the other sample
        {"role": "assistant", "content": [{"type": "text", "text": ""}]},
    ],
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
                },
            ],
        },
        {"role": "assistant", "content": [{"type": "text", "text": "language Chinese<asr_text>"}]},
    ],
]

inputs = processor.apply_chat_template(
    chat_template, tokenize=True, return_dict=True, continue_final_message=True,
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
transcriptions = processor.decode(generated_ids, return_format="transcription_only")
for text in transcriptions:
    print(text)

Pelatihan

Lewatkan transkrip target ke apply_transcription_request dengan transcription untuk mendapatkan labels yang digunakan untuk menghitung loss.

from datasets import Audio, load_dataset
from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
NUM_SAMPLES = 2

processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3ASRForConditionalGeneration.from_pretrained(model_id, device_map="auto")
model.train()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]
text_samples = ds["text"][:NUM_SAMPLES]

inputs = processor.apply_transcription_request(speech_samples, language="en", transcription=text_samples)
inputs.to(model.device, dtype=model.dtype)

loss = model(**inputs).loss
print("Loss:", loss.item())
loss.backward()

Penyelarasan paksa (penandaan waktu tingkat kata)

Gunakan Qwen3ASRForTokenClassification untuk mendapatkan timestamp tingkat kata dari transkrip. Pertama, lakukan transkripsi dengan model ASR, kemudian lakukan perataan dengan forced aligner.

Bahasa-bahasa berikut didukung: Mandarin (zh), Inggris (en), Kanton (yue), Prancis (fr), Jerman (de), Italia (it), Jepang (ja), Korea (ko), Portugis (pt), Rusia (ru), Spanyol (es).

Bahasa Jepang memerlukan pustaka nagisa, sedangkan bahasa Korea memerlukan pustaka soynlp:

pip install nagisa soynlp

Dengan Qwen3 ASR

import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM, AutoModelForTokenClassification

asr_model_id = "Qwen/Qwen3-ASR-0.6B-hf"
aligner_model_id = "Qwen/Qwen3-ForcedAligner-0.6B-hf"

asr_processor = AutoProcessor.from_pretrained(asr_model_id)
asr_model = AutoModelForMultimodalLM.from_pretrained(asr_model_id, device_map="auto")

aligner_processor = AutoProcessor.from_pretrained(aligner_model_id)
aligner_model = AutoModelForTokenClassification.from_pretrained(
    aligner_model_id, dtype=torch.bfloat16, device_map="auto"
)

audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"

# Step 1: Transcribe
inputs = asr_processor.apply_transcription_request(audio=audio_url)
inputs = inputs.to(asr_model.device, asr_model.dtype)
output_ids = asr_model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
parsed = asr_processor.decode(generated_ids, return_format="parsed")[0]
transcript = parsed["transcription"]
language = parsed["language"] or "English"

# Step 2: Prepare alignment inputs
aligner_inputs, word_lists = aligner_processor.prepare_forced_aligner_inputs(
    audio=audio_url, transcript=transcript, language=language,
)
aligner_inputs = aligner_inputs.to(aligner_model.device, aligner_model.dtype)

# Step 3: Run forced aligner
with torch.inference_mode():
    outputs = aligner_model(**aligner_inputs)

# Step 4: Decode timestamps
timestamps = aligner_processor.decode_forced_alignment(
    logits=outputs.logits,
    input_ids=aligner_inputs["input_ids"],
    word_lists=word_lists,
    timestamp_token_id=aligner_model.config.timestamp_token_id,
)[0]

for item in timestamps:
    print(f"{item['text']:<20} {item['start_time']:>8.3f}s → {item['end_time']:>8.3f}s")

"""
Word                  Start (s)    End (s)
------------------------------------------
Mr                        0.560      0.800
Quilter                   0.800      1.280
is                        1.280      1.440
the                       1.440      1.520
apostle                   1.520      2.080
...
"""

Dengan model ASR lain

Forced aligner bersifat model-agnostic, artinya transkrip dari sistem ASR apa pun dapat digunakan. Di bawah ini adalah contoh inferensi batch menggunakan NVIDIA Parakeet CTC untuk transkripsi.

import torch
from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor, AutoModelForTokenClassification

parakeet_processor = AutoProcessor.from_pretrained("nvidia/parakeet-ctc-1.1b")
parakeet_model = AutoModelForCTC.from_pretrained(
    "nvidia/parakeet-ctc-1.1b", dtype="auto", device_map="auto",
)

aligner_model_id = "Qwen/Qwen3-ForcedAligner-0.6B-hf"
aligner_processor = AutoProcessor.from_pretrained(aligner_model_id)
aligner_model = AutoModelForTokenClassification.from_pretrained(
    aligner_model_id, dtype=torch.bfloat16, device_map="auto",
)

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=parakeet_processor.feature_extractor.sampling_rate))
audio_arrays = [ds[i]["audio"]["array"] for i in range(3)]
sr = ds[0]["audio"]["sampling_rate"]

# Batch transcribe with Parakeet
inputs = parakeet_processor(audio_arrays, sampling_rate=sr, return_tensors="pt", padding=True).to(
    parakeet_model.device, dtype=parakeet_model.dtype
)
with torch.inference_mode():
    outputs = parakeet_model.generate(**inputs)
transcripts = parakeet_processor.decode(outputs)

# Batch align with Qwen3 Forced Aligner
aligner_inputs, word_lists = aligner_processor.prepare_forced_aligner_inputs(
    audio=audio_arrays, transcript=transcripts, language="English",
)
aligner_inputs = aligner_inputs.to(aligner_model.device, aligner_model.dtype)

with torch.inference_mode():
    aligner_outputs = aligner_model(**aligner_inputs)

batch_timestamps = aligner_processor.decode_forced_alignment(
    logits=aligner_outputs.logits,
    input_ids=aligner_inputs["input_ids"],
    word_lists=word_lists,
    timestamp_token_id=aligner_model.config.timestamp_token_id,
)

for i, (transcript, timestamps) in enumerate(zip(transcripts, batch_timestamps)):
    print(f"\n[Sample {i}] {transcript}")
    for item in timestamps[:5]:
        print(f"  {item['text']:<20} {item['start_time']:>8.3f}s → {item['end_time']:>8.3f}s")
    if len(timestamps) > 5:
        print(f"  ... ({len(timestamps) - 5} more words)")

Torch compile

Model ASR dan forced aligner sama-sama mendukung torch.compile untuk inferensi yang lebih cepat. Forced aligner sangat cocok untuk kompilasi karena hanya menjalankan satu forward pass (tanpa decoding autoregresif). Hal ini menjadikannya ideal untuk penandaan waktu audio dalam jumlah besar: transkripsi dengan model ASR apa pun, lalu lakukan batch-align menggunakan forced aligner yang telah dikompilasi untuk throughput maksimal.

Aligner paksa

Pada A100, kami mengamati percepatan sekitar 1,2 untuk ukuran batch 4 (script).

import torch
from transformers import AutoProcessor, AutoModelForTokenClassification

model_id = "Qwen/Qwen3-ForcedAligner-0.6B-hf"
num_warmup = 5
batch_size = 4

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda")

# Prepare a batch of 4 samples
audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"
transcript = "Mr. Quilter is the apostle of the middle classes."

aligner_inputs, word_lists = processor.prepare_forced_aligner_inputs(
    audio=[audio_url] * batch_size,
    transcript=[transcript] * batch_size,
    language=["English"] * batch_size,
)
aligner_inputs = aligner_inputs.to(model.device, torch.bfloat16)

# Warm-up and apply model
model = torch.compile(model)
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**aligner_inputs)
with torch.no_grad():
    _ = model(**aligner_inputs)

Model ASR (generate)

Untuk transkripsi autoregresif, torch.compile mempercepat langkah maju per token di dalam generate dengan menyediakan objek CompileConfig.

Pada A100, kami mengamati percepatan sekitar 3,8 untuk ukuran batch 4 (script).

import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM, CompileConfig

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
num_warmup = 3
max_new_tokens = 256

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"
inputs = processor.apply_transcription_request(
    audio=[audio_url] * 4,  # batch of 4
).to(model.device, torch.bfloat16)

compile_config = CompileConfig()

# Warmup
with torch.inference_mode():
    for _ in range(num_warmup):
        _ = model.generate(
            **inputs, max_new_tokens=max_new_tokens, do_sample=False,
            cache_implementation="static", compile_config=compile_config,
        )
torch.cuda.synchronize()

# Apply model
with torch.inference_mode():
    output_ids = model.generate(
        **inputs, max_new_tokens=max_new_tokens, do_sample=False,
        cache_implementation="static", compile_config=compile_config,
    )
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
text_compiled = processor.decode(generated_ids, return_format="transcription_only")[0]
print(f"Output:  {text_compiled}")

Penggunaan Pipeline

from transformers import pipeline

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
pipe = pipeline("any-to-any", model=model_id, device_map="auto")

chat_template = [
    {
        "role": "user",
        "content": [
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
            },
        ],
    }
]
outputs = pipe(text=chat_template, return_full_text=False)
raw_text = outputs[0]["generated_text"]
print(f"Raw: {raw_text}")

# Use processor helper to extract transcription
transcription = pipe.processor.extract_transcription(raw_text)
print(f"Transcription: {transcription}")

Transformers

Qwen3 ASR

Ikhtisar

Penggunaan

Transkripsi sederhana

Memaksa bahasa

Konteks / kata kunci

Inferensi batch

Template chat

Pelatihan

Penyelarasan paksa (penandaan waktu tingkat kata)

Dengan Qwen3 ASR

Dengan model ASR lain

Torch compile

Aligner paksa

Model ASR (generate)

Penggunaan Pipeline

Qwen3ASRConfig

kelas transformers.Qwen3ASRConfig

Qwen3ASREncoderConfig

kelas transformers.Qwen3ASREncoderConfig

Qwen3ASRFeatureExtractor

kelas transformers.Qwen3ASRFeatureExtractor

__call__

Qwen3ASRProcessor

kelas transformers.Qwen3ASRProcessor

__call__

apply_transcription_request

prepare_forced_aligner_inputs

decode_forced_alignment

decode

Qwen3ASREncoder

kelas transformers.Qwen3ASREncoder

forward

Qwen3ASRModel

kelas transformers.Qwen3ASRModel

forward

get_audio_features

get_placeholder_mask

Qwen3ASRForConditionalGeneration

kelas transformers.Qwen3ASRForConditionalGeneration

forward

get_audio_features

Qwen3ASRForTokenClassification

kelas transformers.Qwen3ASRForTokenClassification

forward

call

call