Bytelevel Normalizer

Converts all bytes in the input to their Unicode representation using the GPT-2 byte-to-unicode mapping. Every byte value (0–255) is mapped to a unique visible character so that any arbitrary binary input can be tokenized without needing a special unknown token.

This normalizer is used together with the ByteLevel pre-tokenizer and ByteLevel decoder.

>>> from tokenizers.normalizers import ByteLevel
>>> normalizer = ByteLevel()
>>> normalizer.normalize_str("hello\nworld")
'helloĊworld'

Lowercase Normalizer

Converts all text to lowercase using Unicode-aware lowercasing. This is equivalent to calling str.lower on the input.

>>> from tokenizers.normalizers import Lowercase
>>> normalizer = Lowercase()
>>> normalizer.normalize_str("Hello World")
'hello world'

NFC Unicode Normalizer

Applies Unicode NFC (Canonical Decomposition, followed by Canonical Composition) normalization. First decomposes characters, then recomposes them using canonical composition rules. This produces the canonical composed form.

>>> from tokenizers.normalizers import NFC
>>> normalizer = NFC()
>>> normalizer.normalize_str("e\u0301")  # 'e' + combining accent
'é'

NFD Unicode Normalizer

Applies Unicode NFD (Canonical Decomposition) normalization. Decomposes characters into their canonical components. For example, accented characters like é (U+00E9) are decomposed into e (U+0065) + combining accent (U+0301).

This is often used as a first step before stripping accents with StripAccents.

>>> from tokenizers.normalizers import NFD
>>> normalizer = NFD()
>>> normalizer.normalize_str("Héllo")
'He\u0301llo'

NFKC Unicode Normalizer

Applies Unicode NFKC (Compatibility Decomposition, followed by Canonical Composition) normalization. Like NFC but also maps compatibility characters to their canonical equivalents. This is the normalization used by Python’s str.casefold and by many NLP pipelines.

>>> from tokenizers.normalizers import NFKC
>>> normalizer = NFKC()
>>> normalizer.normalize_str("fine caf\u00e9")
'fine café'

NFKD Unicode Normalizer

Applies Unicode NFKD (Compatibility Decomposition) normalization. Like NFD but also decomposes compatibility characters. For example, the ligature fi (U+FB01) is decomposed into f + i.

>>> from tokenizers.normalizers import NFKD
>>> normalizer = NFKD()
>>> normalizer.normalize_str("fine")
'fine'

Nmt normalizer

Normalizer used in the Google NMT pipeline. It handles various text cleaning tasks including removing control characters, normalizing whitespace, and replacing certain Unicode characters. This is equivalent to the normalization done in the original SentencePiece NMT preprocessing.

>>> from tokenizers.normalizers import Nmt
>>> normalizer = Nmt()
>>> normalizer.normalize_str("Hello\x00World")
'Hello World'

Base class for all normalizers

This class is not supposed to be instantiated directly. Instead, any implementation of a Normalizer will return an instance of this class when instantiated.

Precompiled normalizer

A normalizer that uses a precompiled character map built from a SentencePiece model. This normalizer is automatically extracted from SentencePiece .model files and should not be constructed manually — it is used internally for full compatibility with SentencePiece-based tokenizers.

Replace normalizer

Replaces occurrences of a pattern in the input string with the given content. The pattern can be either a plain string or a regular expression wrapped in Regex.

>>> from tokenizers import Regex
>>> from tokenizers.normalizers import Replace
>>> # Replace a literal string
>>> Replace(".", " ").normalize_str("hello.world")
'hello world'
>>> # Replace using a regex
>>> Replace(Regex(r"\s+"), " ").normalize_str("hello   world")
'hello world'

Allows concatenating multiple other Normalizer as a Sequence. All the normalizers run in sequence in the given order

>>> from tokenizers.normalizers import NFD, Lowercase, StripAccents, Sequence
>>> normalizer = Sequence([NFD(), Lowercase(), StripAccents()])
>>> normalizer.normalize_str("Héllo Wörld")
'hello world'

Strip normalizer

Removes leading and/or trailing whitespace from the input string.

>>> from tokenizers.normalizers import Strip
>>> normalizer = Strip()
>>> normalizer.normalize_str("  hello world  ")
'hello world'
>>> Strip(right=False).normalize_str("  hello  ")
'hello  '

StripAccents normalizer

Strips all accent marks (combining diacritical characters) from the input. This normalizer should typically be used after applying NFD or NFKD decomposition, which separates base characters from their combining accents.

>>> from tokenizers.normalizers import NFD, StripAccents, Sequence
>>> normalizer = Sequence([NFD(), StripAccents()])
>>> normalizer.normalize_str("café")
'cafe'

BertNormalizer

Takes care of normalizing raw text before giving it to a Bert model. This includes cleaning the text, handling accents, chinese chars and lowercasing

>>> from tokenizers.normalizers import BertNormalizer
>>> normalizer = BertNormalizer(lowercase=True)
>>> normalizer.normalize_str("Héllo WORLD")
'hello world'