Tokenization and Text Normalization

Tokenization is the process of splitting a string of text into smaller units called tokens. These tokens can be words, subwords, characters, or punctuation marks. The primary purpose is to convert unstructured text into a sequence of discrete elements that a machine learning model can process. Without tokenization, models would receive raw strings and would not know where one word ends and another begins — they would have to learn boundaries implicitly, which is inefficient and often inaccurate.

Tokenization exists because most NLP models operate on fixed-size input vectors. A model cannot directly ingest a variable-length sentence; it must first break the sentence into tokens, map each token to an integer ID via a vocabulary, and then feed those IDs into the model. The choice of tokenization directly impacts vocabulary size, out-of-vocabulary (OOV) handling, and model performance.

Tokenization algorithms follow a deterministic set of rules. The simplest is whitespace tokenization: split on spaces, tabs, and newlines. For example:

Notice that punctuation is attached to the word. A more advanced approach is punctuation-aware tokenization, which splits on punctuation as well:

Subword tokenization, used in models like BERT and GPT, splits words into smaller units. For instance, the word "unhappiness" might become ["un", "happiness"] or ["un", "happi", "ness"]. The most common subword algorithm is Byte-Pair Encoding (BPE). BPE starts with a base vocabulary of individual characters and iteratively merges the most frequent pair of adjacent tokens. For example, if "th" appears often, it becomes a new token. This process continues until a desired vocabulary size is reached (e.g., 30,000 for BERT).

Text normalization is a set of preprocessing steps applied before or after tokenization to reduce variability in text. Common techniques include:

In Azure AI Language, tokenization and normalization are built into the text analytics APIs. For example, the Azure Text Analytics service automatically tokenizes text when performing sentiment analysis or key phrase extraction. You do not configure tokenization directly; instead, you specify the language and the API handles the rest. However, when using Azure Machine Learning to train custom models, you may use libraries like Hugging Face's transformers or spaCy, where you explicitly load a tokenizer:

from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokens = tokenizer.tokenize("Hello world!")
print(tokens)  # Output: ['hello', 'world', '!']

To verify tokenization, you can inspect the token IDs:

ids = tokenizer.encode("Hello world!")
print(ids)  # Output: [101, 7592, 2088, 999, 102]
# 101 = [CLS], 102 = [SEP], 7592 = 'hello', 2088 = 'world', 999 = '!'

Tokenization interacts with embedding layers, which convert token IDs into dense vectors. The embedding layer is a lookup table of size (vocab_size x embedding_dim). If the vocabulary size is 30,000 and embedding dimension is 768, the embedding matrix has 30,000 * 768 = 23 million parameters. Tokenization also affects attention mechanisms: the attention layer computes relationships between every pair of tokens, so sequence length directly impacts computational cost (O(n^2)). Therefore, efficient tokenization that produces shorter sequences (e.g., subword vs character) is crucial for performance.

A large e-commerce company deploys an Azure AI-powered chatbot to handle customer inquiries. The chatbot uses a pre-trained BERT model fine-tuned on support tickets. The raw customer messages contain typos, emojis, and mixed case. Tokenization and normalization are critical: the tokenizer must handle misspellings (e.g., 'ordr' might become ['ord', '##r']), and lowercasing ensures 'Order' and 'order' are the same token. The company uses Azure AI Language's built-in tokenization, which automatically normalizes text. However, they discovered that emojis like 😊 are tokenized as [UNK] if not in the vocabulary, so they added a preprocessing step to remove emojis. The chatbot handles thousands of messages per second; tokenization must be fast. Azure's tokenizers are optimized for throughput, and the company uses batch processing to maximize efficiency.

A law firm uses Azure Machine Learning to classify legal documents. The documents are lengthy (50,000+ words) and contain specialized legal jargon. Tokenization must handle long sequences: the model's max length is 512 tokens, so the firm uses a sliding window approach — they split documents into overlapping chunks of 512 tokens, tokenize each chunk, and aggregate results. They also use a custom vocabulary that includes legal terms like 'habeas corpus' as single tokens to improve accuracy. Text normalization is minimal because case and punctuation carry legal meaning (e.g., 'U.S.' vs 'us'). The firm uses a cased tokenizer to preserve this information. Misconfiguration of tokenization (e.g., using an uncased model) led to a 15% drop in classification accuracy, so they carefully validated their preprocessing pipeline.

A media monitoring service aggregates news articles in 50 languages. They use Azure's multilingual BERT model, which supports 104 languages. Tokenization is language-specific: for Chinese, the tokenizer uses character-level tokens because words are not space-separated. For Arabic, it handles right-to-left text. The service must normalize text differently per language — e.g., lowercasing is not appropriate for German nouns. They use Azure AI Language's language detection to route text to the correct tokenizer. A common issue is tokenizing mixed-language content (e.g., 'I love Berlin' in English with a German city name). The tokenizer may split 'Berlin' into ['Ber', '##lin'], which still works but may lose the entity. They mitigate this by adding entity-specific tokens to the vocabulary. Performance considerations: tokenization for 50 languages requires more memory for multiple vocabularies, but Azure's cloud infrastructure scales horizontally.