Session 6 — Tokenization + embeddings: how text becomes numbers

Session 6 — Tokenization + embeddings: how text becomes numbers

The bridge between text (symbols) and neural networks (numbers). The first layer of every modern language model, and the one that quietly consumes 57% of huge-base-v1’s parameters.

Where we are

Sessions 1–5 covered what a neural net IS and what it learns. But we’ve been hand-waving over a key question: the model is a function on numbers. Where do the numbers come from when the input is text?

This session covers that bridge — tokenization (text → integer IDs) and embedding (integer IDs → vectors of floats) — and why these choices matter more than most beginners are told.

The problem

Neural networks are functions on numbers. They take numbers in, do arithmetic (matmul, ReLU, etc.), produce numbers out.

But text is symbols: characters, words, punctuation. "the cat sat on the mat" is not numbers. Before any neural net can process it, we need to turn it into numbers.

Naive approach 1: character-level

Assign each character a number (e.g., ASCII).

'a' = 97   'b' = 98   'c' = 99   ...
"cat" = [99, 97, 116]

Why it’s not great: sequences get very long (one token per character), inference is proportionally slow, network learns from very fine-grained signals.

Naive approach 2: word-level

Assign each word a number. Build vocabulary from a corpus.

"the" = 1   "cat" = 2   "sat" = 3   ...
"the cat sat" = [1, 2, 3]

Why it’s not great: vocabulary explodes (English has 1M+ word forms); can’t handle out-of-vocabulary words; subword structure lost (running and ran look unrelated).

Naive approach 3: byte-level

Each byte of UTF-8 is a token. Vocabulary fixed at 256.

"café" → [99, 97, 102, 195, 169]   ("é" is 2 UTF-8 bytes)

Better than character, worse than something smarter:

• Vocabulary fixed at 256 — never grows, never has <UNK>.
• Handles any language, any character.
• But sequences still long (one token per byte).
• This is byte-tinygpt-v0 in the browser. Fine for teaching; inefficient at scale.

The modern answer: BPE (Byte Pair Encoding)

Start with bytes. Find the most common adjacent pair. Merge it into a new token. Repeat until you have the vocabulary size you want.

Tiny worked example

Corpus:

"low lower lowest"

Step 0 — characters:

l o w _ l o w e r _ l o w e s t

Step 1 — l o appears 3 times. Merge to Lo:

Lo w _ Lo w e r _ Lo w e s t

Step 2 — Lo w appears 3 times. Merge to Low:

Low _ Low e r _ Low e s t

Step 3 — Low e appears 2 times. Merge to Lowe:

Low _ Lowe r _ Lowe s t

After enough merges, common patterns become single tokens:

low _ lower _ lowest    ← 3 tokens instead of starting 16

The vocabulary contains: every original byte (no <UNK>) PLUS all learned merges. For SmolLM2’s BPE: 49,152 tokens = 256 bytes + 48,896 learned merges.

What BPE buys

• Common words → single tokens.
• Rare/novel words → graceful degradation to subwords/bytes.
• Subword structure preserved (run + ning).
• Tunable vocab size: 8K → 200K depending on model size and language.

Same text, three tokenizers

"Hello, world! 42 ML":

BPE is ~3× more compact than byte-level on English.

From tokens to vectors: the embedding layer

Tokenization gives integer IDs. The neural network needs vectors of floats. The embedding layer is a lookup table:

Token ID 42 ("cat")
    ↓ lookup in embedding table (row 42)
[0.12, -0.45, 0.78, 0.21, ..., 0.31]    ← 256 floats
    ↓ flows into the transformer

The embedding table is a matrix:

shape: (vocab_size, d_model) = (49152, 256) = 12,582,912 parameters

That’s where the 12.5M comes from — and why it’s 57% of huge-base-v1’s total parameter budget.

What embeddings actually learn

Vectors are random at initialization. Training updates them along with everything else. They end up encoding meaning.

Tokens with similar meanings get similar vectors. The classic word2vec result (Mikolov et al., 2013):

vec("king")   - vec("man")    + vec("woman") ≈ vec("queen")
vec("Paris")  - vec("France") + vec("Italy") ≈ vec("Rome")
vec("walked") - vec("walk")   + vec("run")   ≈ vec("ran")

The embedding space encodes semantic and syntactic relationships as directions. Not by design — it emerges from gradient descent on “predict the next word.”

Embedding stores identity. The body stores reasoning.

A common confusion: thinking of embeddings as “chains” or “trees” of related concepts. They’re not.

Each token has ONE vector. That’s it.

The vector is a fixed POINT in a high-dimensional concept space. Similar concepts cluster near each other; distant concepts are far apart. But there are no chains, no linked lists, no graph structure stored in the embedding.

The “chains” of reasoning — what concept relates to what, given THIS input — are built at runtime by the BODY (attention + MLPs). The embedding is dictionary; the body is reader. See journal Entry 11 for the longer version of this distinction.

Output side: tied vs untied embeddings

The final step predicts probabilities over the vocabulary, requiring a matrix of shape (d_model, vocab_size).

Tying saves 12.5M params at small quality cost. Most small models tie; many large models don’t.

Vocab size as a design trade-off

The trade-off: bigger vocab → fewer tokens per text but bigger embedding; smaller vocab → more tokens per text but more body capacity at the same total params.

For huge-base-v1, SmolLM2’s 49K vocab is structurally too big. 16K–30K would give the body more capacity. The 57% embedding share is a real architectural imbalance — one of the “could have been better” items.

Common pitfalls

• Numbers tokenize inconsistently. “1234” might be one token, two, or three — making arithmetic reasoning surprisingly hard for LLMs.
• Whitespace is sticky. " python" vs "python" are different tokens.
• Non-English handled poorly by English-trained BPE. Chinese characters become 2-3 tokens; English words become 1.
• Code has special needs. Indentation, brackets, keywords get special treatment in code-trained BPEs (StarCoder, Code Llama).
• Tokenizer mismatches break models. Train on tokenizer A, infer with tokenizer B → garbage. Vectors are indexed by token ID.

How d_model is decided (and where the body’s capacity comes from)

Designer choice. Common considerations:

• Param budget. total_params ≈ vocab × d_model + n_layers × const × d_model² (rough). For a fixed total, d_model trades against layers.
• Width vs depth. Bigger d_model → richer per-token representations; bigger n_layers → more compositional reasoning. Empirically the optimum is in the middle.
• Hardware constraints. d_model must be efficient for matmul (often a multiple of 64 or 128).
• Rough heuristic. d_model often grows roughly with the cube root of parameter count.

huge-base-v1: d_model=256, n_layers=12. Narrow for its depth. Deliberate small-teaching choice; “real” small models would use 384 or 512.

Self-check

Don’t peek:

1. Why is byte-level tokenization “free of unknown tokens”?
2. Why does the embedding layer use so many parameters in huge-base-v1? Give the math.
3. If you wanted to make huge-base-v1 better at the SAME parameter budget, what’s one change you could make to the tokenizer?
4. Trap question: if BPE produces fewer tokens per text, doesn’t that mean the model has LESS information to work with per sentence? Why isn’t this a problem?

Where this connects

• The 57% embedding share of huge-base-v1 (Session model dive) is a direct consequence of choosing SmolLM2’s 49K vocab for a 22M model.
• Embedding-vs-body capacity trade-off appears in every model design — see journal Entry 11.
• Next: training mechanics (the actual loop with batches, epochs, schedules) — or open.