Tokens and Embeddings: Turning Words into Meaningful Numbers

From text to tokens

A model cannot read letters. Text is first split into tokens, and each token is given an ID number from a fixed vocabulary.

Modern language models use subword tokens, with vocabularies of roughly 30,000 to 200,000 tokens. A rule of thumb for English: a token is about three quarters of a word, so 100 tokens is roughly 75 words.

Byte pair encoding

The most common way to build a subword vocabulary is byte pair encoding (BPE), an old data compression trick adapted for neural translation in 2015.

1. Start with every word split into single characters.
2. Count every pair of neighbouring symbols across the whole corpus.
3. Merge the most frequent pair into a new symbol, and add it to the vocabulary.
4. Repeat until the vocabulary reaches the size you want.

On a tiny corpus of "low" ×5, "lower" ×2, "newest" ×6 and "widest" ×3:

After a few merges, frequent endings like "est" and whole frequent words like "low" are single tokens, while a word the tokenizer has never seen can still be built from pieces. The hands-on section runs this algorithm.

Why one-hot is not enough

Level 3 turned categories into one-hot vectors. For a 50,000-token vocabulary, each token becomes a list of 50,000 numbers with a single 1.

That has two fatal problems:

• Size: almost every number is zero, which wastes memory and compute.
• No meaning: every pair of one-hot vectors is exactly the same distance apart. "Cat" is as far from "kitten" as from "volcano", so nothing the model learns about cats helps it with kittens.

We want vectors where similar meanings are close together.

Embeddings: meaning as geometry

An embedding gives each token a dense vector of a few hundred or thousand learned numbers: 768 in the original BERT, 12,288 in GPT-3. The vectors are stored as the rows of an embedding matrix, and looking up a token simply means reading its row.

In a good embedding space:

• Words with similar meanings sit close together: cat near kitten near dog.
• Groups form: animals, foods, vehicles, places.
• Directions carry meaning: the step from "man" to "woman" is roughly the same as the step from "king" to "queen".

Real embeddings have hundreds of dimensions, which no one can picture. The 3D views here use a few hand-placed words to show the idea; researchers squash real embeddings down to 2D or 3D with methods such as t-SNE or UMAP to look at them.

Turn the space in the lab, pick a word to see its nearest neighbours, and try your own analogies.

Measuring similarity: cosine

How close are two embeddings? The standard measure is the cosine similarity: the cosine of the angle between the two vectors.

Dividing by the lengths means only the direction counts, not how long the vectors are. This is how search engines, recommendation systems and the retrieval step of AI assistants (Level 11) find the closest matches among millions of items.

Vector arithmetic and analogies

In 2013 Tomas Mikolov and colleagues at Google released word2vec and showed a striking property: you can do arithmetic with meaning.

The difference vector from "man" to "woman" captures something like a gender direction. Adding it to "king" lands near "queen". Similar patterns appear for capital cities (Paris − France + Italy ≈ Rome) and verb tenses (walking − walk + swim ≈ swimming).

How embeddings are learned

Embeddings rest on the distributional hypothesis: words that appear in similar contexts have similar meanings. "I fed the ___" might be followed by cat, dog or kitten, so those words should end up close together.

word2vec turned that into a training task:

• Skip-gram: given a word, predict the words around it.
• CBOW (continuous bag of words): given the surrounding words, predict the word in the middle.

The embeddings are the weights that make these predictions work. No one labels any meanings: the structure emerges from billions of words of ordinary text.

In a modern language model the embedding matrix is simply the first layer, learned along with every other parameter while the model learns to predict the next token. Later layers then refine each token's vector using its context, so that "bank" ends up with a different vector in "river bank" and "bank account". That is the job of attention, in Level 10.

Hands-on: tokens and embeddings in code

Byte pair encoding

from collections import Counter

corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
words = {tuple(w): n for w, n in corpus.items()}       # start from characters

for step in range(1, 7):
    pairs = Counter()
    for symbols, n in words.items():
        for a, b in zip(symbols, symbols[1:]):
            pairs[(a, b)] += n
    best = max(pairs, key=pairs.get)
    merged = {}
    for symbols, n in words.items():
        out, i = [], 0
        while i < len(symbols):
            if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == best:
                out.append(symbols[i] + symbols[i + 1])
                i += 2
            else:
                out.append(symbols[i])
                i += 1
        merged[tuple(out)] = n
    words = merged
    print(f"merge {step}: {best[0]} + {best[1]} ({pairs[best]} times) ->",
          ", ".join(" ".join(s) for s in words))

Cosine similarity and analogies

These embeddings are made by hand with four meaningful dimensions, royal, female, young and animal, so you can see why the arithmetic works. Learned embeddings have hundreds of dimensions with no labels.

import math

emb = {
    "king":   [0.95, 0.05, 0.1, 0.0],  "queen":    [0.95, 0.95, 0.1, 0.0],
    "prince": [0.85, 0.05, 0.6, 0.0],  "princess": [0.85, 0.95, 0.6, 0.0],
    "man":    [0.10, 0.05, 0.1, 0.0],  "woman":    [0.10, 0.95, 0.1, 0.0],
    "boy":    [0.05, 0.05, 0.85, 0.0], "girl":     [0.05, 0.95, 0.85, 0.0],
    "cat":    [0.0, 0.5, 0.2, 0.95],   "kitten":   [0.0, 0.5, 0.9, 0.95],
    "dog":    [0.0, 0.5, 0.2, 0.9],    "puppy":    [0.0, 0.5, 0.9, 0.9],
}

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

def nearest(vector, exclude=()):
    scores = [(cosine(vector, v), w) for w, v in emb.items() if w not in exclude]
    return sorted(scores, reverse=True)[:3]

print("closest to cat:", [(w, round(s, 3)) for s, w in nearest(emb["cat"], {"cat"})])

def analogy(a, b, c):
    # a - b + c
    v = [x - y + z for x, y, z in zip(emb[a], emb[b], emb[c])]
    return nearest(v, {a, b, c})[0][1]

print("king - man + woman =", analogy("king", "man", "woman"))
print("kitten - cat + dog =", analogy("kitten", "cat", "dog"))
print("prince - boy + girl =", analogy("prince", "boy", "girl"))

Try this

1. Add "lion" and "cub" to the embeddings, and check that cub − lion + cat gives kitten.
2. Run more BPE merges. Which whole words become single tokens first, and why those?

Level check

• Compare character, word and subword tokenisation and explain why subwords are used
• Carry out two steps of byte pair encoding on a small corpus
• Explain two problems with one-hot vectors for words
• Describe what an embedding is and what nearness and direction mean in an embedding space
• Calculate a cosine similarity and interpret it
• Explain how word2vec learns embeddings, and why embeddings can contain bias

Key terms you must be able to define

Token

A piece of text, often part of a word, that a language model treats as one unit.

Vocabulary

The fixed set of tokens a model knows, each with an ID number.

Tokeniser

The program that splits text into tokens and turns them into IDs.

Byte pair encoding (BPE)

Building a vocabulary by repeatedly merging the most frequent pair of neighbouring symbols.

Embedding

A learned dense vector representing a token, placed so that similar meanings are close.

Embedding matrix

The table of every token's embedding, one row per token.

Cosine similarity

The cosine of the angle between two vectors: 1 for the same direction, 0 for unrelated.

Distributional hypothesis

Words that appear in similar contexts have similar meanings.

word2vec

A 2013 method that learns word embeddings by predicting words from their neighbours.

Contextual embedding

A token's vector after the model has adjusted it for the words around it.

Flashcards

Go further: notes for future AI engineers

The embedding layer is just a matrix multiply

Looking up row i of the embedding matrix E is exactly the same as multiplying a one-hot vector for token i by E. So an embedding layer is a dense layer with no bias and no activation, applied to one-hot input, implemented as a fast lookup. Many language models also tie their weights, reusing the same matrix (transposed) in the final layer that turns hidden vectors back into scores for every token, which saves a large share of the parameters in small models.

Sentence and document embeddings

The same idea extends from words to whole sentences, paragraphs and images. Models are trained so that texts with similar meanings get nearby vectors, regardless of wording: "How do I reset my password?" lands near "I forgot my login details". These sentence embeddings power semantic search and retrieval-augmented generation (Level 11), and models such as CLIP (Level 12) put images and text in the same space, so a picture of a dog lands near the words "a photo of a dog".

Misconceptions to avoid