MskProdComputing
Back to the topic

AI · 9.1

Tokens and Embeddings: Turning Words into Meaningful Numbers

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 11 marks. Total 21 marks.

Name

Class

Date

Mark  / 21

Section A

Circle the letter of the correct answer. One mark each.

  1. 1Why do modern language models use subword tokens?1
    1. AThey remove the need for a vocabulary entirely
    2. BSubwords are always shorter than single characters
    3. CCommon words stay whole while rare words split into known pieces
    4. DEach subword token is exactly one letter long
  2. 2Roughly how many English words is 400 tokens?1
    1. AAbout 400
    2. BAbout 100
    3. CAbout 1,600
    4. DAbout 300
  3. 3In byte pair encoding, what is merged at each step?1
    1. AEvery pair of vowels in every word
    2. BThe most frequent pair of neighbouring symbols
    3. CThe two longest words in the corpus
    4. DThe two least common letters in the corpus
  4. 4What is wrong with one-hot vectors for words?1
    1. AThey can only represent a handful of different words
    2. BThey change every time the model is used
    3. CEvery word is equally far from every other, so meaning is lost
    4. DThey contain negative numbers that confuse the model
  5. 5In a good embedding space, where is 'kitten'?1
    1. AEqually far from every word
    2. BAt the origin
    3. CFar from all animals
    4. DClose to 'cat'
  6. 6Two vectors point in exactly the same direction. What is their cosine similarity?1
    1. A−1
    2. B1
    3. C0
    4. DIt depends on their lengths
  7. 7Two vectors are a = [1, 0] and b = [0, 1]. What is their cosine similarity?1
    1. A−1
    2. B1
    3. C0
    4. D0.5
  8. 8What does vector('king') − vector('man') + vector('woman') land closest to in word2vec?1
    1. Avector('queen')
    2. Bvector('kingdom')
    3. Cvector('prince')
    4. Dvector('man')
  9. 9What is the distributional hypothesis?1
    1. AEmbeddings should be spread evenly through the space
    2. BEvery word should appear equally often in the training data
    3. CLonger words carry more meaning than short words
    4. DWords that appear in similar contexts have similar meanings
  10. 10Why can word embeddings contain stereotypes?1
    1. AThey are learned from human writing that contains those stereotypes
    2. BEmbeddings are copied from dictionaries
    3. CThe maths of cosine similarity creates them
    4. DProgrammers deliberately add stereotypes to every model

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainExplain what a word embedding is and why it is more useful than a one-hot vector.4
  2. 2CalculateCalculate the cosine similarity of a = [3, 4] and b = [4, 3].3
  3. 3DescribeDescribe how byte pair encoding builds a vocabulary of tokens.4

Mark scheme

Section A

1C2D3B4C5D6B7C8A9D10A

Section B

1 Explain what a word embedding is and why it is more useful than a one-hot vector. 4

One mark for each of the following, to a maximum of 4.

  • An embedding is a dense vector of learned numbers for each word / token
  • Similar meanings have nearby vectors
  • One-hot vectors are all equally distant / carry no meaning
  • One-hot vectors are huge and sparse; embeddings are compact

Model answer. A word embedding represents each word, or token, as a dense vector of a few hundred learned numbers, positioned so that words with similar meanings have nearby vectors and directions in the space carry meaning. A one-hot vector, by contrast, has one position per word in the vocabulary, so it is huge and almost entirely zeros, and every pair of one-hot vectors is exactly the same distance apart, so it carries no information about meaning. With embeddings, what a model learns about one word helps it with similar words.

2 Calculate the cosine similarity of a = [3, 4] and b = [4, 3]. 3

One mark for each of the following, to a maximum of 3.

  • Dot product 3 × 4 + 4 × 3 = 24
  • Lengths are both 5
  • 24 ÷ 25 = 0.96

Model answer. The dot product is 3 × 4 + 4 × 3 = 24. Each vector has length √(9 + 16) = 5. So the cosine similarity is 24 ÷ (5 × 5) = 24 ÷ 25 = 0.96: the vectors point in very similar directions.

3 Describe how byte pair encoding builds a vocabulary of tokens. 4

One mark for each of the following, to a maximum of 4.

  • Start from individual characters
  • Count pairs of neighbouring symbols across the corpus
  • Merge the most frequent pair into a new token
  • Repeat until the vocabulary reaches its target size

Model answer. Byte pair encoding starts with every word in the training text split into individual characters. It counts how often each pair of neighbouring symbols occurs across the whole corpus, then merges the most frequent pair into a single new symbol, which is added to the vocabulary. This is repeated, so frequent pieces grow into longer tokens and common words become single tokens, until the vocabulary reaches the chosen size.