How Machines Learn: Supervised, Unsupervised and Reinforcement

What it means for a machine to learn

In 1997 the computer scientist Tom Mitchell gave the definition still used in textbooks:

Model, parameters, training and inference

A model is a function with adjustable numbers inside it, called parameters or weights. Training means adjusting those numbers until the model's outputs match the examples well. Inference means using the trained model on new inputs.

Training is slow and expensive and happens once, or now and then. Inference is quick and cheap and happens every time someone uses the model. A large language model may take months to train and then answer millions of questions a day.

Supervised learning: learning from labelled examples

In supervised learning, every training example has an input and the correct output, called the label. The model learns to map inputs to labels.

The inputs are described by features: the measurable properties the model can use.

• Classification predicts a category.
• Regression predicts a number.

Supervised learning is the workhorse of industry, but labels are expensive: someone has to mark every example. ImageNet's 14 million labels took tens of thousands of crowd workers years to produce.

Unsupervised learning: finding structure without answers

In unsupervised learning there are no labels. The algorithm looks for structure in the inputs alone.

The catch: nobody tells the algorithm what a good answer is, so a person has to decide whether the structure it found is meaningful. Clustering will always find clusters, even in random noise.

Reinforcement learning: learning from rewards

In reinforcement learning (RL) an agent takes actions in an environment and receives rewards. Nobody tells it the right action. It learns by trial and error which actions lead to the most reward over time.

Two things make RL hard. The reward is often delayed: a Go game lasts about 200 moves and the reward comes only at the end, so which moves deserved the credit? And the agent must balance exploring new actions against exploiting the ones it already knows work. Level 13 builds a reinforcement learner from scratch.

Self-supervised learning: the trick behind modern AI

Labels are expensive. Rewards are slow. So how did language models learn from trillions of words?

Self-supervised learning creates labels from the data itself. Hide part of the input and train the model to predict the hidden part. The hidden part is the label, and it costs nothing.

• Next-word prediction: "The cat sat on the ..." The label is the word that really came next. This is how GPT-style models are trained.
• Masked words: "The cat [MASK] on the mat." Predict the missing word. This is how BERT was trained.
• Images: hide patches of a photo and predict them, or predict whether two crops come from the same photo.

To predict the next word well across all of the internet, a model has to pick up grammar, facts, styles of writing and some reasoning. Prediction turns out to be a remarkably powerful teacher.

Two more terms you will meet

• Transfer learning: start from a model trained on a huge general task, then train it a little more on your small specific task. Almost all practical AI works this way now.
• Semi-supervised learning: a few labelled examples plus many unlabelled ones.

Hands-on: your first learning program

Learning a threshold

Twelve students recorded their hours of revision and whether they passed. The program tries every possible threshold and learns the one that best separates pass from fail. That search is training, in miniature.

# (hours of revision, passed: 1 = yes, 0 = no)
data = [(1, 0), (2, 0), (2.5, 0), (3, 0), (3.5, 0), (3, 1),
        (4, 1), (4.5, 1), (5, 1), (6, 1), (7, 1), (8, 1)]

def accuracy(threshold):
    correct = 0
    for hours, passed in data:
        prediction = 1 if hours >= threshold else 0
        correct += prediction == passed
    return correct / len(data)

candidates = [t / 2 for t in range(0, 21)]      # 0, 0.5, 1, ... 10
for t in candidates[2:11]:
    print(f"threshold {t:>4}: accuracy {accuracy(t):.0%}")

best = max(candidates, key=accuracy)
print("Learned rule: predict a pass if hours >=", best)
print("Prediction for 3.8 hours:", "pass" if 3.8 >= best else "fail")

The learned threshold is one parameter. A large language model has billions of them, but the principle is the same: search for the parameter values that make the fewest mistakes on the examples.

Learning by nearest neighbour

Another way to learn is simply to remember the examples and, for a new input, copy the label of the most similar one.

import math

# (weight in grams, skin smoothness 1 to 10), label
fruit = [((150, 9), "apple"), ((170, 8), "apple"), ((140, 9), "apple"),
         ((130, 3), "orange"), ((160, 2), "orange"), ((145, 4), "orange")]

def nearest(item):
    best = min(fruit, key=lambda f: math.dist(f[0], item))
    return best[1]

for mystery in [(155, 8), (150, 3), (138, 7)]:
    print(mystery, "is probably an", nearest(mystery))

This is the nearest neighbour classifier. Level 4 improves it by letting several neighbours vote, and shows why the features need to be on similar scales.

Activity: which kind of learning?

Decide whether each is supervised, unsupervised, reinforcement or self-supervised learning, and say what the labels or rewards are: predicting exam grades from past records; grouping songs by sound; a drone learning to hover; predicting the next frame of a video; sorting X-rays labelled by doctors; finding unusual network traffic; training a model to fill in missing words.

Level check

• State Mitchell's definition of learning and apply it to a new example
• Explain the difference between a model, its parameters, training and inference
• Describe supervised learning and the difference between classification and regression
• Describe unsupervised learning, with two techniques and examples
• Describe reinforcement learning and explain why delayed rewards make it hard
• Explain how self-supervised learning creates its own labels

Key terms you must be able to define

Training

Adjusting a model's parameters until its outputs match the training examples well.

Inference

Using a trained model to produce outputs for new inputs.

Parameter (weight)

One of the adjustable numbers inside a model that training sets.

Feature

A measurable property of an example that the model uses as input.

Label

The correct output for a training example in supervised learning.

Supervised learning

Learning from examples that include the correct answers.

Classification

Predicting which category an input belongs to.

Regression

Predicting a number.

Unsupervised learning

Finding structure, such as clusters, in data that has no labels.

Reinforcement learning

An agent learning which actions earn the most reward by trial and error.

Self-supervised learning

Learning by hiding part of the data and predicting it, so the data provides its own labels.

Transfer learning

Starting from a model trained on a large general task and adapting it to a specific one.

Flashcards

Go further: notes for future AI engineers

Inductive bias: why every learner makes assumptions

No learner can generalise without assumptions. The threshold learner assumes one cut-off separates the classes. Nearest neighbour assumes similar inputs have similar labels. A convolutional network assumes nearby pixels matter together. These built-in assumptions are called inductive bias, and choosing a model is choosing its assumptions.

The no free lunch theorem (Wolpert, 1996) makes this precise: averaged over every possible problem, no learning algorithm beats any other. A method wins only because its assumptions fit the problems we actually have. That is why engineers try several models, and why understanding the assumptions matters more than memorising algorithm names.

Misconceptions to avoid