The Deep Learning Era: 2012 to Today
2012: the year everything changed
Every year from 2010 the ImageNet challenge asked programs to sort a million and more photographs into 1,000 categories. In 2012 a deep convolutional network called AlexNet, built by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, made a top-5 error of 15.3%. The next best entry made 26.2%.
The gap was so large that within two years almost every computer vision team had switched to deep learning. AlexNet had about 60 million learned numbers, called parameters, and was trained on two gaming graphics cards.
None of the three ingredients was new on its own. What was new was having all three at once:
| Ingredient | What changed |
|---|---|
| Data | The web, smartphones and projects like ImageNet produced labelled examples by the million |
| Compute | Graphics processing units (GPUs), built for games, could do thousands of multiplications at the same time |
| Algorithms | Better tricks: the ReLU activation, dropout, better ways to start training, and later attention |
Seeing, playing and understanding: 2013 to 2017
| Year | Breakthrough | Why it mattered |
|---|---|---|
| 2013 | word2vec | Turned words into lists of numbers where meaning becomes geometry: king minus man plus woman lands near queen |
| 2013 | DeepMind's Atari player | One network learned dozens of video games from the screen pixels and the score alone |
| 2014 | GANs | Two networks compete, one forging images and one spotting fakes, and both improve |
| 2014 | Attention | Let a translation network look back at the most relevant words of the input |
| 2015 | ResNet | Shortcut connections made networks 152 layers deep trainable, and beat the human error estimate on ImageNet |
| 2016 | AlphaGo | Beat Lee Sedol, one of the best Go players in the world, 4 games to 1 in Seoul |
| 2017 | The Transformer | The paper Attention Is All You Need introduced the architecture behind today's language models |
| 2017 | AlphaZero | Taught itself chess, shogi and Go from the rules alone, by playing itself |
In the second game against Lee Sedol, AlphaGo played a move on the fifth line that commentators first thought was a mistake. It turned out to be brilliant, and a move a human professional would almost never consider. It was the moment many people realised these systems could find ideas, not only copy them.
The language and generation era: 2018 to today
From models to products
- 2018: Google's BERT and OpenAI's first GPT showed that a model pretrained on huge amounts of text could then be adapted to many tasks.
- 2020: GPT-3, with 175 billion parameters, could carry out new tasks from a few examples written into its prompt.
- 2020: DeepMind's AlphaFold 2 predicted the 3D shapes of proteins with accuracy close to laboratory methods, solving a 50 year old problem in biology.
- 2022: Stable Diffusion released an image generator that runs on a home computer, and in November ChatGPT put a language model in a simple chat window. It became one of the fastest growing apps ever.
- 2023: GPT-4 accepted images as well as text, and openly released models spread quickly. In November the UK hosted the first AI Safety Summit at Bletchley Park, where 28 countries and the EU signed a declaration on the risks of advanced AI.
- 2024: The EU AI Act, the first broad law regulating AI by risk, entered into force. The Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton for foundational work on neural networks, and half of the Nobel Prize in Chemistry to Demis Hassabis and John Jumper for AlphaFold. Reasoning models, which work through a problem step by step before answering, appeared.
- 2025 and beyond: AI agents that plan, use tools and carry out tasks over many steps, and open reasoning models, became the new frontier.
Why it keeps accelerating
Compute
The computation used to train the most notable AI models has grown by roughly four to five times every year since 2010, according to tracking by the research group Epoch AI. Growth that fast is hard to picture, so the chart below shows what it means.
Scale, and the bitter lesson
Researchers kept finding that bigger models trained on more data with more compute improved predictably. The computer scientist Rich Sutton called this the bitter lesson: over decades, general methods that make use of more computation have beaten approaches built on human cleverness. It is bitter because so much clever hand-designed work was overtaken.
What has not changed
The core ideas in this course are old: neurons from 1943, learning from examples from 1958, backpropagation from 1986, convolution from 1989. What changed was scale, and a handful of new ideas, attention above all. Understand the ideas and you understand the machines, whatever they are called next year.
Hands-on: how fast is exponential?
Run the numbers
If training compute grows four times a year, how long does it take to double, and how much bigger is it after a decade?
import math
growth_per_year = 4 # four times more each year
for years in [1, 2, 5, 10]:
factor = growth_per_year ** years
print(f"After {years:2} years: {factor:>12,} times more compute")
doubling_years = math.log(2) / math.log(growth_per_year)
print(f"Doubling time: {doubling_years:.2f} years, about {doubling_years * 12:.0f} months")A million times more in ten years. Compare that with the famous Moore's law, under which the number of transistors on a chip doubled roughly every two years: change the code to growth_per_year = 2 ** 0.5 and see how much slower that is.
Activity: is it AI?
Sort these into not AI, rules (symbolic AI) and learned from data (machine learning), and give a reason for each: a pocket calculator, a central heating thermostat, an email spam filter, face unlock on a phone, a satnav finding the shortest route, a video app's recommendations, a chess app, a voice assistant.
Research task
Choose one milestone from this lesson. Write a 150 word explainer for a Year 7 student: what happened, why it was hard, and what it changed. Use at least two sources and list them.
Level check
- Explain why AlexNet in 2012 is called the start of the deep learning revolution
- Name the three ingredients behind deep learning and give an example of each
- Describe two breakthroughs from games and two from science
- Explain what the Transformer is, and when it appeared
- Describe how fast training compute has grown, and what the bitter lesson says
- Give two events that show governments responding to AI
Key terms you must be able to define
- Deep learning
- Machine learning with neural networks of many layers, which learn their own features from raw data.
- AlexNet
- The deep convolutional network that won the 2012 ImageNet challenge by a huge margin.
- GPU
- Graphics processing unit: a chip that performs thousands of calculations at once, ideal for training neural networks.
- Parameter
- A number inside a model, such as a weight, that is learned during training.
- Transformer
- The 2017 neural network architecture, built on attention, behind modern language models.
- Generative AI
- AI that creates new content, such as text, images, audio or video.
- Large language model
- A very large transformer trained to predict the next word on huge amounts of text.
- Bitter lesson
- Rich Sutton's observation that general methods using more computation have beaten hand-crafted cleverness.
- AI agent
- A system that plans and carries out multi-step tasks, often by using tools.
Flashcards
Go further: notes for future AI engineers
Scaling, measured
In 2020 researchers at OpenAI (Kaplan and colleagues) showed that a language model's error falls along a smooth curve, a power law, as its parameters, data and compute grow. In 2022 DeepMind's Chinchilla paper (Hoffmann and colleagues) refined the recipe: for a fixed compute budget, train a smaller model on more data, roughly 20 tokens of text per parameter. Level 11 uses these numbers to estimate what training a model costs.
Benchmarks saturate
Every era has a test that seemed hard and then fell: ImageNet between 2012 and 2015, the GLUE and SuperGLUE language tests around 2019, then exam-style and coding tests. When a benchmark saturates, researchers build a harder one. Treat headline scores with care: benchmark questions can leak onto the web and end up in training data, so a model may have seen the answers.
Misconceptions to avoid
- Thinking the 2012 breakthrough came from a brand new idea. Convolutional networks existed in 1989; data and GPUs were what was new.
- Believing AI progress is only about bigger models. New ideas such as attention were essential too.
- Treating ChatGPT as the first language model. It was a chat interface on top of years of GPT research.
- Confusing compute with data. They are separate ingredients, and both grew.
AlexNet, the network that won the 2012 ImageNet contest, was trained on two gaming graphics cards in a bedroom at the home of one of its creators. Gaming hardware turned out to be exactly what deep learning needed.
Knowledge check
Check you could explain this
- LAlexNet was a deep neural network that won ImageNet
- LIt won by a very large margin
- LIt combined large labelled data with GPUs
- LResearchers switched to deep learning as a result
AlexNet was a deep convolutional neural network that entered the 2012 ImageNet challenge and won it by a huge margin, with a top-5 error of 15.3% against 26.2% for the next best method. It showed that a large network, trained on millions of labelled images using graphics cards, could beat years of hand-designed approaches. The result was so clear that within about two years almost every computer vision group had switched to deep learning, and the same approach then spread to speech, language and games.
- LData
- LAn example of data
- LCompute
- LAn example of compute
- LAlgorithms
- LAn example of an algorithm
The first ingredient is data: deep networks learn from examples and need millions of them, which the web, smartphones and projects such as ImageNet, with over 14 million labelled images, supplied. The second is compute: graphics processing units, designed for video games, can perform thousands of multiplications at once, which is exactly what training a network needs, and AlexNet was trained on two of them. The third is algorithms: better methods such as the ReLU activation, dropout, residual connections and later attention and the Transformer made deep networks trainable and far more capable.
- LScale: compute grew around four to five times a year
- LScale: bigger models with more data improved predictably, the bitter lesson
- LIdeas: the Transformer or attention were essential
- LIdeas: other techniques such as ReLU, dropout or residual connections
- LOld ideas only worked once scale arrived
- LA judgement weighing both
Scale has clearly been a huge driver. Training compute for notable models has grown about four to five times a year since 2010, and researchers repeatedly found that bigger models trained on more data improved predictably, which Rich Sutton called the bitter lesson. Many of the core ideas, such as backpropagation from 1986 and convolutional networks from 1989, only became powerful once that scale arrived. But scale alone was not enough: new ideas were essential, above all attention and the Transformer in 2017, along with techniques such as ReLU, dropout and residual connections that made very deep networks trainable at all. Overall the two worked together: new ideas made it possible to use more compute well, and more compute revealed what those ideas could do.