MskProdComputing
Back to the topic

AI · 1.2

The Deep Learning Era: 2012 to Today

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 16 marks. Total 26 marks.

Name

Class

Date

Mark  / 26

Section A

Circle the letter of the correct answer. One mark each.

  1. 1What happened at the 2012 ImageNet challenge?1
    1. AThe contest was cancelled
    2. BAlexNet, a deep network, won by a huge margin
    3. CChatGPT was released
    4. DA human beat every computer
  2. 2Which three ingredients came together to start the deep learning era?1
    1. AData, compute and algorithms
    2. BRules, logic and search
    3. CKeyboards, mice and screens
    4. DRobots, sensors and batteries
  3. 3Why are GPUs good for training neural networks?1
    1. AThey connect to the internet faster
    2. BThey do thousands of multiplications at the same time
    3. CThey need no electricity
    4. DThey store more files
  4. 4What did AlphaGo do in 2016?1
    1. ATranslated every language
    2. BWon at chess against Kasparov
    3. CBeat Lee Sedol at Go, 4 games to 1
    4. DPredicted protein structures
  5. 5Which 2017 paper introduced the Transformer?1
    1. AComputing Machinery and Intelligence
    2. BPerceptrons
    3. CAttention Is All You Need
    4. DLearning Representations by Back-propagating Errors
  6. 6What did AlphaFold 2 achieve in 2020?1
    1. APassed the Turing test in front of a panel of judges
    2. BGenerated realistic photographs from short text prompts
    3. CBeat the best human players at dozens of video games
    4. DPredicted protein structures with near laboratory accuracy
  7. 7Roughly how fast has training compute for notable models grown since 2010?1
    1. AAbout four to five times every year
    2. BAbout 1% a year
    3. CIt halves each year
    4. DIt has stayed the same
  8. 8What does the 'bitter lesson' say?1
    1. AHand-written rules always beat learning in the end
    2. BGeneral methods that use more computation win over time
    3. CSmall, carefully designed models are always better
    4. DAI progress always stalls once data runs out
  9. 9Who won the 2024 Nobel Prize in Physics for work on neural networks?1
    1. AYann LeCun and Fei-Fei Li
    2. BAlan Turing and Ada Lovelace
    3. CMarvin Minsky and Seymour Papert
    4. DJohn Hopfield and Geoffrey Hinton
  10. 10What happened at Bletchley Park in November 2023?1
    1. AThe EU AI Act was signed into law by member states
    2. BA new national computing museum opened to visitors
    3. CThe first AI Safety Summit, with a declaration on AI risks
    4. DChatGPT was launched to the public for the first time

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainExplain why AlexNet's win in 2012 is seen as the start of the deep learning revolution.4
  2. 2DescribeDescribe the three ingredients that drove the deep learning era, with an example of each.6
  3. 3DiscussDiscuss whether the growth of AI since 2012 has been driven more by scale or by new ideas.6

Mark scheme

Section A

1B2A3B4C5C6D7A8B9D10C

Section B

1 Explain why AlexNet's win in 2012 is seen as the start of the deep learning revolution. 4

One mark for each of the following, to a maximum of 4.

  • AlexNet was a deep neural network that won ImageNet
  • It won by a very large margin
  • It combined large labelled data with GPUs
  • Researchers switched to deep learning as a result

Model answer. AlexNet was a deep convolutional neural network that entered the 2012 ImageNet challenge and won it by a huge margin, with a top-5 error of 15.3% against 26.2% for the next best method. It showed that a large network, trained on millions of labelled images using graphics cards, could beat years of hand-designed approaches. The result was so clear that within about two years almost every computer vision group had switched to deep learning, and the same approach then spread to speech, language and games.

2 Describe the three ingredients that drove the deep learning era, with an example of each. 6

One mark for each of the following, to a maximum of 6.

  • Data
  • An example of data
  • Compute
  • An example of compute
  • Algorithms
  • An example of an algorithm

Model answer. The first ingredient is data: deep networks learn from examples and need millions of them, which the web, smartphones and projects such as ImageNet, with over 14 million labelled images, supplied. The second is compute: graphics processing units, designed for video games, can perform thousands of multiplications at once, which is exactly what training a network needs, and AlexNet was trained on two of them. The third is algorithms: better methods such as the ReLU activation, dropout, residual connections and later attention and the Transformer made deep networks trainable and far more capable.

3 Discuss whether the growth of AI since 2012 has been driven more by scale or by new ideas. 6

One mark for each of the following, to a maximum of 6.

  • Scale: compute grew around four to five times a year
  • Scale: bigger models with more data improved predictably, the bitter lesson
  • Ideas: the Transformer or attention were essential
  • Ideas: other techniques such as ReLU, dropout or residual connections
  • Old ideas only worked once scale arrived
  • A judgement weighing both

Model answer. Scale has clearly been a huge driver. Training compute for notable models has grown about four to five times a year since 2010, and researchers repeatedly found that bigger models trained on more data improved predictably, which Rich Sutton called the bitter lesson. Many of the core ideas, such as backpropagation from 1986 and convolutional networks from 1989, only became powerful once that scale arrived. But scale alone was not enough: new ideas were essential, above all attention and the Transformer in 2017, along with techniques such as ReLU, dropout and residual connections that made very deep networks trainable at all. Overall the two worked together: new ideas made it possible to use more compute well, and more compute revealed what those ideas could do.