MskProdComputing
Back to the topic

AI · 7.1

Neural Networks: Layers, Matrices and the Forward Pass

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 12 marks. Total 22 marks.

Name

Class

Date

Mark  / 22

Section A

Circle the letter of the correct answer. One mark each.

  1. 1What is a dense layer?1
    1. AA layer that only connects each neuron to one input
    2. BA layer where every neuron receives every input from the previous layer
    3. CA layer whose weights are all fixed and never trained
    4. DA layer with more neurons than any other layer in the network
  2. 2A layer has 5 inputs and 3 neurons. What shape is its weight matrix?1
    1. A3 × 3
    2. B5 × 5
    3. C3 × 5
    4. D5 × 3
  3. 3How is a matrix multiplied by a vector?1
    1. AEach entry of the matrix is added to the vector
    2. BThe matrix is turned into a vector first
    3. COnly the first row is used
    4. DEach row of the matrix is dot-producted with the vector
  4. 4In the XOR network, h₁ = ReLU(x₁ + x₂) and h₂ = ReLU(x₁ + x₂ − 1). What are h₁ and h₂ for input (1, 1)?1
    1. A2 and 1
    2. B1 and 0
    3. C1 and 1
    4. D2 and 0
  5. 5How does a hidden layer let a network solve XOR?1
    1. AIt changes the labels of two of the points
    2. BIt moves the points to a new space where one line separates them
    3. CIt removes the points that are hard to classify
    4. DIt adds more training examples to the data
  6. 6How many parameters does a dense layer from 10 inputs to 4 neurons have?1
    1. A4
    2. B14
    3. C40
    4. D44
  7. 7Why are GPUs good at running neural networks?1
    1. AGPUs can run Python code faster than processors can
    2. BGPUs have more memory than any other kind of chip
    3. CLayers are matrix multiplications, which GPUs do massively in parallel
    4. DGPUs store the training data permanently
  8. 8What do the early layers of an image network typically learn?1
    1. ASimple features such as edges and colours
    2. BNothing until the last layer has trained
    3. CWhole objects such as faces
    4. DThe labels of the training images
  9. 9What does 'deep' mean in deep learning?1
    1. AThe network uses a lot of data
    2. BThe network has many hidden layers
    3. CThe network understands meaning deeply
    4. DThe network is trained for a long time
  10. 10A model has 1 million parameters stored as 32-bit numbers. Roughly how much memory is that?1
    1. A4 MB
    2. B1 MB
    3. C4 GB
    4. D32 MB

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainExplain why a layer of a neural network can be written as a single matrix multiplication.4
  2. 2CalculateA network has 4 inputs, one hidden layer of 5 neurons and 2 outputs. Calculate its total number of parameters, showing your working.4
  3. 3ExplainExplain how the hidden layers of a deep image classifier build up an understanding of an image.4

Mark scheme

Section A

1B2C3D4A5B6D7C8A9B10A

Section B

1 Explain why a layer of a neural network can be written as a single matrix multiplication. 4

One mark for each of the following, to a maximum of 4.

  • Each neuron computes a weighted sum of the same inputs
  • Each neuron's weights form a row of a matrix
  • Multiplying the matrix by the input vector gives every weighted sum at once
  • Then the bias is added and the activation applied, f(Wx + b)

Model answer. Every neuron in a layer computes a weighted sum of the same inputs, each with its own weights. If each neuron's weights are written as one row of a matrix W, then multiplying W by the input vector x takes the dot product of every row with x, giving all of the neurons' weighted sums in one operation. Adding the bias vector and applying the activation function gives the whole layer's output, f(Wx + b).

2 A network has 4 inputs, one hidden layer of 5 neurons and 2 outputs. Calculate its total number of parameters, showing your working. 4

One mark for each of the following, to a maximum of 4.

  • 4 × 5 = 20 weights into the hidden layer
  • 5 hidden biases, so 25 for the first layer
  • 5 × 2 = 10 weights and 2 biases, so 12 for the second layer
  • Total 37

Model answer. The first layer has 4 × 5 = 20 weights and 5 biases, which is 25 parameters. The second layer has 5 × 2 = 10 weights and 2 biases, which is 12 parameters. The total is 25 + 12 = 37 parameters.

3 Explain how the hidden layers of a deep image classifier build up an understanding of an image. 4

One mark for each of the following, to a maximum of 4.

  • Early layers detect simple features such as edges or colours
  • Later layers combine these into textures, parts or shapes
  • The final layers represent whole objects or classes
  • These features are learned in training, not programmed

Model answer. Each layer re-represents the image for the next. The early layers learn to detect simple features such as edges, lines and patches of colour. Later layers combine those into more complex features such as textures, corners and parts of objects, like an eye or a wheel. The final layers combine parts into whole objects, so the output layer can decide between classes such as cat or aeroplane. None of these features is programmed by a person: they are learned automatically during training.