MskProdComputing
Back to the topic

AI · 6.2

Activation Functions: The Bend That Makes Networks Powerful

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 12 marks. Total 22 marks.

Name

Class

Date

Mark  / 22

Section A

Circle the letter of the correct answer. One mark each.

  1. 1What happens if a deep network has no activation functions?1
    1. AAll its layers collapse into one linear function
    2. BIts outputs are always exactly zero
    3. CIt becomes able to learn any function at all
    4. DIt trains much faster and more accurately
  2. 2What is ReLU(−3)?1
    1. A−3
    2. B0.05
    3. C0
    4. D3
  3. 3What is the output range of tanh?1
    1. A0 to infinity
    2. B−1 to 1
    3. C0 to 1
    4. D−infinity to infinity
  4. 4Why did ReLU help deep networks train?1
    1. AIt outputs probabilities that always add up to 1
    2. BIts slope is 1 for positive inputs, so gradients do not shrink
    3. CIt uses exponentials, which are very fast to compute
    4. DIt stops any neuron from ever outputting zero
  5. 5What is a dead ReLU neuron?1
    1. AOne that has been deleted from the network on purpose
    2. BOne that outputs the same positive value for every input
    3. COne whose weights have all been set to exactly one
    4. DOne whose input is negative for every example, so it never learns
  6. 6What is the largest possible slope of the sigmoid function?1
    1. A0.25
    2. B0.5
    3. C1
    4. D4
  7. 7A gradient passes back through 5 sigmoid layers, each multiplying it by 0.25. What fraction is left?1
    1. A1.25
    2. BAbout 0.001
    3. CAbout 0.1
    4. D0.25
  8. 8Which activation is usual for the output layer of a classifier choosing one class out of ten?1
    1. ASoftmax
    2. BTanh
    3. CNo activation
    4. DReLU
  9. 9Softmax is applied to logits [3, 1, 1]. Which statement is true?1
    1. AAll three probabilities are exactly equal to each other
    2. BThe probabilities add up to 5, the sum of the logits
    3. CThe second and third probabilities are both zero
    4. DThe first probability is the largest and all three add up to 1
  10. 10What does a high softmax temperature, such as T = 5, do?1
    1. AReverses the order of the classes
    2. BSharpens them so the top choice gets nearly all of it
    3. CFlattens the probabilities towards an even spread
    4. DMakes the probabilities add up to 5

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainExplain why neural networks need non-linear activation functions.4
  2. 2CompareCompare the sigmoid and ReLU activation functions for use in the hidden layers of a deep network.5
  3. 3CalculateA classifier outputs the logits [2, 0, 0] for three classes. Calculate the softmax probability of the first class, given that e² ≈ 7.39.3

Mark scheme

Section A

1A2C3B4B5D6A7B8A9D10C

Section B

1 Explain why neural networks need non-linear activation functions. 4

One mark for each of the following, to a maximum of 4.

  • Without them each layer is a linear function / matrix multiply
  • Stacked linear layers collapse into a single linear layer
  • So depth adds nothing / can only model straight lines or planes
  • Non-linear activations add bends that combine into curves / any function

Model answer. Without an activation function, each layer only multiplies its inputs by a matrix and adds a bias, which is a linear function. Stacking linear functions gives another linear function, so however many layers the network had it would collapse to a single linear layer, and depth would add nothing: it could only ever model straight lines and flat planes. A non-linear activation puts a bend into every neuron, and many bends can be combined into curves that can approximate almost any function.

2 Compare the sigmoid and ReLU activation functions for use in the hidden layers of a deep network. 5

One mark for each of the following, to a maximum of 5.

  • Sigmoid squashes outputs into 0 to 1
  • Sigmoid's slope is at most 0.25 / saturates for large inputs
  • So gradients vanish through many sigmoid layers
  • ReLU outputs max(0, z) with slope 1 for positive inputs
  • ReLU is cheap but can produce dead neurons

Model answer. The sigmoid squashes its input into the range 0 to 1, but its slope is never more than 0.25 and is almost zero for large positive or negative inputs, where it saturates. In a deep network the gradients are multiplied by these small slopes layer after layer, so they vanish and the early layers barely learn. ReLU outputs max(0, z), whose slope is exactly 1 for any positive input, so gradients pass back without shrinking, and it is very cheap to compute because it needs no exponential. Its drawback is that a neuron whose input is always negative outputs 0 with zero slope and dies. Overall ReLU is much better for hidden layers, which is why it is the default.

3 A classifier outputs the logits [2, 0, 0] for three classes. Calculate the softmax probability of the first class, given that e² ≈ 7.39. 3

One mark for each of the following, to a maximum of 3.

  • e⁰ = 1 for the other two classes
  • 7.39 ÷ (7.39 + 1 + 1) = 7.39 ÷ 9.39
  • ≈ 0.79

Model answer. e⁰ = 1, so the sum of the exponentials is 7.39 + 1 + 1 = 9.39. The probability of the first class is 7.39 ÷ 9.39 ≈ 0.79, and the other two classes get about 0.11 each.