AI · 13.1

Reinforcement Learning: Learning by Trial and Error

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 12 marks. Total 22 marks.

Name

Class

Date

Mark / 22

Section A

Circle the letter of the correct answer. One mark each.

1In reinforcement learning, what does the agent receive instead of correct answers?1
1. ARewards for the outcomes of its actions
2. BThe best action for each state
3. CA label for every state
4. DThe rules of the environment
2With γ = 0.9, an agent's next three rewards are 0, 0 and 10. What is its discounted return?1
1. A7.29
2. B8.1
3. C9
4. D10
3What does the value V(s) of a state mean?1
1. AThe reward for entering that state
2. BThe distance from the state to the start
3. CThe return the agent can expect from that state
4. DThe number of times the state was visited
4A corridor's goal pays 10 when entered and γ = 0.9. If the cell next to the goal has value 10, what is the value of the cell before it?1
1. A8.1
2. B9
3. C10
4. D19
5In Q-learning, Q(s, a) = 2.0, r = −1, γ = 0.9, the best Q in the next state is 6.0 and α = 0.5. What is the new Q(s, a)?1
1. A2.4
2. B3.2
3. C4.4
4. D5.0
6Why must an agent explore?1
1. AExploring always earns more reward
2. BExploring makes the values smaller
3. CThe environment forbids repeating an action
4. DIt may never discover better actions if it only exploits
7With ε-greedy and ε = 0.1, how often does the agent take a random action?1
1. AAbout 10% of the time
2. BAbout 90% of the time
3. COnly at the very start
4. DNever
8Why did DQN use a neural network instead of a Q-table?1
1. AThere were far too many possible screens for a table
2. BAtari games have no rewards
3. CTables cannot store negative numbers
4. DNeural networks never need to explore
9What does experience replay do in DQN?1
1. ACopies a human expert's moves
2. BRestarts the game after every move
3. CStores past moves and trains on random batches of them
4. DReplays the best game to the player
10An agent rewarded for points circles to hit the same targets instead of finishing the race. What is this?1
1. ADiscounting
2. BOverfitting
3. CExploration
4. DReward hacking

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

1ExplainExplain how Q-learning allows an agent to learn a good policy without knowing the rules of its environment.5
2ExplainExplain the exploration–exploitation trade-off, using the example of choosing between slot machines.3
3DiscussDiscuss the problem of reward hacking, with an example.4

Mark scheme

Section A

1A2B3C4B5B6D7A8A9C10D

Section B

1 Explain how Q-learning allows an agent to learn a good policy without knowing the rules of its environment. 5

One mark for each of the following, to a maximum of 5.

The agent keeps an estimate Q(s, a) of the return for each action in each state
It acts, and observes the reward and the next state
It computes a target: reward + γ × the best Q-value in the next state
It moves Q(s, a) a fraction α towards the target / by the temporal difference error
It explores, for example with ε-greedy, so all actions are tried
The policy is to take the action with the highest Q-value in each state

Model answer. The agent keeps an estimate Q(s, a) of the return it can expect from taking each action in each state, starting from zero. Each time it acts, it observes the reward and the state it lands in, and computes a target: the reward plus γ times the best Q-value available in the new state. It then moves its estimate a fraction α of the way towards that target, so every surprise improves the estimate, and rewards gradually spread back to the states and actions that led to them. To make sure it tries every action, it explores, for example taking a random action with probability ε. Once the estimates settle, the policy is simply to take the action with the highest Q-value in each state, and none of this needs a model of the rules.

2 Explain the exploration–exploitation trade-off, using the example of choosing between slot machines. 3

One mark for each of the following, to a maximum of 3.

Exploiting means choosing the machine that currently looks best
Exploring means trying other machines to learn their payouts
Too little exploration can miss a better machine; too much wastes pulls on bad machines
A strategy such as ε-greedy or UCB balances the two

Model answer. A player who always exploits picks the machine whose estimated payout is highest so far, but that estimate may be based on only a few pulls, so a better machine may never be discovered. A player who explores tries other machines to learn their payouts, but every pull on a poor machine wastes money. The trade-off is balancing the two, for example with ε-greedy, which picks a random machine a small fraction of the time and the best-looking one otherwise.

3 Discuss the problem of reward hacking, with an example. 4

One mark for each of the following, to a maximum of 4.

The agent maximises the reward exactly as written, not the designer's intention
Example: the boat that circled hitting targets instead of finishing the race
Real-world risk: e.g. recommender optimising watch time promoting harmful or outrageous content
It shows that designing rewards / aligning goals is hard and needs testing and oversight

Model answer. Reward hacking happens because a reinforcement learning agent maximises exactly the reward it is given, not what its designers meant by it. In the boat racing game CoastRunners, an agent rewarded for points discovered it could score more by circling a lagoon to hit the same reappearing targets, crashing and catching fire, than by finishing the race. In a game this is funny, but in the real world a recommendation system rewarded for watch time might learn to promote outrage or harmful content because it keeps people watching. It shows that rewards are very hard to specify completely, so systems need careful reward design, testing for unexpected strategies, and human oversight.