deep rl

How does the n-step TD approach differ from TD(0)?

• TD(0) is only used for deterministic policies, while n-step TD is for stochastic policies.

• n-step TD randomly selects how many steps to wait before an update

• n-step TD updates only at the end of the episode, just like Monte Carlo

• TD(0) is an on-policy method while n-step TD is off-policy.

• n-step TD uses longer traces of rewards and states before performing a single update, rather than a one-step lookahead.

Which key feature distinguishes Temporal Difference (TD) learning from Monte Carlo methods?

• TD requires access to the full model of the environment’s transition probabilities.

• TD is only applicable to deterministic policies.

• TD waits until the end of an episode to update value estimates.

• TD uses bootstrapping from current estimates rather than waiting for the final outcome.

• TD cannot update its estimates online.

In DQN, what is the key purpose of the Experience Replay Buffer?

• It amplifies the most recent transition repeatedly to speed up learning

• It only stores states without actions or rewards

• It replaces the need for a target network

• It ensures that all experiences are used exactly once to avoid correlation

• It stores past experiences and samples them randomly to break correlation in sequential data

Q-Learning and SARSA both estimate Q-values, but Q-Learning is considered off-policy because

• The actions used in the bootstrapped target are always taken from the same policy that generates behavior

• It ignores the discount factor in updates

• It never uses \epsilon-greedy exploration

• It uses the same policy for both exploration and evaluation

• It updates using a greedy action for the next state, not necessarily the one followed by the agent during data collection

Why does DQN use a separate target network?

• To generate randomized actions for exploration

• To eliminate the need for discounting future rewards

• To stabilize Q-value updates by keeping target estimates fixed for a while

• To convert a continuous action space into a discrete one

• To independently learn a model of the transition probabilities

Double Q-Learning was introduced primarily to address which issue?

• Handling continuous actions without an actor-critic method

• The inability of Q-Learning to handle function approximation

• The instability caused by batch updates in Q-Learning

• Overestimation bias in Q-value updates due to using \max over the same Q function

• Lack of exploration in Q-Learning

TD methods differ from Dynamic Programming (DP) primarily because TD methods

• Do not use the concept of value functions

• Only work in deterministic environments

• Always converge faster than DP methods

• Can learn directly from raw experience without knowing transition probabilities

• Require a perfect model of the environment’s transitions

In SARSA, the next action used in the update is

• Irrelevant, because SARSA does not require a next action

• The action actually chosen by the current (often \epsilon-greedy) policy in the next time step

• Determined by a different policy than the one being evaluated

• The greedy action that maximizes the Q-value in the next state

• The greedy action that maximizes the Q-value in the next state

Why is Q-Learning considered an off-policy method?

• It does not require a replay buffer or any historical data.

• It updates its estimates based on Monte Carlo returns only.

• It never explores and only exploits the best action.

• It requires running a secondary “behavior policy” to gather experience but updates the values as if it followed a greedy policy.

• It abandons temporal difference learning.

In the context of TD methods, “bootstrapping” refers to which concept?

• Learning only from complete returns collected at the end of each episode

• Combining multiple policies simultaneously

• Updating value estimates based partly on other learned estimates rather than exclusively on actual returns

• Taking random actions in the environment to initialize the replay buffer

• Bypassing the need for an explicit Q-value function