AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

spinning-up-deep-rl

Use when learning or applying deep RL from OpenAI's Spinning Up: MDPs, policy gradients, VPG, TRPO, PPO, DDPG, TD3, SAC, debugging RL code that fails silently, or multi-seed experiments. Indexes the 20 chapters for on-demand reading.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--spinning-up-deep-rl at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 24

chapters/ch18-td3.md

Chapter 18: Twin Delayed DDPG (TD3)

Core Idea

DDPG's common failure mode is that the learned Q-function begins to dramatically overestimate Q-values, which then breaks the policy, because the policy exploits the errors in the Q-function. TD3 fixes this with three tricks, and nothing else changes.

Quick Facts

  • Off-policy.
  • Continuous action spaces only.
  • The Spinning Up implementation does not support parallelization.

Frameworks Introduced

  • Trick One: Clipped Double-Q Learning. Learn two Q-functions instead of one (hence "twin") and use the smaller of the two Q-values to form the targets in the Bellman error loss functions. Both are then regressed toward that single shared target:

    y(r,s',d) = r + gamma*(1-d) * min_{i=1,2} Q_{phi_i,targ}(s', a'(s'))
    L(phi_1, D) = E[ (Q_phi_1(s,a) - y)^2 ]
    L(phi_2, D) = E[ (Q_phi_2(s,a) - y)^2 ]
    

    Using the smaller Q-value for the target, and regressing toward that, helps fend off overestimation — a pessimistic estimate cannot be exploited the way an optimistic one can.

  • Trick Two: "Delayed" Policy Updates. Update the policy (and the target networks) less frequently than the Q-function. The paper recommends one policy update for every two Q-function updates. This damps the volatility that normally arises in DDPG because of how a policy update changes the target. Controlled by policy_delay.

  • Trick Three: Target Policy Smoothing. Add clipped noise to each dimension of the target action, then clip the result back into the valid action range:

    a'(s') = clip( mu_theta_targ(s') + clip(epsilon, -c, c), a_Low, a_High ),  epsilon ~ N(0, sigma)
    

    This is a regularizer. It addresses a specific DDPG failure: if the Q-function approximator develops an incorrect sharp peak for some actions, the policy will quickly exploit that peak and then have brittle or incorrect behavior. Smoothing Q over similar actions averts it.

  • Policy learning is unchanged from DDPG except in which critic it uses: maximize E_{s~D}[ Q_phi_1(s, mu_theta(s)) ] — just the first Q-function. (SAC differs here; see ch19.)

Reference Tables

| Trick | Fixes | Mechanism | |-------|-------|-----------| | Clipped double-Q | Q-value overestimation | Two critics, min of the two forms the shared target | | Delayed policy updates | Volatility from policy updates moving the target | One policy + target update per policy_delay critic updates | | Target policy smoothing | Policy exploiting sharp incorrect peaks in Q | Clipped Gaussian noise on the target action, then clip to action bounds |

| Hyperparameter | Role | |----------------|------| | sigma | Std dev of the target-smoothing noise | | c | Clip bound on that noise | | policy_delay | Q-updates per policy update; paper recommends 2 | | rho (polyak) | Target network averaging coefficient, inherited from DDPG | | start_steps | Initial uniform-random action steps, inherited from DDPG |

The loop is DDPG's, with two changes inside the update block: compute the smoothed target action a'(s') first, update both Q-functions toward the clipped-double-Q target, and then — only when j mod policy_delay == 0 — take the policy ascent step and polyak-update all three target networks.

Exploration vs. Exploitation

Identical to DDPG: deterministic policy explored with uncorrelated mean-zero Gaussian action noise at training time, optionally decayed (Spinning Up keeps it fixed), no noise at test time, and a start_steps phase of uniform random actions at the beginning.

Note the distinction that is easy to blur: exploration noise is added to the acting policy; target policy smoothing noise is added to the target action inside the update. Two different noises, two different jobs.

Mental Models

  • All three tricks are forms of pessimism or patience. Take the smaller estimate, wait longer before acting on the critic, and refuse to believe a Q-value that does not hold up under a small action perturbation. Overestimation is the disease; conservatism is the cure.
  • Think of the policy as an adversary against your critic's errors. Anything the critic gets wrong in the optimistic direction is exactly what the policy will find. This is the same failure shape as model bias in ch8's model-based methods.
  • "Together, these three tricks result in substantially improved performance over baseline DDPG" — they are presented as a package, and the exercise (ch12, 1.3) has you implement the losses that contain all three.

Anti-patterns

  • Implementing only clipped double-Q and calling it TD3. The three tricks address three distinct failure modes and the paper's result is for the package.
  • Using min over the two critics for the policy loss. TD3 uses Q_phi_1 alone for the policy; using the min there is a SAC choice, not a TD3 one.
  • Updating the target networks on every critic step. The target updates are inside the policy_delay branch, along with the policy update.

Key Takeaways

  1. TD3 = DDPG + clipped double-Q + delayed policy updates + target policy smoothing.
  2. The disease being treated is Q-value overestimation, which the policy actively exploits.
  3. min of two critics forms the target; the policy maximizes Q_phi_1 only.
  4. Policy and target-network updates both live behind policy_delay (recommended 2).
  5. Target smoothing noise is clipped twice: clip the noise to [-c, c], then clip the resulting action to the valid range.
  6. Relevant paper: Fujimoto et al, 2018, Addressing Function Approximation Error in Actor-Critic Methods.

Connects To

  • Ch 17: DDPG, whose brittleness is TD3's entire premise.
  • Ch 19: SAC, which borrows clipped double-Q and gets smoothing for free from stochasticity.
  • Ch 12: Exercise 1.3 asks you to write exactly these losses.
  • Ch 13: TD3 is one of the three research-grade Spinning Up implementations.