AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

spinning-up-deep-rl

Use when learning or applying deep RL from OpenAI's Spinning Up: MDPs, policy gradients, VPG, TRPO, PPO, DDPG, TD3, SAC, debugging RL code that fails silently, or multi-seed experiments. Indexes the 20 chapters for on-demand reading.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--spinning-up-deep-rl at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 24

patterns.md

Patterns

Concrete techniques from Spinning Up, each with when to use it, how it works, and what it costs.

Reward-to-Go Weighting

When to use — always, over full-trajectory-return weighting. It is a two-line change. How — weight each grad log pi(a_t|s_t) by sum_{t'=t}^T r_t' instead of R(tau). Trade-offs — strictly reduces variance at no cost. The dropped terms had zero mean and nonzero variance, so removing them removes pure noise and cuts the sample trajectories needed. (Ch 9)

Value Function Baseline

When to use — any policy gradient method. Standard in VPG, TRPO, PPO and A2C. How — subtract V_phi(s_t) from the weight; fit V_phi by minimizing E[(V_phi(s_t) - Rhat_t)^2] with one or more gradient steps per epoch, starting from the previous parameters, so it tracks the most recent policy. Trade-offs — provably unbiased by EGLP; reduces variance and gives faster, more stable learning. Costs a second network and its fitting budget. Under-fitting it is one of the most drastic performance failures in policy gradients (Exercise 2.1). (Ch 9, Ch 12)

Trust Region via KL Constraint

When to use — when a single bad policy gradient step can collapse performance, and you can afford second-order machinery. How — maximize surrogate advantage subject to Dbar_KL <= delta; Taylor-expand, solve analytically via Lagrangian duality, then backtracking line search until the exact KL constraint holds and surrogate advantage is positive. Use conjugate gradient for H^{-1}g. Trade-offs — buys monotone-ish, fast improvement and larger safe steps. Costs complexity, Hessian-vector products, and (in Spinning Up) no PyTorch implementation. (Ch 15)

Clipped Surrogate Objective

When to use — as the first-order replacement for a trust region. The default modern choice. How — L = min(ratio * A, g(eps, A)) where g caps at (1+eps)A for positive advantage and (1-eps)A for negative. Take multiple minibatch SGD steps per batch. Trade-offs — much simpler than TRPO and empirically at least as good, but it removes the incentive to move far rather than guaranteeing you do not. Pair it with KL early stopping. (Ch 16)

KL Early Stopping

When to use — alongside clipping, whenever you take many gradient steps per batch. How — if the mean KL-divergence of the new policy from the old exceeds a threshold, stop taking gradient steps for this batch. Trade-offs — one extra measurement per step; catches the drift the clip alone allows. Other PPO implementations use different tricks here. (Ch 16)

Experience Replay Buffer

When to use — every off-policy value-learning algorithm. How — store (s,a,r,s',d) tuples; sample minibatches uniformly for MSBE minimization. Licensed by the fact that the Bellman equation is indifferent to how transitions were collected. Trade-offs — wrong in both directions. Too small and you overfit to the most recent data and things break; too large and learning slows. It needs tuning. (Ch 17)

Target Networks

When to use — any MSBE minimization, because the target otherwise depends on the parameters being trained. How — keep a lagged copy. DQN-style: copy every fixed number of steps. DDPG-style: polyak average once per main update, phi_targ <- rho*phi_targ + (1-rho)*phi, rho near 1. Trade-offs — stabilizes learning; adds a network and a hyperparameter, and slows the propagation of new information into the target. (Ch 17)

Amortizing a Continuous Argmax into a Policy

When to use — an inner max over a continuous variable inside a loop you run constantly. How — assume differentiability with respect to that variable, learn mu_theta(s) by gradient ascent on Q_phi(s, mu_theta(s)), and substitute max_a Q(s,a) ~= Q(s, mu(s)). Trade-offs — turns an intractable per-step optimization into a forward pass. Costs an approximation whose errors the policy will actively exploit. (Ch 17)

Clipped Double-Q

When to use — whenever a learned Q-function is being maximized over by a policy, i.e. the whole DDPG family. How — learn two Q-functions, use min of the two target values as the shared regression target for both. Trade-offs — fends off overestimation, which is DDPG's dominant failure. Costs a second critic and introduces some underestimation bias. TD3 uses Q_phi_1 alone in the policy loss; SAC uses the min there too. (Ch 18, Ch 19)

Delayed Policy Updates

When to use — when policy updates destabilize the Bellman target. How — update the policy and all target networks once per policy_delay critic updates; the paper recommends 2. Trade-offs — damps volatility; slows policy improvement per environment step. (Ch 18)

Target Policy Smoothing

When to use — deterministic policies over continuous actions. How — a'(s') = clip(mu_targ(s') + clip(eps, -c, c), a_Low, a_High), eps ~ N(0, sigma). Trade-offs — regularizes Q along the action dimension so a spurious sharp peak cannot be exploited. A stochastic policy (SAC) gets a similar effect for free. (Ch 18, Ch 19)

Entropy Regularization

When to use — when premature convergence to a bad local optimum is the risk, or when you want an explicit explore-exploit dial. How — add alpha * H(pi(.|s_t)) to the reward at each timestep; propagate the term into the value functions and the Bellman target. Trade-offs — higher alpha means more exploration and faster later learning; the right value is environment-specific and needs careful tuning. The entropy-constrained variant that adapts alpha is generally preferred by practitioners over the fixed one. (Ch 19)

Reparameterization Trick for Stochastic Policies

When to use — differentiating through an expectation whose distribution depends on the parameters. How — sample by a deterministic function of state, parameters and independent noise: a~ = tanh(mu_theta(s) + sigma_theta(s) * xi), xi ~ N(0,I), converting the expectation over actions into an expectation over noise. Trade-offs — makes the policy differentiable end to end. The tanh squash bounds actions but changes the distribution; log-probabilities are still closed-form. (Ch 19)

Uniform Random Warm-Up (start_steps)

When to use — every off-policy algorithm here, at the beginning of training. How — for a fixed number of steps take actions sampled uniformly over valid actions, then switch to normal exploration. Trade-offs — fills the replay buffer with diverse data before the policy can bias it. Wasted steps if the environment is expensive. (Ch 17, Ch 18, Ch 19)

Multi-Seed Reporting with Variance Bands

When to use — every reported RL result. How — at least 3 seeds, 10 or more to be thorough; plot the mean as a solid line and the standard deviation as a shaded band; smooth for display only. Trade-offs — multiplies compute by the seed count. Without it, deep RL's seed sensitivity means two seed groups can look like different distributions entirely. (Ch 10, Ch 13)

Precommitted Final Runs

When to use — before reporting any comparison. How — use the tuning stage to form hypotheses, then launch fresh final experiments for every method compared and commit in advance to reporting whatever comes out. Trade-offs — a weak form of preregistration; costs one more full experiment round and removes the largest source of accidental self-deception. (Ch 10)

Per-Claim Ablation

When to use — any method with more than one design decision, which is all of them. How — swap out or remove each design element separately and measure. Trade-offs — turns one bundled claim into several separately-confident ones. Costs one experiment per element. Also the fastest way to learn which parts you can delete. (Ch 10)