AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

spinning-up-deep-rl

Use when learning or applying deep RL from OpenAI's Spinning Up: MDPs, policy gradients, VPG, TRPO, PPO, DDPG, TD3, SAC, debugging RL code that fails silently, or multi-seed experiments. Indexes the 20 chapters for on-demand reading.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--spinning-up-deep-rl at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 24

glossary.md

Glossary

Action space — the set of all valid actions in an environment; discrete or continuous (Ch 7) Action-value function — see Q-function (Ch 7) Advantage function — A^pi(s,a) = Q^pi(s,a) - V^pi(s); how much better a specific action is than randomly selecting one per pi (Ch 7, Ch 9) Agent — the learner and decider; often used interchangeably with "policy" (Ch 7) Backtracking line search — shrink the step by alpha^j until the exact KL constraint holds and surrogate advantage is positive (Ch 15) Baseline — any state-only function b(s) subtractable from the policy gradient weight without changing its expectation (Ch 9) Bellman backup — the right-hand side of a Bellman equation: reward plus next value (Ch 7) Bellman equation — self-consistency condition: the value of your starting point is the reward you expect from being there plus the value of where you land next (Ch 7) Categorical policy — stochastic policy for discrete actions, built like a classifier: logits then softmax (Ch 7) Clipped double-Q learning — learn two Q-functions and use the smaller value to form the target (Ch 18, Ch 19) Deadly triad — function approximation + bootstrapping + off-policy data, jointly causing instability in value learning (Ch 8) Diagonal Gaussian policy — stochastic policy for continuous actions; mean network plus a diagonal covariance represented as a vector of log stds (Ch 7) Discount factor (gamma) — in (0,1); makes an infinite reward sum converge and encodes "cash now beats cash later" (Ch 7) EGLP lemma — Expected Grad-Log-Prob: E_{x~P_theta}[grad log P_theta(x)] = 0 (Ch 9) Entropy — H(P) = E_{x~P}[-log P(x)]; how random a random variable is (Ch 19) Entropy regularization — adding an entropy bonus to the reward at each timestep, scaled by alpha (Ch 19) Epoch — one experience-collection phase plus one policy update (Ch 9) ExperimentGrid — Spinning Up's hyperparameter ablation tool; backs the CLI (Ch 4, Ch 20) Fully / partially observed — whether the agent sees the complete state or only part of it (Ch 7) GAE — Generalized Advantage Estimation; used by Spinning Up's VPG, TRPO and PPO (Ch 9) Log-derivative trick — grad P = P * grad log P (Ch 9) Markov property — transitions depend only on the most recent state and action (Ch 7) MDP — Markov Decision Process, the 5-tuple <S, A, R, P, rho_0> (Ch 7) Model — a function predicting state transitions and rewards (Ch 8) Model-based / model-free — whether the agent has or learns a model (Ch 8) MPC — model-predictive control: plan over a window, execute the first action, discard the rest, re-plan (Ch 8) MSBE — Mean-Squared Bellman Error, the loss deep Q-learning minimizes (Ch 17) Natural Policy Gradient — TRPO's analytic solution before the line search is added (Ch 15) Observation — a partial description of a state, possibly omitting information (Ch 7) Off-policy — updates may use data collected at any point in training (Ch 3, Ch 8) On-policy — updates use only data from the most recent policy (Ch 3, Ch 8) Optimal policy (pi*) — argmax_pi J(pi) (Ch 7) Parameterized policy — a policy whose output depends on adjustable parameters, e.g. network weights (Ch 7) Performance — the plotter's family-aware alias, resolving to AverageEpRet (on-policy) or AverageTestEpRet (off-policy) (Ch 6) Policy — the rule for choosing actions; deterministic mu(s) or stochastic pi(.|s) (Ch 7) Policy delay — TD3's ratio of Q-updates to policy updates; recommended 2 (Ch 18) Policy gradient — grad_theta J(pi_theta) (Ch 9) Polyak averaging — target update phi_targ <- rho*phi_targ + (1-rho)*phi; polyak in the code (Ch 17) Q-function — Q^pi(s,a), expected return from taking a in s then following pi (Ch 7) Replay buffer — the stored set of past transitions that off-policy algorithms sample from (Ch 17) Reparameterization trick — sample by a deterministic function of state, parameters and independent noise (Ch 19) Return — cumulative reward over a trajectory; finite-horizon undiscounted or infinite-horizon discounted (Ch 7) Reward-to-go — Rhat_t = sum_{t'=t}^T R(s_t', a_t', s_{t'+1}) (Ch 9) Silent failure — broken RL code that runs correctly and simply never learns (Ch 10, Ch 12) Squashed Gaussian policy — a Gaussian passed through tanh to bound actions; SAC's policy (Ch 19) start_steps — initial phase of uniform-random actions used by DDPG, TD3 and SAC (Ch 17) State — a complete description of the world, with nothing hidden (Ch 7) Surrogate advantage — importance-weighted advantage measuring new-policy performance on old-policy data (Ch 15) Surrogate objective — a local approximation of J(pi_theta) that is safe to maximize (Ch 8, Ch 16) Target network — a time-lagged copy of a network used to compute stable Bellman targets (Ch 17) Target policy smoothing — clipped noise added to the target action to prevent exploiting sharp Q peaks (Ch 18) Trajectory (tau) — a sequence of states and actions; also called episode or rollout (Ch 7) Trust region — a KL-divergence bound on how far the new policy may move from the old (Ch 15) Value function — V^pi(s), expected return from s following pi (Ch 7)