glossary.md
Glossary
Action space — the set of all valid actions in an environment; discrete or continuous (Ch 7)
Action-value function — see Q-function (Ch 7)
Advantage function — A^pi(s,a) = Q^pi(s,a) - V^pi(s); how much better a specific action is than randomly selecting one per pi (Ch 7, Ch 9)
Agent — the learner and decider; often used interchangeably with "policy" (Ch 7)
Backtracking line search — shrink the step by alpha^j until the exact KL constraint holds and surrogate advantage is positive (Ch 15)
Baseline — any state-only function b(s) subtractable from the policy gradient weight without changing its expectation (Ch 9)
Bellman backup — the right-hand side of a Bellman equation: reward plus next value (Ch 7)
Bellman equation — self-consistency condition: the value of your starting point is the reward you expect from being there plus the value of where you land next (Ch 7)
Categorical policy — stochastic policy for discrete actions, built like a classifier: logits then softmax (Ch 7)
Clipped double-Q learning — learn two Q-functions and use the smaller value to form the target (Ch 18, Ch 19)
Deadly triad — function approximation + bootstrapping + off-policy data, jointly causing instability in value learning (Ch 8)
Diagonal Gaussian policy — stochastic policy for continuous actions; mean network plus a diagonal covariance represented as a vector of log stds (Ch 7)
Discount factor (gamma) — in (0,1); makes an infinite reward sum converge and encodes "cash now beats cash later" (Ch 7)
EGLP lemma — Expected Grad-Log-Prob: E_{x~P_theta}[grad log P_theta(x)] = 0 (Ch 9)
Entropy — H(P) = E_{x~P}[-log P(x)]; how random a random variable is (Ch 19)
Entropy regularization — adding an entropy bonus to the reward at each timestep, scaled by alpha (Ch 19)
Epoch — one experience-collection phase plus one policy update (Ch 9)
ExperimentGrid — Spinning Up's hyperparameter ablation tool; backs the CLI (Ch 4, Ch 20)
Fully / partially observed — whether the agent sees the complete state or only part of it (Ch 7)
GAE — Generalized Advantage Estimation; used by Spinning Up's VPG, TRPO and PPO (Ch 9)
Log-derivative trick — grad P = P * grad log P (Ch 9)
Markov property — transitions depend only on the most recent state and action (Ch 7)
MDP — Markov Decision Process, the 5-tuple <S, A, R, P, rho_0> (Ch 7)
Model — a function predicting state transitions and rewards (Ch 8)
Model-based / model-free — whether the agent has or learns a model (Ch 8)
MPC — model-predictive control: plan over a window, execute the first action, discard the rest, re-plan (Ch 8)
MSBE — Mean-Squared Bellman Error, the loss deep Q-learning minimizes (Ch 17)
Natural Policy Gradient — TRPO's analytic solution before the line search is added (Ch 15)
Observation — a partial description of a state, possibly omitting information (Ch 7)
Off-policy — updates may use data collected at any point in training (Ch 3, Ch 8)
On-policy — updates use only data from the most recent policy (Ch 3, Ch 8)
Optimal policy (pi*) — argmax_pi J(pi) (Ch 7)
Parameterized policy — a policy whose output depends on adjustable parameters, e.g. network weights (Ch 7)
Performance — the plotter's family-aware alias, resolving to AverageEpRet (on-policy) or AverageTestEpRet (off-policy) (Ch 6)
Policy — the rule for choosing actions; deterministic mu(s) or stochastic pi(.|s) (Ch 7)
Policy delay — TD3's ratio of Q-updates to policy updates; recommended 2 (Ch 18)
Policy gradient — grad_theta J(pi_theta) (Ch 9)
Polyak averaging — target update phi_targ <- rho*phi_targ + (1-rho)*phi; polyak in the code (Ch 17)
Q-function — Q^pi(s,a), expected return from taking a in s then following pi (Ch 7)
Replay buffer — the stored set of past transitions that off-policy algorithms sample from (Ch 17)
Reparameterization trick — sample by a deterministic function of state, parameters and independent noise (Ch 19)
Return — cumulative reward over a trajectory; finite-horizon undiscounted or infinite-horizon discounted (Ch 7)
Reward-to-go — Rhat_t = sum_{t'=t}^T R(s_t', a_t', s_{t'+1}) (Ch 9)
Silent failure — broken RL code that runs correctly and simply never learns (Ch 10, Ch 12)
Squashed Gaussian policy — a Gaussian passed through tanh to bound actions; SAC's policy (Ch 19)
start_steps — initial phase of uniform-random actions used by DDPG, TD3 and SAC (Ch 17)
State — a complete description of the world, with nothing hidden (Ch 7)
Surrogate advantage — importance-weighted advantage measuring new-policy performance on old-policy data (Ch 15)
Surrogate objective — a local approximation of J(pi_theta) that is safe to maximize (Ch 8, Ch 16)
Target network — a time-lagged copy of a network used to compute stable Bellman targets (Ch 17)
Target policy smoothing — clipped noise added to the target action to prevent exploiting sharp Q peaks (Ch 18)
Trajectory (tau) — a sequence of states and actions; also called episode or rollout (Ch 7)
Trust region — a KL-divergence bound on how far the new policy may move from the old (Ch 15)
Value function — V^pi(s), expected return from s following pi (Ch 7)