Reinforcement Learning

An MDP is the tuple $(s, a, r, s')$ with transition dynamics and reward, satisfying the Markov property: the next state depends only on the current state and action.

The return is $G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}$. Discounting ($\gamma < 1$) keeps infinite sums finite and weights near-term reward more heavily.

$V^\pi(s) = \mathbb{E}_{a\sim\pi}\big[r + \gamma V^\pi(s')\big]$

$Q^\pi(s,a) = \mathbb{E}\big[r + \gamma \mathbb{E}_{a'\sim\pi} Q^\pi(s',a')\big]$

This recursive relationship — value now equals immediate reward plus discounted value of what follows — is what every value-based method exploits.

Q-learning updates toward the best next action regardless of what was actually taken (off-policy): $Q(s,a) \leftarrow Q(s,a) + \alpha\big[r + \gamma \max_{a'} Q(s',a') - Q(s,a)\big]$.

DQN replaces the table with a neural network and adds experience replay and a target network to stabilize training with correlated, non-stationary data.

REINFORCE directly increases the probability of actions that led to high return: it moves $\pi$'s parameters along $\nabla \log \pi(a|s) \cdot G_t$.

Actor-critic methods (PPO, SAC) add a learned value function (the critic) to reduce the variance of that gradient estimate, and constrain or regularize the policy update for stability — PPO clips the policy change per step, SAC adds an entropy bonus to keep exploring.

An agent must trade exploiting the best-known action against exploring to discover better ones — common tactics are $\epsilon$-greedy, entropy bonuses, or optimism under uncertainty.

On-policy methods (REINFORCE, PPO) learn only from data generated by the current policy; off-policy methods (Q-learning, SAC) can reuse past experience from any policy, which is usually more sample-efficient but less stable.

Policies trained in simulation often fail on the real robot because dynamics, sensor noise, and visuals differ from the sim model.

Domain randomization varies simulated physics/appearance during training so the policy generalizes across that uncertainty; system identification instead tunes the simulator's parameters to better match the real robot. Reward shaping — adding intermediate signals — helps sparse-reward tasks learn faster but risks encouraging unintended shortcuts if misdesigned.