策略梯度算法汇总

强化学习笔记

GSPO

\[\begin{aligned} \mathcal{J}_{GSPO} \left( \theta \right) &= \mathbb{E}_{x \sim \mathcal{D}, \{y_{i}\}_{i=1}^{G} \sim \pi_{\theta_{old}} \left( \cdot | x \right)} \\ &= \left[ \frac{1}{G} \sum_{i=1}^{G} \text{min} \left( s_{i}\left( \theta \right) \hat{A_{i}}, \text{clip} \left(s_{i}\left( \theta \right), 1-\epsilon, 1 + \epsilon \right)\hat{A_{i}} \right) \right] \end{aligned}\]