Policy Optimization

NeurIPS 2025arXiv:2410.07163

#3

Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning

Chongyu Fan, Jiancheng Liu, Licong Lin et al.

llm unlearningnegative preference optimizationreference model biasgradient ascent methods+4

70

NeurIPS 2025arXiv:2503.22342

#4

CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models

Zhihang Lin, Mingbao Lin, Yuan Xie et al.

policy optimizationtraining accelerationreasoning modelscompletion pruning+2

47

NeurIPS 2025arXiv:2502.00604

#5

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

Audrey Huang, Wenhao Zhan, Tengyang Xie et al.

Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective

Sifan Wang, Ananyae bhartari, Bowen Li et al.

32

ICLR 2025arXiv:2403.15180

#7

CPPO: Continual Learning for Reinforcement Learning with Human Feedback

Han Zhang, Yu Lei, Lin Gui et al.

Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity

Eduard Gorbunov, Nazarii Tupitsa, Sayantan Choudhury et al.

Self-Improvement for Neural Combinatorial Optimization: Sample Without Replacement, but Improvement

Dominik Grimm, Jonathan Pirnay

26

ICML 2025arXiv:2502.00361

#10

Efficient Online Reinforcement Learning for Diffusion Policy

Haitong Ma, Tianyi Chen, Kai Wang et al.

24

AAAI 2024arXiv:2305.16830

#11

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

Yuzi Yan, Yibo Miao, Jialian Li et al.

Leaving the Nest: Going beyond Local Loss Functions for Predict-Then-Optimize

Sanket Shah, Bryan Wilder, Andrew Perrault et al.

predict-then-optimizedecision-making under uncertaintytask-specific loss functionssample efficiency+1

20

AAAI 2025arXiv:2304.11787

#13

B2Opt: Learning to Optimize Black-box Optimization with Little Budget

Xiaobin Li, Kai Wu, Xiaoyu Zhang et al.

18

ICLR 2025arXiv:2410.04203

#14

RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization

Hanyang Zhao, Genta Winata, Anirban Das et al.

17

NeurIPS 2025arXiv:2505.23564

#15

Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods

Oussama Zekri, Nicolas Boulle

Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models

Yiran Guo, Lijie Xu, Jie Liu et al.

15

AAAI 2024arXiv:2311.13460

#17

Multi-Objective Bayesian Optimization with Active Preference Learning

Ryota Ozaki, Kazuki Ishikawa, Youhei Kanzaki et al.

bayesian optimizationmulti-objective optimizationpreference learningpareto front identification+4

14

ICLR 2024arXiv:2310.02671

#18

Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods

Sara Klein, Simon Weissmann, Leif Döring

12

ICLR 2024arXiv:2401.09587

#19

Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis

Jie Hao, Xiaochuan Gong, Mingrui Liu

11

AAAI 2024arXiv:2207.05631

#20

DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization

Wenze Chen, Shiyu Huang, Yuan Chiang et al.

reinforcement learningdiverse strategy discoverypolicy optimizationinformation-theoretic diversity+3

9

ICLR 2025arXiv:2409.17401

#21

Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

Qining Zhang, Lei Ying

9

ICLR 2024arXiv:2311.08384

#22

Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking

Paria Rashidinejad, Yuandong Tian

Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees

Yifei Zhou, Ayush Sekhari, Yuda Song et al.

8

AAAI 2025arXiv:2412.07639

#24

Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization

Zongkai Liu, Qian Lin, Chao Yu et al.

8

AAAI 2024arXiv:2308.07272

#25

Dialogue for Prompting: A Policy-Gradient-Based Discrete Prompt Generation for Few-Shot Learning

Chengzhengxu Li, Xiaoming Liu, Yichen Wang et al.

prompt-based learningfew-shot learningdiscrete prompt optimizationpolicy gradient methods+4

7

AAAI 2025arXiv:2409.05622

#26

Forward KL Regularized Preference Optimization for Aligning Diffusion Policies

Zhao Shan, Chenyou Fan, Shuang Qiu et al.

7

ICLR 2024arXiv:2401.06604

#27

Identifying Policy Gradient Subspaces

Jan Schneider, Pierre Schumacher, Simon Guist et al.

7

ICML 2025arXiv:2502.04164

#28

Efficient Distributed Optimization under Heavy-Tailed Noise

Su Hyeong Lee, Manzil Zaheer, Tian Li

6

ICLR 2024arXiv:2310.13863

#29

Distributionally Robust Optimization with Bias and Variance Reduction

Ronak Mehta, Vincent Roulet, Krishna Pillutla et al.

6

NeurIPS 2025arXiv:2509.17940

#30

DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving

Shuyao Shang, Yuntao Chen, Yuqi Wang et al.

end-to-end autonomous drivingpolicy learningsafety preference optimizationtrajectory prediction+3

6

AAAI 2025arXiv:2412.10961

#31

Optimizing Posterior Samples for Bayesian Optimization via Rootfinding

Taiwo Adebiyi, Bach Do, Ruda Zhang

Risk-Controlling Model Selection via Guided Bayesian Optimization

Adam Fisch, Regina Barzilay, Bracha Laufer-Goldshtein et al.

PSMGD: Periodic Stochastic Multi-Gradient Descent for Fast Multi-Objective Optimization

Mingjing Xu, Peizhong Ju, Jia Liu et al.

5

ICML 2025arXiv:2502.12631

#34

Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport

Mingyang Sun, Pengxiang Ding, Weinan Zhang et al.

ICLR 2024arXiv:2407.04864

#35

POCE: Primal Policy Optimization with Conservative Estimation for Multi-constraint Offline Reinforcement Learning

Jiayi Guan, Li Shen, Ao Zhou et al.

Augmented Bayesian Policy Search

Mahdi Kallel, Debabrota Basu, Riad Akrour et al.

ICML 2025arXiv:2502.18699

#37

MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

Tianze Wang, Dongnan Gui, Yifan Hu et al.

ICML 2025arXiv:2502.19255

#38

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective

Jiawei Huang, Bingcong Li, Christoph Dann et al.

ICML 2025arXiv:2502.15145

#39

Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF

Nuoya Xiong, Aarti Singh

NeurIPS 2025arXiv:2311.01104

#40

On the Convergence of Projected Policy Gradient for Any Constant Step Sizes

Jiacai Liu, Wenye Li, Dachao Lin et al.

policy gradient methodsreinforcement learningpolicy optimizationconvergence analysis+3

NeurIPS 2025arXiv:2505.18763

#41

HyPoGen: Optimization-Biased Hypernetworks for Generalizable Policy Generation

Hanxiang Ren, Li Sun, Xulong Wang et al.

GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning

Shutong Ding, Ke Hu, Shan Zhong et al.

NeurIPS 2025arXiv:2505.19238

#43

Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees

Sourav Ganguly, Kishan Panaganti, Arnob Ghosh et al.

NeurIPS 2025arXiv:2502.08021

#44

Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol

Pai Liu, Lingfeng Zhao, Shivangi Agarwal et al.

off-policy evaluationhyperparameter tuningmodel-free selectionmodel-based selection+4

AAAI 2025arXiv:2501.01874

#45

On-the-fly Preference Alignment via Principle-Guided Decoding

Mingye Zhu, Yi Liu, Lei Zhang et al.

Direct Distributional Optimization for Provable Alignment of Diffusion Models

Ryotaro Kawata, Kazusato Oko, Atsushi Nitanda et al.

RePO: Understanding Preference Learning Through ReLU-Based Optimization

Junkang Wu, Kexin Huang, xue wang et al.

DFF: Decision-Focused Fine-Tuning for Smarter Predict-Then-Optimize with Limited Data

Jiaqi Yang, Enming Liang, Zicheng Su et al.

ICLR 2025arXiv:2410.04834

#49

As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss

Xin Mao, Huimin Xu, Feng-Lin Li et al.

ICLR 2025arXiv:2410.02226

#50

Doubly Optimal Policy Evaluation for Reinforcement Learning

Shuze Liu, Claire Chen, Shangtong Zhang

AAAI 2024arXiv:2312.12067

#51

Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence beyond the Minty Property

I. Anagnostides, Ioannis Panageas, Gabriele Farina et al.

policy gradient methodsmulti-player markov gamessingle controller gamescoarse correlated equilibria+4

ICML 2025arXiv:2505.00663

#52

Wasserstein Policy Optimization

David Pfau, Ian Davies, Diana Borsa et al.

ICML 2025arXiv:2502.06861

#53

Design Considerations in Offline Preference-based RL

Alekh Agarwal, Christoph Dann, Teodor Vanislavov Marinov

ICML 2025arXiv:2406.01939

#54

Speeding up Policy Simulation in Supply Chain RL

Vivek Farias, Joren Gijsbrechts, Aryan Khojandi et al.

ICML 2025arXiv:2502.14096

#55

Aligned Multi Objective Optimization

Yonathan Efroni, Ben Kretzu, Daniel Jiang et al.

ICML 2025arXiv:2505.00793

#56

Scalable Meta-Learning via Mixed-Mode Differentiation

Iurii Kemaev, Dan Andrei Calian, Luisa Zintgraf et al.

ICML 2025arXiv:2412.05204

#57

Global Optimization with a Power-Transformed Objective and Gaussian Smoothing

Chen Xu

NeurIPS 2025arXiv:2505.22224

#58

Solver-Free Decision-Focused Learning for Linear Optimization Problems

Senne Berden, Ali Mahmutoğulları, Dimos Tsouros et al.

decision-focused learningpredict-then-optimize problemslinear optimizationfeasible polytope+2

NeurIPS 2025arXiv:2411.06568

#59

KL Penalty Control via Perturbation for Direct Preference Optimization

Sangkyu Lee, Janghoon Han, Hosung Song et al.

Meta-Learning Objectives for Preference Optimization

Carlo Alfano, Silvia Sapora, Jakob Foerster et al.

preference optimizationmirror descentevolutionary strategiesllm alignment+3

AAAI 2025arXiv:2408.08706

#61

Global Convergence of Policy Gradient in Average Reward MDPs

Navdeep Kumar, Yashaswini Murthy, Itai Shufaro et al.

The Directionality of Optimization Trajectories in Neural Networks

Sidak Pal Singh, Bobby He, Thomas Hofmann et al.

Efficient Multi-Policy Evaluation for Reinforcement Learning

Shuze Daniel Liu, Claire Chen, Shangtong Zhang

AAAI 2025arXiv:2502.06971

#64

User Preference Meets Pareto-Optimality in Multi-Objective Bayesian Optimization

Joshua Hang Sai Ip, Ankush Chakrabarty, Ali Mesbah et al.

ICLR 2025arXiv:2310.14768

#65

Policy Gradient with Kernel Quadrature

Tetsuro Morimura, Satoshi Hayakawa

policy gradient methodskernel quadraturegaussian processesreinforcement learning+3

AAAI 2025arXiv:2408.10015

#66

Multi-agent cooperation through learning-aware policy gradients

Alexander Meulemans, Seijin Kobayashi, Johannes von Oswald et al.

Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs

Sergio Rozada, Dongsheng Ding, Antonio G. Marques et al.

AAAI 2025arXiv:2210.07703

#68

Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers for Faster Convergence

Shayan Talaei, Matin Ansaripour, Giorgi Nadiradze et al.

AAAI 2025arXiv:2410.10417

#69

A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning

Minyoung Kim, Timothy Hospedales

AAAI 2025arXiv:2412.13184

#70

Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning

Chenglin Li, Guangchun Ruan, Hua Geng

AAAI 2025arXiv:2408.09891

#71

Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Rates

Puning Zhao, Jiafei Wu, Zhe Liu et al.

NeurIPS 2025arXiv:2505.19601

#72

Consensus Based Stochastic Optimal Control

Liyao Lyu, Jingrun Chen

Instance-wise Supervision-level Optimization in Active Learning

Shinnosuke Matsuo, Riku Togashi, Ryoma Bise et al.

Preference Optimization by Estimating the Ratio of the Data Distribution

Yeongmin Kim, HeeSun Bae, Byeonghu Na et al.

NeurIPS 2025arXiv:2505.09496

#75

REINFORCEMENT LEARNING FOR INDIVIDUAL OPTIMAL POLICY FROM HETEROGENEOUS DATA

Rui Miao, Babak Shahbaba, Annie Qu

offline reinforcement learningheterogeneous dataindividualized policy optimizationmarkov decision processes+4

NeurIPS 2025arXiv:2505.18524

#76

metaTextGrad: Automatically optimizing language model optimizers

Guowei Xu, Mert Yuksekgonul, Carlos Guestrin et al.

ICLR 2025arXiv:2410.05655

#77

Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning

Claire Chen, Shuze Liu, Shangtong Zhang

NeurIPS 2025arXiv:2506.12994

#78

Differentially Private Bilevel Optimization: Efficient Algorithms with Near-Optimal Rates

Andrew Lowy, Daogao Liu

differential privacybilevel optimizationhyperparameter optimizationmeta-learning+4

NeurIPS 2025arXiv:2501.18962

#79

Pareto Prompt Optimization

Guang Zhao, Byung-Jun Yoon, Gilchan Park et al.

Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrapping

Pu Yang, Yunzhen Feng, Ziyuan Chen et al.

synthetic data generationbudget allocation strategiesdiffusion probabilistic modelsmath reasoning+3

ICLR 2025arXiv:2408.00751

#81

Constructing an Optimal Behavior Basis for the Option Keyboard

Lucas N. Alegre, Ana Bazzan, Andre Barreto et al.

A Policy-Gradient Approach to Solving Imperfect-Information Games with Best-Iterate Convergence

Mingyang Liu, Gabriele Farina, Asuman Ozdaglar

policy gradient methodsimperfect-information gamesextensive-form gamescounterfactual values+3

NeurIPS 2025arXiv:2507.08068

#83

Feasibility-Aware Decision-Focused Learning for Predicting Parameters in the Constraints

Jayanta Mandi, Marianne Defresne, Senne Berden et al.

PROFIT: A Specialized Optimizer for Deep Fine Tuning

Anirudh Chakravarthy, Shuai Zheng, Xin Huang et al.

Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions

Simon Matrenok, Skander Moalla, Caglar Gulcehre

policy optimizationreward modelingkl-regularized rloffline alignment+3

ICLR 2025arXiv:2410.11820

#86

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF

Shicong Cen, Jincheng Mei, Katayoon Goshvadi et al.

Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws

Yiding Jiang, Allan Zhou, Zhili Feng et al.

pretraining data compositiondata distribution optimizationscaling lawsfoundation models+3

NeurIPS 2025arXiv:2511.10859

#88

Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form

Toshinori Kitamura, Tadashi Kozuno, Wataru Kumagai et al.

Fat-to-Thin Policy Optimization: Offline Reinforcement Learning with Sparse Policies

Lingwei Zhu, Han Wang, Yukie Nagai

Private Zeroth-Order Optimization with Public Data

Xuchen Gong, Tian Li

differentially private optimizationzeroth-order methodsgradient approximationprivacy/utility tradeoff+3

NeurIPS 2025arXiv:2510.18353

#91

On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning

Anas Barakat, Souradip Chakraborty, Peihong Yu et al.

Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback

Yi-Lun Wu, Bo-Kai Ruan, Chiang Tseng et al.

preference optimizationdiffusion modelsimplicit user feedbackranking problem+4

NeurIPS 2025arXiv:2504.14882

#93

Policy Gradient Methods Converge Globally in Imperfect-Information Extensive-Form Games

Fivos Kalogiannis, Gabriele Farina

Bilevel Optimization for Adversarial Learning Problems: Sharpness, Generation, and Beyond

Risheng Liu, Zhu Liu, Weihao Mao et al.

Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness

Mojtaba Kolahdouzi, Hatice Gunes, Ali Etemad

optimization algorithmsgroup fairnessadaptive optimizersstochastic optimizers+3

policy gradient methodsscaling function analysisasymptotic convergence guaranteessoftmax natural policy gradient+4

#96

$\phi$-Update: A Class of Policy Update Methods with Policy Convergence Guarantee

Wenye Li, Jiacai Liu, Ke Wei

ICLR 2025

NeurIPS 2025arXiv:2511.20906

#97

Dynamic Test-Time Compute Scaling in Control Policy: Difficulty-Aware Stochastic Interpolant Policy

Inkook Chun, Seungjae Lee, Michael Albergo et al.

diffusion-based policiesflow-based policiesrobotic manipulationimitation learning+4

#98

Preference-Guided Diffusion for Multi-Objective Offline Optimization

Yashas Annadani, Syrine Belakaria, Stefano Ermon et al.

Hessian Free Efficient Single Loop Iterative Differentiation Methods for Bi-Level Optimization Problems

Peiran Yu, Junyi Li, Heng Huang

Optimizing the Unknown: Black Box Bayesian Optimization with Energy-Based Model and Reinforcement Learning

RUIYAO MIAO, Junren Xiao, Shiya Tsang et al.

NeurIPS 2025