Parameter-Efficient Fine-Tuning

ICLR 2024arXiv:2307.05695

#3

ReLoRA: High-Rank Training Through Low-Rank Updates

Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde et al.

179

ICLR 2024arXiv:2309.05444

#4

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning

Ted Zadouri, Ahmet Üstün, Arash Ahmadian et al.

138

CVPR 2024arXiv:2404.00228

#5

InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning

Yan-Shuo Liang, Wu-Jun Li

117

ICLR 2024arXiv:2402.14811

#6

Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking

Nikhil Prakash, Tamar Shaham, Tal Haklay et al.

97

ICLR 2024arXiv:2311.12786

#7

Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks

Samyak Jain, Robert Kirk, Ekdeep Singh Lubana et al.

89

ECCV 2024arXiv:2403.05231

#8

Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance

Liting Lin, Heng Fan, Zhipeng Zhang et al.

visual object trackingparameter-efficient fine-tuningvision transformer backbonelora adaptation+4

89

AAAI 2024arXiv:2312.08733

#9

VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding

Yi Xin, Junlong Du, Qiang Wang et al.

parameter-efficient transfer learningmulti-task adaptationdense scene understandingvision transformer adapter+2

82

ECCV 2024arXiv:2403.19522

#10

Model Stock: All we need is just a few fine-tuned models

Dong-Hwan Jang, Sangdoo Yun, Dongyoon Han

71

ICLR 2025arXiv:2407.10490

#11

Fine-tuning can cripple your foundation model; preserving features may be the solution

Philip Torr, Puneet Dokania, Jishnu Mukhoti et al.

Learning Dynamics of LLM Finetuning

YI REN, Danica Sutherland

61

ICLR 2024arXiv:2302.06605

#13

UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

Haoyu Lu, Yuqi Huo, Guoxing Yang et al.

54

ICLR 2025arXiv:2407.18242

#14

Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack

Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi et al.

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

Zhengbo Wang, Jian Liang, Ran He et al.

47

ICLR 2025arXiv:2410.19735

#16

Model merging with SVD to tie the Knots

George Stoica, Pratik Ramesh, Boglarka Ecsedi et al.

46

ICLR 2025arXiv:2412.15287

#17

Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models

Yinlam Chow, Guy Tennenholtz, Izzeddin Gur et al.

43

ICLR 2024arXiv:2211.00635

#18

Two-stage LLM Fine-tuning with Less Specialization and More Generalization

Yihan Wang, Si Si, Daliang Li et al.

42

ICLR 2024arXiv:2309.05173

#19

DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning

Zhengxiang Shi, Aldo Lipani

41

ICLR 2024arXiv:2310.19698

#20

When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations

Aleksandar Petrov, Philip Torr, Adel Bibi

38

CVPR 2024arXiv:2404.02478

#21

FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning

Rishub Tamirisa, Chulin Xie, Wenxuan Bao et al.

36

ICLR 2025arXiv:2409.16167

#22

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

Ziyu Zhao, tao shen, Didi Zhu et al.

low-rank adaptationparameter clusteringmodel mergingmodular fine-tuning+3

34

AAAI 2024arXiv:2310.03059

#23

Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models

Jerry Yao-Chieh Hu, Maojiang Su, En-Jui Kuo et al.

Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models

Yiwen Tang, Ray Zhang, Zoey Guo et al.

parameter-efficient fine-tuning3d pre-trained modelspoint cloud adaptationgeometry-aware adapter+3

33

AAAI 2024arXiv:2305.10329

#25

G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks

Anchun Gui, Jinqiang Ye, Han Xiao

parameter-efficient fine-tuninggraph transformer networksgraph convolution operationfeature distribution shift+4

31

CVPR 2024arXiv:2308.14316

#26

Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models

Zhejun Zhang, Peter Karkus, Maximilian Igl et al.

UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory

Haiwen Diao, Bo Wan, Ying Zhang et al.

27

ICLR 2025arXiv:2408.15313

#28

RouteLLM: Learning to Route LLMs from Preference Data

Isaac Ong, Amjad Almahairi, Vincent Wu et al.

Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

Wenxuan Zhang, Philip Torr, Mohamed Elhoseiny et al.

23

CVPR 2024arXiv:2302.06637

#30

PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees

Chulin Xie, De-An Huang, Wenda Chu et al.

20

ICLR 2024arXiv:2309.13192

#31

Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation

Kai Huang, Hanyun Yin, Heng Huang et al.

17

ICLR 2024arXiv:2507.03003

#32

A Quantum Circuit-Based Compression Perspective for Parameter-Efficient Learning

Chen-Yu Liu, Chao-Han Huck Yang, Hsi-Sheng Goan et al.

Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages

Wanru Zhao, Yihong Chen, Royson Lee et al.

16

ICLR 2025arXiv:2410.20625

#34

LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization

Jui-Nan Yen, Si Si, Zhao Meng et al.

15

ICLR 2025arXiv:2405.13053

#35

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur et al.

Unified Parameter-Efficient Unlearning for LLMs

Chenlu Ding, Jiancan Wu, Yancheng Yuan et al.

MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models

Jingwei Xu, Junyu Lai, Yunpeng Huang

13

AAAI 2025arXiv:2410.09437

#38

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning

Yaming Yang, Dilxat Muhtar, Yelong Shen et al.

13

CVPR 2025arXiv:2501.06655

#39

Personalized Preference Fine-tuning of Diffusion Models

Meihua Dang, Anikait Singh, Linqi Zhou et al.

13

CVPR 2025arXiv:2412.09622

#40

Robust Test-Time Adaptation for Zero-Shot Prompt Tuning

Ding-Chu Zhang, Zhi Zhou, Yufeng Li

Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation

Yibo Wang, Tiansheng Huang, Li Shen et al.

LoRACLR: Contrastive Adaptation for Customization of Diffusion Models

Enis Simsar, Thomas Hofmann, Federico Tombari et al.

11

ICLR 2025arXiv:2504.02620

#43

Efficient Model Editing with Task-Localized Sparse Fine-tuning

Leonardo Iurada, Marco Ciccone, Tatiana Tommasi

10

CVPR 2024arXiv:2404.06244

#44

AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning

Yehonathan Refael, Jonathan Svirsky, Boris Shustin et al.

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning

Qitao Tan, Jun Liu, Zheng Zhan et al.

Anchor-based Robust Finetuning of Vision-Language Models

Jinwei Han, Zhiwen Lin, Zhongyisun Sun et al.

10

ICML 2025arXiv:2502.01235

#47

MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards

Sheng Wang, Liheng Chen, Pengan CHEN et al.

Quantum-PEFT: Ultra parameter-efficient fine-tuning

Toshiaki Koike-Akino, Francesco Tonin, Yongtao Wu et al.

DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models

Wenlong Deng, Yize Zhao, Vala Vakilian et al.

LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently

Yuanhe Zhang, Fanghui Liu, Yudong Chen

8

ICLR 2025arXiv:2411.05877

#51

HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA Experts

Mengqi Liao, Wei Chen, Junfeng Shen et al.

Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

Tong Chen, Hao Fang, Patrick Xia et al.

language model adaptationlow-rank adaptersself-supervised learningknowledge acquisition+4

NeurIPS 2025arXiv:2502.17159

#53

RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness

Fanhu Zeng, Haiyang Guo, Fei Zhu et al.

ICLR 2025arXiv:2504.14662

#54

Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning

Yeoreum Lee, Jinwook Jung, Sungyong Baik

CVPR 2025arXiv:2409.16434

#55

Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition

Zheda Mai, Ping Zhang, Cheng-Hao Tu et al.

parameter-efficient fine-tuningvision transformerslow-shot learningdistribution shift robustness+4

ICCV 2025arXiv:2507.22872

#56

Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation

Zelin Peng, Zhengqin Xu, Zhilin Zeng et al.

TR-PTS: Task-Relevant Parameter and Token Selection for Efficient Tuning

Siqi Luo, Haoran Yang, Yi Xin et al.

ICML 2025arXiv:2410.09016

#58

SMT: Fine-Tuning Large Language Models with Sparse Matrices

Haoze He, Juncheng Li, Xuan Jiang et al.

Parameter-Efficient Fine-Tuning of State Space Models

Kevin Galim, Wonjun Kang, Yuchen Zeng et al.

ICML 2025arXiv:2410.05026

#60

Active Fine-Tuning of Multi-Task Policies

Marco Bagatella, Jonas Hübotter, Georg Martius et al.

ICLR 2025arXiv:2503.18225

#61

Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models

Jun Zhang, Jue Wang, Huan Li et al.

Decoupling Angles and Strength in Low-rank Adaptation

Massimo Bini, Leander Girrbach, Zeynep Akata

parameter-efficient finetuninglow-rank adaptationsubject-driven image generationnatural language understanding+4

AAAI 2025arXiv:2412.12865

#63

Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models

Yuchen Fan, Yuzhong Hong, Qiushi Wang et al.

5

ICLR 2025arXiv:2501.16559

#64

Efficient Hyperparameter Optimization with Adaptive Fidelity Identification

Jiantong Jiang, Zeyi Wen, Atif Mansoor et al.

LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation

Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse et al.

parameter-efficient fine-tuninglow-rank adaptationcross-model adaptationfoundation models+4

5

ICLR 2025arXiv:2502.00089

#66

Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation

Yi-Chen Li, Fuxiang Zhang, Wenjie Qiu et al.

Zeroth-Order Fine-Tuning of LLMs with Transferable Static Sparsity

Wentao Guo, Jikai Long, Yimeng Zeng et al.

KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models

Fan Wang, Juyong Jiang, Chansung Park et al.

Ensembles of Low-Rank Expert Adapters

Yinghao Li, Vianne Gao, Chao Zhang et al.

5

ICLR 2025arXiv:2407.06172

#70

GoRA: Gradient-driven Adaptive Low Rank Adaptation

haonan he, Peng Ye, Yuchen Ren et al.

Towards Robust Parameter-Efficient Fine-Tuning for Federated Learning

Xiuwen Fang, Mang Ye

On Speeding Up Language Model Evaluation

Jin Zhou, Christian Belardi, Ruihan Wu et al.

ICCV 2025arXiv:2508.00230

#73

STAFF: Speculative Coreset Selection for Task-Specific Fine-tuning

Xiaoyu Zhang, Juan Zhai, Shiqing Ma et al.

Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration

Dongwon Park, Hayeon Kim, Se Young Chun

Towards Higher Effective Rank in Parameter-Efficient Fine-tuning using Khatri-Rao Product

Paul Albert, Frederic Zhang, Hemanth Saratchandran et al.

parameter-efficient fine-tuninglow-rank adaptationkhatri-rao producteffective rank+4

ICLR 2025arXiv:2410.10880

#76

Fine-tuning can Help Detect Pretraining Data from Large Language Models

Hengxiang Zhang, Songxin Zhang, Bingyi Jing et al.

pretraining data detectionlarge language modelsfine-tuning methodsperplexity scoring+3

AAAI 2025arXiv:2502.10705

#77

CoPEFT: Fast Adaptation Framework for Multi-Agent Collaborative Perception with Parameter-Efficient Fine-Tuning

Quanmin Wei, Penglin Dai, Wei Li et al.

AAAI 2025arXiv:2501.01874

#78

DFF: Decision-Focused Fine-Tuning for Smarter Predict-Then-Optimize with Limited Data

Jiaqi Yang, Enming Liang, Zicheng Su et al.

AAAI 2025arXiv:2406.10181

#79

Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors

Siyuan Chen, Zhuofeng Wang, Zelong Guan et al.

ICCV 2025arXiv:2507.10855

#80

Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation

Tianran Chen, Jiarui Chen, Baoquan Zhang et al.

Sparse Fine-Tuning of Transformers for Generative Tasks

Wei Chen, Jingxi Yu, Zichen Miao et al.

ICML 2025arXiv:2409.06277

#82

BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning

Hao Zhu, Yifei Zhang, Junhao Dong et al.

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models

Yao Shu, Wenyang Hu, See-Kiong Ng et al.

ICML 2025arXiv:2505.12871

#84

Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?

Zi Liang, Haibo Hu, Qingqing Ye et al.

NeurIPS 2025arXiv:2506.16406

#85

Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights

Zhiyuan Liang, Dongwen Tang, Yuhao Zhou et al.

ICLR 2025arXiv:2410.10870

#86

PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches

Rana Muhammad Shahroz Khan, Pingzhi Li, Sukwon Yun et al.

large language modelsparameter-efficient fine-tuningmodel personalizationtraining-free adaptation+4

NeurIPS 2025arXiv:2506.00799

#87

LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning

Zhekai Du, Yinjie Min, Jingjing Li et al.

Uni-LoRA: One Vector is All You Need

Kaiyang Li, Shaobo Han, Qing Su et al.

parameter-efficient fine-tuninglow-rank adaptationlarge language modelstrainable parameter reduction+2

NeurIPS 2025arXiv:2510.27172

#89

Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler

Zixuan Hu, Li Shen, Zhenyi Wang et al.

harmful fine-tuning defensebayesian inferencesafety attribute learningdata scheduling+4

NeurIPS 2025arXiv:2504.06426

#90

S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning

Hanqing Zeng, Yinglong Xia, Zhuokai Zhao et al.

parameter-efficient fine-tuninglow-rank adaptationmixture-of-expertslarge language models+4

NeurIPS 2025arXiv:2506.08001

#91

Reparameterized LLM Training via Orthogonal Equivalence Transformation

Zeju Qiu, Simon Buchholz, Tim Xiao et al.

parameter-efficient fine-tuninglora tuningnoise reductioninstruction tuning+4

#92

Fine-tuning with Reserved Majority for Noise Reduction

Shuyang Jiang, Yusheng Liao, Ya Zhang et al.

ICLR 2025

NeurIPS 2025arXiv:2505.13515

#93

LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades

Yanan Li, Fanxu Meng, Muhan Zhang et al.

ICML 2025arXiv:2506.01901

#94

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods

Yifan HAO, xingyuan pan, Hanning Zhang et al.

ICML 2025arXiv:2512.21708

#95

MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles

Jing Han, Binwei Yan, Tianyu Guo et al.

AAAI 2024arXiv:2401.10962

#96

One Step Learning, One Step Review

Huang Xiaolong, Qiankun Li, Xueran Li et al.

visual fine-tuningknowledge forgettingweight rollbackimage classification+4

NeurIPS 2025arXiv:2510.01938

#97

PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning

Zongqian Li, Yixuan Su, Nigel Collier

LoRAverse: A Submodular Framework to Retrieve Diverse Adapters for Diffusion Models

Mert Sonmezer, Matthew Zheng, Pinar Yanardag

An Optimal Control View of LoRA and Binary Controller Design for Vision Transformers

CHI Zhang, Jingpu Cheng, Qianxiao Li

StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold

Zhizhong Li, Sina Sajadmanesh, Jingtao Li et al.

low-rank adaptationparameter-efficient fine-tuningstiefel manifold optimizationsubspace learning+4

1