Parameter-Efficient Fine-Tuning

ICML 2024arXiv:2402.09353

#2

DoRA: Weight-Decomposed Low-Rank Adaptation

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin et al.

parameter-efficient fine-tuninglow-rank adaptationweight decomposition analysisdirectional updates+4

706

ICML 2024arXiv:2402.04333

#3

LESS: Selecting Influential Data for Targeted Instruction Tuning

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan et al.

instruction tuningdata selectioninfluence estimationgradient similarity search+4

400

ICML 2024arXiv:2402.12354

#4

LoRA+: Efficient Low Rank Adaptation of Large Models

Soufiane Hayou, Nikhil Ghosh, Bin Yu

low rank adaptationparameter efficient finetuninglarge language modelslearning rate optimization+3

341

ICLR 2024arXiv:2310.11454

#5

VeRA: Vector-based Random Matrix Adaptation

Dawid Kopiczko, Tijmen Blankevoort, Yuki Asano

276

ICLR 2025arXiv:2405.09673

#6

LoRA Learns Less and Forgets Less

Jonathan Frankle, Jose Javier Gonzalez Ortiz, Cody Blakeney et al.

low-rank adaptationparameter-efficient finetuninglarge language modelsinstruction finetuning+3

252

ICLR 2024arXiv:2402.17193

#7

When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

Biao Zhang, Zhongtao Liu, Colin Cherry et al.

238

ICLR 2024arXiv:2309.12307

#8

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

Yukang Chen, Shengju Qian, Haotian Tang et al.

235

ICLR 2024arXiv:2310.01352

#9

RA-DIT: Retrieval-Augmented Dual Instruction Tuning

Victoria Lin, Xilun Chen, Mingda Chen et al.

210

ICLR 2024arXiv:2310.08659

#10

LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models

Yixiao Li, Yifan Yu, Chen Liang et al.

202

ICLR 2024arXiv:2307.05695

#11

ReLoRA: High-Rank Training Through Low-Rank Updates

Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde et al.

187

ICML 2024arXiv:2404.14367

#12

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

Fahim Tajwar, Anikait Singh, Archit Sharma et al.

preference fine-tuninglarge language modelson-policy reinforcement learningcontrastive learning+4

179

ICLR 2024arXiv:2309.14717

#13

QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models

Yuhui Xu, Lingxi Xie, Xiaotao Gu et al.

156

ICLR 2024arXiv:2309.05444

#14

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning

Ted Zadouri, Ahmet Üstün, Arash Ahmadian et al.

143

ICLR 2024arXiv:2403.12313

#15

Improving LoRA in Privacy-preserving Federated Learning

Youbang Sun, Zitao Li, Yaliang Li et al.

141

ICLR 2024arXiv:2404.13628

#16

Mixture of LoRA Experts

xun wu, Shaohan Huang, Furu Wei

136

CVPR 2024arXiv:2404.00228

#17

InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning

Yan-Shuo Liang, Wu-Jun Li

126

ICML 2024arXiv:2402.11592

#18

Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark

Yihua Zhang, Pingzhi Li, Junyuan Hong et al.

zeroth-order optimizationmemory-efficient fine-tuninglarge language modelsback-propagation-free training+4

107

ICLR 2024arXiv:2310.17513

#19

The Expressive Power of Low-Rank Adaptation

Yuchen Zeng, Kangwook Lee

101

ICLR 2024arXiv:2402.14811

#20

Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking

Nikhil Prakash, Tamar Shaham, Tal Haklay et al.

99

ICLR 2024arXiv:2311.06243

#21

Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization

Weiyang Liu, Zeju Qiu, Yao Feng et al.

98

ECCV 2024arXiv:2403.05231

#22

Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance

Liting Lin, Heng Fan, Zhipeng Zhang et al.

visual object trackingparameter-efficient fine-tuningvision transformer backbonelora adaptation+4

97

ICLR 2024arXiv:2311.12786

#23

Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks

Samyak Jain, Robert Kirk, Ekdeep Singh Lubana et al.

97

ICML 2024arXiv:2402.03293

#24

Flora: Low-Rank Adapters Are Secretly Gradient Compressors

Yongchang Hao, Yanshuai Cao, Lili Mou

low-rank adaptationgradient compressionoptimization statesrandom projection+3

96

ICLR 2024arXiv:2308.13111

#25

Bayesian Low-rank Adaptation for Large Language Models

Adam Yang, Maxime Robeyns, Xi Wang et al.

92

ICLR 2024arXiv:2311.12023

#26

LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

Han Guo, Philip Greengard, Eric Xing et al.

85

AAAI 2024arXiv:2312.08733

#27

VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding

Yi Xin, Junlong Du, Qiang Wang et al.

parameter-efficient transfer learningmulti-task adaptationdense scene understandingvision transformer adapter+2

85

ICML 2024arXiv:2402.05119

#28

A Closer Look at the Limitations of Instruction Tuning

Sreyan Ghosh, Chandra Kiran Evuru, Sonal Kumar et al.

instruction tuninglarge language modelsknowledge degradationhallucination mitigation+3

83

ICLR 2024arXiv:2401.17868

#29

Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model

Zihan Zhong, Zhiqiang Tang, Tong He et al.

81

ECCV 2024arXiv:2403.19522

#30

Model Stock: All we need is just a few fine-tuned models

Dong-Hwan Jang, Sangdoo Yun, Dongyoon Han

fine-tuning methodsweight space analysismodel averagingout-of-distribution generalization+3

77

ICLR 2025arXiv:2308.13320

#31

Fine-tuning can cripple your foundation model; preserving features may be the solution

Philip Torr, Puneet Dokania, Jishnu Mukhoti et al.

76

ICLR 2024arXiv:2310.08915

#32

Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs

Yuxin Zhang, Lirui Zhao, Mingbao Lin et al.

74

ICML 2024arXiv:2402.05445

#33

Accurate LoRA-Finetuning Quantization of LLMs via Information Retention

Haotong Qin, Xudong Ma, Xingyu Zheng et al.

low-rank adaptationmodel quantizationinformation retentionlarge language models+2

74

NEURIPS 2025arXiv:2410.21228

#34

LoRA vs Full Fine-tuning: An Illusion of Equivalence

Reece Shuttleworth, Jacob Andreas, Antonio Torralba et al.

low-rank adaptationspectral properties analysissingular value decompositioncontinual learning+4

70

ICML 2024arXiv:2402.16842

#35

Asymmetry in Low-Rank Adapters of Foundation Models

Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi et al.

low-rank adaptationparameter-efficient fine-tuningfoundation modelsadapter asymmetry+2

68

ICLR 2025arXiv:2407.10490

#36

Learning Dynamics of LLM Finetuning

YI REN, Danica Sutherland

learning dynamicsinstruction tuningpreference tuningdirect preference optimization+4

67

ICLR 2024arXiv:2310.12962

#37

An Emulator for Fine-tuning Large Language Models using Small Language Models

Eric Mitchell, Rafael Rafailov, Archit Sharma et al.

66

ICML 2024arXiv:2309.10019

#38

Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts

Jiang-Xin Shi, Tong Wei, Zhi Zhou et al.

long-tail learningfoundation modelsfine-tuning paradigmclass imbalance+2

66

CVPR 2024arXiv:2312.12730

#39

A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models

Julio Silva-Rodríguez, Sina Hajimiri, Ismail Ben Ayed et al.

few-shot adaptationvision-language modelsefficient transfer learninglinear probing+4

65

ICML 2024arXiv:2406.20053

#40

Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation

Danny Halawi, Alexander Wei, Eric Wallace et al.

black-box finetuningmodel safetymalicious adaptationevasion detection+4

65

ICML 2024arXiv:2312.06353

#41

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes

Zhen Qin, Daoyuan Chen, Bingchen Qian et al.

federated learninglarge language modelsfull-parameter tuningzeroth-order optimization+3

61

ICML 2024arXiv:2405.03003

#42

Parameter-Efficient Fine-Tuning with Discrete Fourier Transform

Ziqi Gao, Qichao Wang, Aochuan Chen et al.

parameter-efficient fine-tuninglow-rank adaptationfourier transform compressionspectral coefficient learning+3

60

ICML 2024arXiv:2404.01650

#43

Test-Time Model Adaptation with Only Forward Passes

Shuaicheng Niu, Chunyan Miao, Guohao Chen et al.

test-time adaptationforward pass optimizationcovariance matrix adaptationmodel quantization+4

60

ICML 2024arXiv:2405.11157

#44

Towards Modular LLMs by Building and Reusing a Library of LoRAs

Oleksiy Ostapenko, Zhan Su, Edoardo Ponti et al.

parameter-efficient adaptersmulti-task learningzero-shot routingadapter library+4

58

ICLR 2024arXiv:2302.06605

#45

UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

Haoyu Lu, Yuqi Huo, Guoxing Yang et al.

parameter-efficient transfer learningcross-modal modelingvision-language pre-trained modelsmultimodal adapters+4

55

ICLR 2024arXiv:2312.11420

#46

Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning

Bingchen Zhao, Haoqin Tu, Chen Wei et al.

55

ICLR 2025arXiv:2410.19735

#47

Model merging with SVD to tie the Knots

George Stoica, Pratik Ramesh, Boglarka Ecsedi et al.

model merginglora finetuningsvd transformationparameter alignment+3

55

ICLR 2025arXiv:2407.18242

#48

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

Zhengbo Wang, Jian Liang, Ran He et al.

low-rank adaptationparameter-efficient fine-tuningfoundation modelsgradient approximation+3

54

ICLR 2025arXiv:2410.07471

#49

SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

Han Shen, Pin-Yu Chen, Payel Das et al.

54

ICML 2024arXiv:2407.05872

#50

Scaling Exponents Across Parameterizations and Optimizers

Katie Everett, Lechao Xiao, Mitchell Wortsman et al.

scaling exponentsparameterization methodsoptimizer designlearning rate scaling+4

51

AAAI 2024arXiv:2304.01917

#51

Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning

Samyadeep Basu, Shell Hu, Daniela Massiceti et al.

few-shot classificationvision transformer fine-tuningparameter efficient fine-tuninglayer norm tuning+4

50

CVPR 2024arXiv:2403.01439

#52

Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis

Xin Zhou, Dingkang Liang, Wei Xu et al.

49

ICLR 2025arXiv:2412.15287

#53

Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models

Yinlam Chow, Guy Tennenholtz, Izzeddin Gur et al.

inference-time computebest-of-n samplingimitation learningreinforcement learning+4

49

ICML 2024arXiv:2402.12048

#54

Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack

Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi et al.

Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models

Didi Zhu, Zhongyi Sun, Zexi Li et al.

catastrophic forgettingmulti-modal large language modelspost-training adjustmentparameter-efficient fine-tuning+3

48

ICML 2024arXiv:2401.04679

#56

RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation

Mahdi Nikdan, Soroush Tabesh, Elvir Crnčević et al.

parameter-efficient fine-tuninglarge language modelsrobust principal component analysislow-rank adaptation+4

48

CVPR 2024arXiv:2312.10136

#57

Gradient-based Parameter Selection for Efficient Fine-Tuning

Zhi Zhang, Qizhe Zhang, Zijun Gao et al.

45

ICLR 2024arXiv:2211.00635

#58

Two-stage LLM Fine-tuning with Less Specialization and More Generalization

Yihan Wang, Si Si, Daliang Li et al.

large language modelsfine-tuningin-context learningformat specialization+4

43

CVPR 2025arXiv:2408.08345

#59

5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks

Dongshuo Yin, Leiyi Hu, Bin Li et al.

43

ICLR 2024arXiv:2309.05173

#60

DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning

Zhengxiang Shi, Aldo Lipani

41

AAAI 2024arXiv:2308.08090

#61

Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation

Xinshuo Hu, Dongfang Li, Zihao Zheng et al.

40

ICML 2024arXiv:2404.08080

#62

Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models

Tanmay Gautam, Youngsuk Park, Hao Zhou et al.

zeroth-order optimizationlanguage model fine-tuningvariance reduction techniquesmemory-efficient optimization+4

39

ICLR 2024arXiv:2310.19698

#63

When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations

Aleksandar Petrov, Philip Torr, Adel Bibi

39

ICLR 2024arXiv:2310.05905

#64

TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models

Zuxin Liu, Jesse Zhang, Kavosh Asadi et al.

39

NEURIPS 2025arXiv:2402.15751

#65

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning

Yong Liu, Zirui Zhu, Chaoyu Gong et al.

zeroth-order optimizationmemory-efficient fine-tuninglarge language modelsparameter-efficient fine-tuning+3

37

ICLR 2025arXiv:2501.01765

#66

SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation

Mingjie Li, Wai Man Si, Michael Backes et al.

37

AAAI 2024arXiv:2308.14604

#67

SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction

Zelin Peng, Zhengqin Xu, Zhilin Zeng et al.

parameter space reconstructionobject segmentationsegment anything modelmatrix decomposition+2

37

ICLR 2025arXiv:2409.16167

#68

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

Ziyu Zhao, tao shen, Didi Zhu et al.

low-rank adaptationparameter clusteringmodel mergingmodular fine-tuning+3

35

ICML 2024arXiv:2402.11867

#69

LoRA Training in the NTK Regime has No Spurious Local Minima

Uijeong Jang, Jason Lee, Ernest Ryu

low-rank adaptationneural tangent kernelparameter-efficient fine-tuninglarge language models+3

35

ICML 2024arXiv:2402.02347

#70

Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models

Fangzhao Zhang, Mert Pilanci

low-rank adaptationparameter-efficient fine-tuningriemannian optimizationfoundation model fine-tuning+3

35

AAAI 2024arXiv:2310.03059

#71

Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models

Yiwen Tang, Ray Zhang, Zoey Guo et al.

parameter-efficient fine-tuning3d pre-trained modelspoint cloud adaptationgeometry-aware adapter+3

ICLR 2025arXiv:2501.13198

#72

SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning

Yichen Wu, Hongming Piao, Long-Kai Huang et al.

class incremental learninglow-rank adaptationcontinual learningfoundation models+3

ICLR 2025arXiv:2410.08020

#73

Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs

Jonas Hübotter, Sascha Bongni, Ido Hakimi et al.

ICLR 2025arXiv:2406.03136

#74

Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models

Jerry Yao-Chieh Hu, Maojiang Su, En-Jui Kuo et al.

ICLR 2025arXiv:2412.13337

#75

Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang et al.

supervised fine-tuninginstruction-tuning datasetssmall language modelstraining configurations+4

ICLR 2024arXiv:2405.02774

#76

Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs

Feiyang Kang, Hoang Anh Just, Yifan Sun et al.

32

AAAI 2024arXiv:2305.10329

#77

G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks

Anchun Gui, Jinqiang Ye, Han Xiao

parameter-efficient fine-tuninggraph transformer networksgraph convolution operationfeature distribution shift+4

31

ICLR 2024arXiv:2312.05677

#78

Batched Low-Rank Adaptation of Foundation Models

Yeming Wen, Swarat Chaudhuri

29

ICML 2024arXiv:2402.02314

#79

Selecting Large Language Model to Fine-tune via Rectified Scaling Law

Haowei Lin, Baizhou Huang, Haotian Ye et al.

scaling lawsfine-tuning performance predictionlarge language model selectionpre-learned data size+3

29

ICLR 2025arXiv:2410.07698

#80

Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures

Yiming Chen, Yuan Zhang, Liyuan Cao et al.

ICLR 2024arXiv:2310.02556

#81

NOLA: Compressing LoRA using Linear Combination of Random Basis

Soroush Abbasi Koohpayegani, K L Navaneet, Parsa Nooralinejad et al.

AAAI 2025arXiv:2412.12497

#82

NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning

Xin Yi, Shunfan Zheng, Linlin Wang et al.

ICML 2024arXiv:2401.12200

#83

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference

Bowen Zhao, Hannaneh Hajishirzi, Qingqing Cao

parameter-efficient fine-tuningstructured pruninglanguage model compressionadaptive parameter tuning+4

ICLR 2024arXiv:2310.05055

#84

FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis

Raman Dutt, Ondrej Bohdal, Sotirios Tsaftaris et al.

27

ECCV 2024arXiv:2403.09192

#85

PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation

Yizhe Xiong, Hui Chen, Tianxiang Hao et al.

parameter-efficient fine-tuningmodel compressiontraining-inference efficiencytoken merging+3

26

ICML 2024arXiv:2402.02868

#86

Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem

Maciej Wołczyk, Bartłomiej Cupiał, Mateusz Ostaszewski et al.

reinforcement learningfine-tuningforgetting mitigationknowledge retention+4

26

ICML 2024arXiv:2406.04112

#87

Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation

Can Yaras, Peng Wang, Laura Balzano et al.

low-rank matrix completionoverparameterized modelscompressible learning dynamicslow-rank adaptation+4

25

ICLR 2025arXiv:2410.20672

#88

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

Sangmin Bae, Adam Fisch, Hrayr Harutyunyan et al.

25

large language modelsmodel routingpreference learningcost-performance trade-off+3

#89

RouteLLM: Learning to Route LLMs from Preference Data

Isaac Ong, Amjad Almahairi, Vincent Wu et al.

ICLR 2025

24

AAAI 2024arXiv:2304.09595

#90

AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNs

Shengrui Li, Xueting Han, Jing Bai

23

CVPR 2024arXiv:2402.02887

#91

Time- Memory- and Parameter-Efficient Visual Adaptation

Otniel-Bogdan Mercea, Alexey Gritsenko, Cordelia Schmid et al.

22

CVPR 2024arXiv:2403.19067

#92

Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach

Wei Dong, Xing Zhang, Bihui Chen et al.

22

ICML 2024arXiv:2310.09639

#93

DPZero: Private Fine-Tuning of Language Models without Backpropagation

Liang Zhang, Bingcong Li, Kiran Thekumparampil et al.

differentially private optimizationzeroth-order methodslanguage model fine-tuningmemory-efficient training+2

22

CVPR 2024arXiv:2302.06637

#94

PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees

Chulin Xie, De-An Huang, Wenda Chu et al.

ECCV 2024arXiv:2311.11227

#95

FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients

Shangchao Su, Bin Li, Xiangyang Xue

federated learningfoundation modelsheterogeneous clientstransformer-based models+3

ICML 2025arXiv:2407.00066

#96

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead

Rickard Gabrielsson, Jiacheng Zhu, Onkar Bhardwaj et al.

AAAI 2025arXiv:2501.08580

#97

Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation

Jiaqi Huang, Zunnan Xu, Ting Liu et al.

referring image segmentationparameter-efficient tuningmultimodal feature alignmentmisaligned encoders+4

AAAI 2024arXiv:2403.06208

#98

Personalized LoRA for Human-Centered Text Understanding

You Zhang, Jin Wang, Liang-Chih Yu et al.

20

NEURIPS 2025arXiv:2505.11711

#99

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur et al.

20

NEURIPS 2025arXiv:2506.05316

#100

Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay

Yifan Sun, Jingyan Shen, Yibin Wang et al.

20