Parameter-Efficient Fine-Tuning

ICLR 2024arXiv:2310.08659

#2

LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models

Yixiao Li, Yifan Yu, Chen Liang et al.

194

ICLR 2024arXiv:2307.05695

#3

ReLoRA: High-Rank Training Through Low-Rank Updates

Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde et al.

179

ICLR 2024arXiv:2309.05444

#4

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning

Ted Zadouri, Ahmet Üstün, Arash Ahmadian et al.

138

CVPR 2024arXiv:2404.00228

#5

InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning

Yan-Shuo Liang, Wu-Jun Li

117

ICLR 2024arXiv:2402.14811

#6

Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking

Nikhil Prakash, Tamar Shaham, Tal Haklay et al.

97

ECCV 2024arXiv:2403.05231

#7

Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance

Liting Lin, Heng Fan, Zhipeng Zhang et al.

visual object trackingparameter-efficient fine-tuningvision transformer backbonelora adaptation+4

92

ICLR 2024arXiv:2311.12786

#8

Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks

Samyak Jain, Robert Kirk, Ekdeep Singh Lubana et al.

89

AAAI 2024arXiv:2312.08733

#9

VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding

Yi Xin, Junlong Du, Qiang Wang et al.

parameter-efficient transfer learningmulti-task adaptationdense scene understandingvision transformer adapter+2

82

ECCV 2024arXiv:2403.19522

#10

Model Stock: All we need is just a few fine-tuned models

Dong-Hwan Jang, Sangdoo Yun, Dongyoon Han

fine-tuning methodsweight space analysismodel averagingout-of-distribution generalization+3

71

ICLR 2025arXiv:2308.13320

#11

Fine-tuning can cripple your foundation model; preserving features may be the solution

Philip Torr, Puneet Dokania, Jishnu Mukhoti et al.

70

CVPR 2024arXiv:2312.12730

#12

A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models

Julio Silva-Rodríguez, Sina Hajimiri, Ismail Ben Ayed et al.

61

ICLR 2025arXiv:2407.10490

#13

Learning Dynamics of LLM Finetuning

YI REN, Danica Sutherland

learning dynamicsinstruction tuningpreference tuningdirect preference optimization+4

61

NEURIPS 2025arXiv:2410.21228

#14

LoRA vs Full Fine-tuning: An Illusion of Equivalence

Reece Shuttleworth, Jacob Andreas, Antonio Torralba et al.

low-rank adaptationspectral properties analysissingular value decompositioncontinual learning+4

58

ICLR 2024arXiv:2302.06605

#15

UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

Haoyu Lu, Yuqi Huo, Guoxing Yang et al.

54

ICLR 2025arXiv:2407.18242

#16

Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack

Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi et al.

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

Zhengbo Wang, Jian Liang, Ran He et al.

low-rank adaptationparameter-efficient fine-tuningfoundation modelsgradient approximation+3

47

ICLR 2025arXiv:2410.19735

#18

Model merging with SVD to tie the Knots

George Stoica, Pratik Ramesh, Boglarka Ecsedi et al.

model merginglora finetuningsvd transformationparameter alignment+3

46

ICLR 2025arXiv:2412.15287

#19

Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models

Yinlam Chow, Guy Tennenholtz, Izzeddin Gur et al.

inference-time computebest-of-n samplingimitation learningreinforcement learning+4

43

ICLR 2024arXiv:2211.00635

#20

Two-stage LLM Fine-tuning with Less Specialization and More Generalization

Yihan Wang, Si Si, Daliang Li et al.

42

ICLR 2024arXiv:2309.05173

#21

DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning

Zhengxiang Shi, Aldo Lipani

41

CVPR 2025arXiv:2408.08345

#22

5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks

Dongshuo Yin, Leiyi Hu, Bin Li et al.

38

ICLR 2024arXiv:2310.19698

#23

When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations

Aleksandar Petrov, Philip Torr, Adel Bibi

38

NEURIPS 2025arXiv:2402.15751

#24

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning

Yong Liu, Zirui Zhu, Chaoyu Gong et al.

zeroth-order optimizationmemory-efficient fine-tuninglarge language modelsparameter-efficient fine-tuning+3

36

ICLR 2025arXiv:2406.03136

#25

Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models

Jerry Yao-Chieh Hu, Maojiang Su, En-Jui Kuo et al.

34

ICLR 2025arXiv:2409.16167

#26

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

Ziyu Zhao, tao shen, Didi Zhu et al.

low-rank adaptationparameter clusteringmodel mergingmodular fine-tuning+3

33

AAAI 2024arXiv:2310.03059

#27

Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models

Yiwen Tang, Ray Zhang, Zoey Guo et al.

parameter-efficient fine-tuning3d pre-trained modelspoint cloud adaptationgeometry-aware adapter+3

33

AAAI 2024arXiv:2305.10329

#28

G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks

Anchun Gui, Jinqiang Ye, Han Xiao

parameter-efficient fine-tuninggraph transformer networksgraph convolution operationfeature distribution shift+4

31

ICLR 2025arXiv:2412.13337

#29

Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang et al.

supervised fine-tuninginstruction-tuning datasetssmall language modelstraining configurations+4

30

ICLR 2025arXiv:2501.13198

#30

SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning

Yichen Wu, Hongming Piao, Long-Kai Huang et al.

class incremental learninglow-rank adaptationcontinual learningfoundation models+3

26

ECCV 2024arXiv:2403.09192

#31

PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation

Yizhe Xiong, Hui Chen, Tianxiang Hao et al.

parameter-efficient fine-tuningmodel compressiontraining-inference efficiencytoken merging+3

26

AAAI 2025arXiv:2412.12497

#32

NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning

Xin Yi, Shunfan Zheng, Linlin Wang et al.

25

#33

RouteLLM: Learning to Route LLMs from Preference Data

Isaac Ong, Amjad Almahairi, Vincent Wu et al.

large language modelsmodel routingpreference learningcost-performance trade-off+3

24

AAAI 2025arXiv:2501.08580

#34

Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation

Jiaqi Huang, Zunnan Xu, Ting Liu et al.

referring image segmentationparameter-efficient tuningmultimodal feature alignmentmisaligned encoders+4

20

ECCV 2024arXiv:2311.11227

#35

FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients

Shangchao Su, Bin Li, Xiangyang Xue

federated learningfoundation modelsheterogeneous clientstransformer-based models+3

20

CVPR 2024arXiv:2302.06637

#36

PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees

Chulin Xie, De-An Huang, Wenda Chu et al.

20

CVPR 2024arXiv:2403.19067

#37

Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach

Wei Dong, Xing Zhang, Bihui Chen et al.

19

ICLR 2024arXiv:2309.13192

#38

Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation

Kai Huang, Hanyun Yin, Heng Huang et al.

17

ICCV 2025arXiv:2411.14961

#39

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement

Jieming Bian, Lei Wang, Letian Zhang et al.

federated learningparameter-efficient fine-tuninglow-rank adaptationfoundation models+3

17

ICLR 2025arXiv:2410.09846

#40

A Quantum Circuit-Based Compression Perspective for Parameter-Efficient Learning

Chen-Yu Liu, Chao-Han Huck Yang, Hsi-Sheng Goan et al.

17

ICLR 2024arXiv:2507.03003

#41

Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages

Wanru Zhao, Yihong Chen, Royson Lee et al.

16

ICLR 2025arXiv:2410.20625

#42

LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization

Jui-Nan Yen, Si Si, Zhao Meng et al.

low-rank adaptationtransformation invariancematrix preconditioningparameter-efficient finetuning+3

15

NEURIPS 2025arXiv:2505.11711

#43

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur et al.

15

ICLR 2025arXiv:2412.00383

#44

Unified Parameter-Efficient Unlearning for LLMs

Chenlu Ding, Jiancan Wu, Yancheng Yuan et al.

14

ICLR 2025arXiv:2405.13053

#45

MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models

Jingwei Xu, Junyu Lai, Yunpeng Huang

parameter-efficient fine-tuninglow-rank adaptationmixture-of-experts architecturetask-specific adapters+4

13

AAAI 2025arXiv:2410.09437

#46

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning

Yaming Yang, Dilxat Muhtar, Yelong Shen et al.

13

CVPR 2025arXiv:2501.06655

#47

Personalized Preference Fine-tuning of Diffusion Models

Meihua Dang, Anikait Singh, Linqi Zhou et al.

diffusion modelspreference fine-tuningmulti-reward optimizationpersonalized generation+4

13

NEURIPS 2025arXiv:2501.18100

#48

Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation

Yibo Wang, Tiansheng Huang, Li Shen et al.

12

ICLR 2025arXiv:2503.00306

#49

Unlocking Efficient, Scalable, and Continual Knowledge Editing with Basis-Level Representation Fine-Tuning

Tianci Liu, Ruirui Li, Yunzhe Qi et al.

knowledge editinglarge language modelsrepresentation fine-tuningediting-locality trade-off+3

12

ICLR 2025arXiv:2410.02200

#50

Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts

Minh Le, Chau Nguyen, Huy Nguyen et al.

prefix-tuningprompt-based fine-tuningparameter reparameterizationmixture of experts+4

12

NEURIPS 2025arXiv:2410.04010

#51

Robust Test-Time Adaptation for Zero-Shot Prompt Tuning

Ding-Chu Zhang, Zhi Zhou, Yufeng Li

Hyperbolic Fine-Tuning for Large Language Models

Menglin Yang, Ram Samarth B B, Aosong Feng et al.

hyperbolic fine-tuningtoken embedding geometrynon-euclidean embeddingslatent tree-like structures+3

11

CVPR 2025arXiv:2503.16843

#53

LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models

Jian Liang, Wenke Huang, Guancheng Wan et al.

11

ICLR 2025arXiv:2410.00938

#54

MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards

Sheng Wang, Liheng Chen, Pengan CHEN et al.

ICLR 2025arXiv:2410.17881

#55

AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning

Yehonathan Refael, Jonathan Svirsky, Boris Shustin et al.

NEURIPS 2025arXiv:2502.03304

#56

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning

Qitao Tan, Jun Liu, Zheng Zhan et al.

ICLR 2025arXiv:2504.02620

#57

Efficient Model Editing with Task-Localized Sparse Fine-tuning

Leonardo Iurada, Marco Ciccone, Tatiana Tommasi

task arithmeticmodel editingsparse fine-tuningweight disentanglement+2

ICLR 2025arXiv:2411.04425

#58

DELIFT: Data Efficient Language model Instruction Fine-Tuning

Ishika Agarwal, Krishnateja Killamsetty, Lucian Popa et al.

instruction tuningdata selection optimizationpairwise utility metricsubmodular functions+3

ICML 2025arXiv:2502.01235

#59

LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently

Yuanhe Zhang, Fanghui Liu, Yudong Chen

ICLR 2025arXiv:2411.05877

#60

Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

Tong Chen, Hao Fang, Patrick Xia et al.

language model adaptationlow-rank adaptersself-supervised learningknowledge acquisition+4

ICLR 2025arXiv:2410.09344

#61

DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models

Wenlong Deng, Yize Zhao, Vala Vakilian et al.

ECCV 2024arXiv:2401.04339

#62

Memory-Efficient Fine-Tuning for Quantized Diffusion Model

Hyogon Ryu, Seohyun Lim, Hyunjung Shim

quantized diffusion modelsmemory-efficient fine-tuningmodel quantizationgenerative ai+3

ICLR 2025arXiv:2503.05431

#63

Quantum-PEFT: Ultra parameter-efficient fine-tuning

Toshiaki Koike-Akino, Francesco Tonin, Yongtao Wu et al.

CVPR 2024arXiv:2310.10700

#64

PELA: Learning Parameter-Efficient Models with Low-Rank Approximation

Yangyang Guo, Guangzhi Wang, Mohan Kankanhalli

AAAI 2025arXiv:2402.01911

#65

From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers

Bharat Runwal, Tejaswini Pedapati, Pin-Yu Chen

CVPR 2024arXiv:2311.17352

#66

Efficient Stitchable Task Adaptation

Haoyu He, Zizheng Pan, Jing Liu et al.

ICLR 2025arXiv:2504.14662

#67

Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning

Yeoreum Lee, Jinwook Jung, Sungyong Baik

model mergingparameter interferencesharpness-aware minimizationmulti-task learning+4

#68

SMT: Fine-Tuning Large Language Models with Sparse Matrices

Haoze He, Juncheng Li, Xuan Jiang et al.

parameter-efficient fine-tuningsparse matrix selectionlarge language modelsgradient-based selection+2

#69

HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA Experts

Mengqi Liao, Wei Chen, Junfeng Shen et al.

mixture of expertsparameter-efficient fine-tuninglarge language modelshierarchical routing+4

CVPR 2025arXiv:2405.18840

#70

Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation

Zelin Peng, Zhengqin Xu, Zhilin Zeng et al.

#71

Better autoregressive regression with LLMs via regression-aware fine-tuning

Michal Lukasik, Zhao Meng, Harikrishna Narasimhan et al.

autoregressive regressionlarge language modelsregression-aware fine-tuningcross-entropy loss+3

ICLR 2025arXiv:2503.18225

#72

Decoupling Angles and Strength in Low-rank Adaptation

Massimo Bini, Leander Girrbach, Zeynep Akata

parameter-efficient finetuninglow-rank adaptationsubject-driven image generationnatural language understanding+4

ICLR 2025arXiv:2501.16559

#73

LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation

Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse et al.

parameter-efficient fine-tuninglow-rank adaptationcross-model adaptationfoundation models+4

ICLR 2025arXiv:2407.19580

#74

Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures

Dang Nguyen, Wenhan Yang, Rathul Anand et al.

mini-batch coresetsmemory-efficient traininggradient matchingadam optimizer adaptation+4

CVPR 2025arXiv:2505.24816

#75

CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning

Jiangpeng He, Zhihao Duan, Fengqing Zhu

class-incremental learningparameter-efficient fine-tuninglow-rank adaptationdual-adapter architecture+4

ICLR 2025arXiv:2410.02147

#76

Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement

Gaurav Patel, Christopher M. Sandino, Behrooz Mahasseni et al.

source-free domain adaptationtime-series adaptationparameter efficiencytucker decomposition+4

ICCV 2025arXiv:2410.08989

#77

Zeroth-Order Fine-Tuning of LLMs in Random Subspaces

Ziming Yu, Pan Zhou, Sike Wang et al.

#78

Federated Residual Low-Rank Adaption of Large Language Models

Yunlu Yan, Chun-Mei Feng, Wangmeng Zuo et al.

federated learninglow-rank adaptationlarge language modelsparameter-efficient fine-tuning+4

NEURIPS 2025arXiv:2505.23387

#79

Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization

Mingzhe Du, Anh Tuan Luu, Yue Liu et al.

code efficiency optimizationreinforcement learningtest-time optimizationexecution feedback+4

ICCV 2025arXiv:2507.22872

#80

TR-PTS: Task-Relevant Parameter and Token Selection for Efficient Tuning

Siqi Luo, Haoran Yang, Yi Xin et al.

parameter-efficient fine-tuningtask-relevant parameter selectionfisher information matrixtask-relevant token selection+3

ICLR 2025arXiv:2502.13533

#81

Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models

Jun Zhang, Jue Wang, Huan Li et al.

ICML 2025arXiv:2410.09016

#82

Parameter-Efficient Fine-Tuning of State Space Models

Kevin Galim, Wonjun Kang, Yuchen Zeng et al.

ICML 2025arXiv:2410.05026

#83

Active Fine-Tuning of Multi-Task Policies

Marco Bagatella, Jonas Hübotter, Georg Martius et al.

AAAI 2025arXiv:2412.12865

#84

Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models

Yuchen Fan, Yuzhong Hong, Qiushi Wang et al.

parameter-efficient fine-tuninglow-rank adaptationvisual task adaptationdynamic training structure+4

#85

Efficient Hyperparameter Optimization with Adaptive Fidelity Identification

Jiantong Jiang, Zeyi Wen, Atif Mansoor et al.

Dropout Mixture Low-Rank Adaptation for Visual Parameters-Efficient Fine-Tuning

Zhengyi Fang, Yue Wang, Ran Yi et al.

ECCV 2024

NEURIPS 2025arXiv:2502.01755

#87

Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA

Shuangyi Chen, Yuanxin Guo, Yue Ju et al.

federated learningparameter-efficient fine-tuninglow-rank adaptationalternating optimization+4

CVPR 2025arXiv:2411.15231

#88

IterIS: Iterative Inference-Solving Alignment for LoRA Merging

Hongxu chen, Zhen Wang, Runshi Li et al.

lora mergingoptimization-based methodsiterative inference-solvingtext-to-image diffusion+3

ICLR 2025arXiv:2407.03856

#89

Zeroth-Order Fine-Tuning of LLMs with Transferable Static Sparsity

Wentao Guo, Jikai Long, Yimeng Zeng et al.

Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation

Yi-Chen Li, Fuxiang Zhang, Wenjie Qiu et al.

ICCV 2025arXiv:2403.19776

#91

Contrastive Test-Time Composition of Multiple LoRA Models for Image Generation

Tuna Meral, Enis Simsar, Federico Tombari et al.

low-rank adaptationmulti-concept image generationattention mechanismtest-time composition+3

ICLR 2025arXiv:2502.00089

#92

Ensembles of Low-Rank Expert Adapters

Yinghao Li, Vianne Gao, Chao Zhang et al.

low-rank adaptationexpert adaptersgradient direction clusteringinstruction fine-tuning+3

ICLR 2025arXiv:2412.06071

#93

KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models

Fan Wang, Juyong Jiang, Chansung Park et al.

NEURIPS 2025arXiv:2410.07170

#94

Parameter Efficient Fine-tuning via Explained Variance Adaptation

Fabian Paischer, Lukas Hauzenberger, Thomas Schmied et al.

parameter efficient fine-tuninglow-rank adaptationexplained variance adaptationactivation variance analysis+4

ECCV 2024arXiv:2403.09377

#95

Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottlenecks

Tingyu Qu, Tinne Tuytelaars, Marie-Francine Moens

vision-language tasksparameter-efficient fine-tuninglow-rank bottlenecksmultimodal alignment+3

ECCV 2024arXiv:2405.07913

#96

CTRLorALTer: Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Models

Nick Stracke, Stefan Andreas Baumann, Joshua Susskind et al.

text-to-image generationdiffusion modelszero-shot controlconditional lora+3

ICCV 2025arXiv:2508.00230

#97

Towards Higher Effective Rank in Parameter-Efficient Fine-tuning using Khatri-Rao Product

Paul Albert, Frederic Zhang, Hemanth Saratchandran et al.

parameter-efficient fine-tuninglow-rank adaptationkhatri-rao producteffective rank+4

ICLR 2025arXiv:2503.12576

#98

RaSA: Rank-Sharing Low-Rank Adaptation

Zhiwei He, Zhaopeng Tu, Xing Wang et al.

low-rank adaptationparameter-efficient fine-tuninglarge language modelscode generation+2

CVPR 2025arXiv:2504.16023

#99

PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learning

Song Wang, Xiaolu Liu, Lingdong Kong et al.

low-rank adaptationparameter-efficient fine-tuningpoint cloud learningtoken selection+3

#100

Towards Robust Parameter-Efficient Fine-Tuning for Federated Learning

Xiuwen Fang, Mang Ye

NEURIPS 2025