Transformer Architecture

ICLR 2025arXiv:2412.06464

#2

Gated Delta Networks: Improving Mamba2 with Delta Rule

Songlin Yang, Jan Kautz, Ali Hatamizadeh

linear transformersgating mechanismsdelta update rulelong-context tasks+3

141

AAAI 2024arXiv:2312.10325

#3

An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention

Yehjin Shin, Jeongwhan Choi, Hyowon Wi et al.

sequential recommendationself-attention mechanismoversmoothing problemtransformer-based models+4

99

ICLR 2024arXiv:2402.04347

#4

The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry

Michael Zhang, Kush Bhatia, Hermann Kumbong et al.

84

ICLR 2025arXiv:2409.10594

#5

Kolmogorov-Arnold Transformer

Xingyi Yang, Xinchao Wang

83

NeurIPS 2025arXiv:2409.10516

#6

RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval

Di Liu, Meng Chen, Baotong Lu et al.

attention mechanismlong-context inferencevector retrievalapproximate nearest neighbor search+3

81

CVPR 2024arXiv:2403.16131

#7

Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement

Xiuquan Hou, Meiqin Liu, Senlin Zhang et al.

78

CVPR 2024arXiv:2312.10113

#8

Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation

guo, Tianwei Lin

63

CVPR 2024arXiv:2305.17328

#9

Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers

Hongjie Wang, Bhishma Dedhia, Niraj Jha

59

ICLR 2025arXiv:2503.03321

#10

HiT-SR: Hierarchical Transformer for Efficient Image Super-Resolution

Xiang Zhang, Yulun Zhang, Fisher Yu

See What You Are Told: Visual Attention Sink in Large Multimodal Models

Seil Kang, Jinyeong Kim, Junhyeok Kim et al.

attention mechanismvisual attention sinkmultimodal modelsvision-language tasks+4

50

ICLR 2024arXiv:2311.01906

#12

An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels

Duy-Kien Nguyen, Mahmoud Assran, Unnat Jain et al.

Simplifying Transformer Blocks

Bobby He, Thomas Hofmann

47

ICLR 2024arXiv:2310.00535

#14

JoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention

Yuandong Tian, Yiping Wang, Zhenyu Zhang et al.

46

AAAI 2024arXiv:2210.12381

#15

S2WAT: Image Style Transfer via Hierarchical Vision Transformer Using Strips Window Attention

Chiyu Zhang, Xiaogang Xu, Lei Wang et al.

style transfervision transformerattention mechanismwindow attention+4

46

CVPR 2024arXiv:2404.02117

#16

Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners

Keon Hee Park, Kyungwoo Song, Gyeong-Moon Park

45

CVPR 2025arXiv:2412.18597

#17

DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation

Minghong Cai, Xiaodong Cun, Xiaoyu Li et al.

44

CVPR 2024arXiv:2404.01882

#18

Scene Adaptive Sparse Transformer for Event-based Object Detection

Yansong Peng, Li Hebei, Yueyi Zhang et al.

40

ICML 2025arXiv:2410.11842

#19

DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention

Lianghui Zhu, Zilong Huang, Bencheng Liao et al.

MoH: Multi-Head Attention as Mixture-of-Head Attention

Peng Jin, Bo Zhu, Li Yuan et al.

37

CVPR 2024arXiv:2402.05472

#21

Question Aware Vision Transformer for Multimodal Reasoning

Roy Ganz, Yair Kittenplon, Aviad Aberdam et al.

36

ICLR 2025arXiv:2501.15061

#22

PolaFormer: Polarity-aware Linear Attention for Vision Transformers

Weikang Meng, Yadan Luo, Xin Li et al.

linear attentionvision transformersattention mechanismquery-key interactions+2

35

ICML 2025arXiv:2502.01951

#23

On the Emergence of Position Bias in Transformers

Xinyi Wu, Yifei Wang, Stefanie Jegelka et al.

33

ICLR 2025arXiv:2409.04431

#24

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

Jason Ramapuram, Federico Danieli, Eeshan Gunesh Dhekane et al.

sigmoid attentiontransformer architectureuniversal function approximationattention mechanism alternatives+2

33

AAAI 2024arXiv:2402.04563

#25

Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention

Saebom Leem, Hyunseok Seo

vision transformerattention mechanismvisual explanationsweakly-supervised localization+4

31

ICLR 2024arXiv:2310.10375

#26

GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers

Takeru Miyato, Bernhard Jaeger, Max Welling et al.

31

ICLR 2024arXiv:2310.03817

#27

Logical Languages Accepted by Transformer Encoders with Hard Attention

Pablo Barcelo, Alexander Kozachinskiy, Anthony W. Lin et al.

29

ICLR 2024arXiv:2302.00456

#28

Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi et al.

28

CVPR 2024arXiv:2406.05478

#29

Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis

Zanlin Ni, Yulin Wang, Renping Zhou et al.

28

AAAI 2024arXiv:2305.04440

#30

Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Counting

Zhicheng Wang, Liwen Xiao, Zhiguo Cao et al.

class-agnostic countingvision transformerfew-shot learningself-attention mechanism+3

28

ICLR 2024arXiv:2309.16354

#31

Transformer-VQ: Linear-Time Transformers via Vector Quantization

Lucas D. Lingle

26

AAAI 2025arXiv:2409.09649

#32

Understanding Factual Recall in Transformers via Associative Memories

Eshaan Nichani, Jason Lee, Alberto Bietti

SparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks

Meng Lou, Yunxiang Fu, Yizhou Yu

24

ICLR 2025arXiv:2410.02984

#34

Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient

George Wang, Jesse Hoogland, Stan van Wingerden et al.

23

ICLR 2025arXiv:2408.12578

#35

A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language

Ekdeep Singh Lubana, Kyogo Kawaguchi, Robert Dick et al.

23

CVPR 2024arXiv:2303.09383

#36

Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers

Zhibo Yang, Sounak Mondal, Seoyoung Ahn et al.

22

NeurIPS 2025arXiv:2412.16112

#37

CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up

Songhua Liu, Zhenxiong Tan, Xinchao Wang

20

NeurIPS 2025arXiv:2501.18795

#38

The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer

Weixian Lei, Jiacong Wang, Haochen Wang et al.

Rope to Nope and Back Again: A New Hybrid Attention Strategy

Bowen Yang, Bharat Venkitesh, Dwaraknath Gnaneshwar Talupuru et al.

20

CVPR 2024arXiv:2403.19067

#40

Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach

Wei Dong, Xing Zhang, Bihui Chen et al.

19

ICLR 2024arXiv:2403.11686

#41

Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding

Tatsunori Taniai, Ryo Igarashi, Yuta Suzuki et al.

17

CVPR 2024arXiv:2312.01897

#42

Adapting Short-Term Transformers for Action Detection in Untrimmed Videos

Min Yang, gaohuan, Ping Guo et al.

17

ICLR 2024arXiv:2402.04882

#43

LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units

Zeyu Liu, Gourav Datta, Anni Li et al.

17

CVPR 2024arXiv:2403.14552

#44

Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer

Junyi Wu, Bin Duan, Weitai Kang et al.

16

CVPR 2025arXiv:2409.19764

#45

Spiking Transformer with Spatial-Temporal Attention

Donghyun Lee, Yuhang Li, Youngeun Kim et al.

16

ICML 2025arXiv:2502.17019

#46

Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems

Maksim Zhdanov, Max Welling, Jan-Willem van de Meent

15

ICLR 2024arXiv:2310.08049

#47

Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability

Ivan Lee, Nan Jiang, Taylor Berg-Kirkpatrick

15

ICLR 2024arXiv:2310.01749

#48

Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns

Brian DuSell, David Chiang

15

ICLR 2025arXiv:2405.13861

#49

Equivariant Neural Functional Networks for Transformers

Viet-Hoang Tran, Thieu Vo, An Nguyen et al.

Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning

Jiuqi Wang, Ethan Blaser, Hadi Daneshmand et al.

in-context reinforcement learningtemporal difference learningpolicy evaluationtransformer architecture+3

14

ICML 2025arXiv:2501.16825

#51

Can Transformers Learn Full Bayesian Inference in Context?

Arik Reuter, Tim G. J. Rudner, Vincent Fortuin et al.

14

ICLR 2025arXiv:2502.12677

#52

Multi-Turn Jailbreaking Large Language Models via Attention Shifting

Xiaohu Du, Fan Mo, Ming Wen et al.

Spiking Vision Transformer with Saccadic Attention

Shuai Wang, Malu Zhang, Dehao Zhang et al.

spiking neural networksvision transformersattention mechanismsaccadic attention+3

14

ECCV 2024arXiv:2401.00912

#54

ScatterFormer: Efficient Voxel Transformer with Scattered Linear Attention

Chenhang He, Ruihuang Li, Guowen Zhang et al.

voxel transformerlinear attentionpoint cloud understanding3d object detection+3

13

AAAI 2024arXiv:2402.12946

#55

Mr. DETR: Instructive Multi-Route Training for Detection Transformers

Chang-Bin Zhang, Yujie Zhong, Kai Han

Cell Graph Transformer for Nuclei Classification

Wei Lou, Guanbin Li, Xiang Wan et al.

cell graph transformernuclei classificationhistopathology imagesgraph neural networks+4

12

ICLR 2025arXiv:2503.04416

#57

Learning Transformer-based World Models with Contrastive Predictive Coding

Maxime Burchi, Radu Timofte

transformer-based world modelscontrastive predictive codingmodel-based reinforcement learningnext state prediction+2

12

ICLR 2025arXiv:2310.10845

#58

CoTFormer: A Chain of Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference

Amirkeivan Mohtashami, Matteo Pagliardini, Martin Jaggi

chain of thought reasoningattention mechanismtransformer architecturecompute adaptive inference+3

12

CVPR 2024arXiv:2401.14405

#59

PaTH Attention: Position Encoding via Accumulating Householder Transformations

Songlin Yang, Yikang Shen, Kaiyue Wen et al.

Transformer-Squared: Self-adaptive LLMs

Qi Sun, Edoardo Cetin, Yujin Tang

Towards Understanding and Improving Adversarial Robustness of Vision Transformers

Samyak Jain, Tanima Dutta

Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities

Yiyuan Zhang, Xiaohan Ding, Kaixiong Gong et al.

11

CVPR 2024arXiv:2405.05502

#63

Dynamic Layer Tying for Parameter-Efficient Transformers

Tamir David-Hay, Lior Wolf

Towards Accurate and Robust Architectures via Neural Architecture Search

Yuwei Ou, Yuqi Feng, Yanan Sun

10

AAAI 2025arXiv:2501.15774

#65

Efficient Attention-Sharing Information Distillation Transformer for Lightweight Single Image Super-Resolution

Karam Park, Jae Woong Soh, Nam Ik Cho

10

ICCV 2025arXiv:2503.22796

#66

DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers

Hanling Zhang, Rundong Su, Zhihang Yuan et al.

10

CVPR 2025arXiv:2410.00871

#67

Rethinking Query-based Transformer for Continual Image Segmentation

Yuchen Zhu, Cheng Shi, Dingyou Wang et al.

MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretraining

Yunze Liu, Li Yi

ICLR 2025arXiv:2410.10986

#69

What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis

Weronika Ormaniec, Felix Dangel, Sidak Pal Singh

attention mechanismhessian analysisoptimization landscapetransformer architecture+4

ICLR 2024arXiv:2311.13541

#70

Linear Log-Normal Attention with Unbiased Concentration

Yury Nahshan, Joseph Kampeas, Emir Haleva

ICML 2025arXiv:2408.15138

#71

How Transformers Learn Structured Data: Insights From Hierarchical Filtering

Jerome Garnier-Brun, Marc Mezard, Emanuele Moscato et al.

ICML 2025arXiv:2505.22697

#72

Update Your Transformer to the Latest Release: Re-Basin of Task Vectors

Filippo Rinaldi, Giacomo Capitani, Lorenzo Bonicelli et al.

ICML 2025arXiv:2505.24717

#73

PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations

Benjamin Holzschuh, Qiang Liu, Georg Kohl et al.

ICML 2025arXiv:2502.12170

#74

MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections

Da Xiao, Qingye Meng, Shengping Li et al.

CVPR 2024arXiv:2403.15835

#75

Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression

Hancheng Ye, Chong Yu, Peng Ye et al.

ICLR 2025arXiv:2406.09827

#76

A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention

Heejun Lee, Geon Park, Youngwan Lee et al.

attention mechanismlong-context generationtransformer servingattention locality+3

8

NeurIPS 2025arXiv:2512.17351

#77

Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

Zeyuan Allen-Zhu

canon layerssynthetic pretraining tasksarchitecture designhorizontal information flow+4

8

ICLR 2025arXiv:2403.06925

#78

Transformers Learn Low Sensitivity Functions: Investigations and Implications

Bhavya Vasudeva, Deqing Fu, Tianyi Zhou et al.

inductive biassensitivity analysisrobustness improvementloss landscape geometry+4

AAAI 2025arXiv:2407.20021

#79

PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer

Pierre-David Letourneau, Manish Singh, Hsin-Pai Cheng et al.

MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity

Kanghyun Choi, Hyeyoon Lee, Dain Kwon et al.

AAAI 2024arXiv:2401.03540

#81

Context Enhanced Transformer for Single Image Object Detection in Video Data

Seungjun An, Seonghoon Park, Gyeongnyeon Kim et al.

SeTformer Is What You Need for Vision and Language

Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger et al.

self-optimal transportattention mechanismcomputational efficiencyvision and language tasks+4

CVPR 2025arXiv:2503.02689

#83

Dependency Structure-Enhanced Graph Attention Networks for Event Detection

Qizhi Wan, Changxuan wan, Keli Xiao et al.

STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networks

Tianqing Zhang, Kairong Yu, Xian Zhong et al.

ICLR 2025arXiv:2403.13838

#85

Rethinking Spiking Self-Attention Mechanism: Implementing α-XNOR Similarity Calculation in Spiking Transformers

Yichen Xiao, Shuai Wang, Dehao Zhang et al.

Circuit Transformer: A Transformer That Preserves Logical Equivalence

Xihan Li, Xing Li, Lei Chen et al.

6

ICLR 2025arXiv:2410.17174

#87

From Attention to Activation: Unraveling the Enigmas of Large Language Models

Prannay Kaul, Chengcheng Ma, Ismail Elezi et al.

6

NeurIPS 2025arXiv:2505.17863

#88

Multi-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent

Tong Yang, Yu Huang, Yingbin Liang et al.

The emergence of sparse attention: impact of data distribution and benefits of repetition

Nicolas Zucchet, Francesco D'Angelo, Andrew Lampinen et al.

AAAI 2024arXiv:2312.12742

#90

A Solvable Attention for Neural Scaling Laws

Bochen Lyu, Di Wang, Zhanxing Zhu

Cached Transformers: Improving Transformers with Differentiable Memory Cached

Zhaoyang Zhang, Wenqi Shao, Yixiao Ge et al.

attention mechanismdifferentiable memory cachelanguage modelingmachine translation+4

ICML 2025arXiv:2502.01954

#92

CAR-Transformer: Cross-Attention Reinforcement Transformer for Cross-Lingual Summarization

Yuang Cai, Yuyu Yuan

Constrained Belief Updates Explain Geometric Structures in Transformer Representations

Mateusz Piotrowski, Paul Riechers, Daniel Filan et al.

ICML 2025arXiv:2410.20210

#94

TTFSFormer: A TTFS-based Lossless Conversion of Spiking Transformer

Lusen Zhao, Zihan Huang, Ding Jianhao et al.

Looking Beyond the Top-1: Transformers Determine Top Tokens in Order

Daria Lioubashevski, Tomer Schlank, Gabriel Stanovsky et al.

CVPR 2024arXiv:2311.17902

#96

Language-conditioned Detection Transformer

Jang Hyun Cho, Philipp Krähenbühl

ICML 2025arXiv:2411.19574

#97

KV Shifting Attention Enhances Language Modeling

Mingyu Xu, Bingning Wang, Weipeng Chen

ICML 2025arXiv:2502.05164

#98

In-Context Denoising with One-Layer Transformers: Connections between Attention and Associative Memory Retrieval

Matthew Smart, Alberto Bietti, Anirvan Sengupta

CVPR 2025arXiv:2503.18359

#99

Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities

Yifang Chen, Xiaoyu Li, Yingyu Liang et al.

Context-Enhanced Memory-Refined Transformer for Online Action Detection

Zhanzhong Pang, Fadime Sener, Angela Yao

online action detectionstreaming video analysisshort-term memory encodinglong-term memory encoding+3