Recurrent Networks
RNNs, LSTMs, and sequential models
Top Papers
RNNs are not Transformers (Yet): The Key Bottleneck on In-Context Retrieval
Kaiyue Wen, Xingyu Dang, Kaifeng Lyu
TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling
Shimin Zhang, Qu Yang, Chenxiang Ma et al.
Parallelizing non-linear sequential models over the sequence length
Yi Heng Lim, Qi Zhu, Joshua Selfridge et al.
TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
Andreas Auer, Patrick Podest, Daniel Klotz et al.
EAS-SNN: End-to-End Adaptive Sampling and Representation for Event-based Detection with Recurrent Spiking Neural Networks
Ziming Wang, Ziling Wang, Huaning Li et al.
DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
Julien Siems, Timur Carstensen, Arber Zela et al.
Exploring the Promise and Limits of Real-Time Recurrent Learning
Kazuki Irie, Anand Gopalakrishnan, Jürgen Schmidhuber
LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units
Zeyu Liu, Gourav Datta, Anni Li et al.
xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories
Maurice Kraus, Felix Divo, Devendra Singh Dhami et al.
Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang et al.
Long-Sequence Recommendation Models Need Decoupled Embeddings
Ningya Feng, Junwei Pan, Jialong Wu et al.
A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks
Thomas Schmied, Thomas Adler, Vihang Patil et al.
Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networks
Khurram Javed, Haseeb Shah, Richard Sutton et al.
Plastic Learning with Deep Fourier Features
Alex Lewandowski, Dale Schuurmans, Marlos C. Machado
Random-Set Neural Networks
Shireen Kudukkil Manchingal, Muhammad Mubashar, Kaizheng Wang et al.
IVP-VAE: Modeling EHR Time Series with Initial Value Problem Solvers
Jingge Xiao, Leonie Basso, Wolfgang Nejdl et al.
Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks
Lukas Braun, Erin Grant, Andrew Saxe
Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels
Maximilian Beck, Korbinian Pöppel, Phillip Lippe et al.
LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking
Jialin Li, Qiang Nie, Weifu Fu et al.
The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks.
Aaron Spieler, Nasim Rahaman, Georg Martius et al.
In-context Time Series Predictor
Jiecheng Lu, Yan Sun, Shihao Yang
BRAID: Input-driven Nonlinear Dynamical Modeling of Neural-Behavioral Data
Parsa Vahidi, Omid G. Sani, Maryam Shanechi
Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models
Benjamin Walker, Lingyi Yang, Nicola Muca Cirone et al.
Expand and Compress: Exploring Tuning Principles for Continual Spatio-Temporal Graph Forecasting
Wei Chen, Yuxuan Liang
Building Variable-Sized Models via Learngene Pool
Boyu Shi, Shiyu Xia, Xu Yang et al.
NetFormer: An interpretable model for recovering dynamical connectivity in neuronal population dynamics
Ziyu Lu, Wuwei Zhang, Trung Le et al.
Spike-Temporal Latent Representation for Energy-Efficient Event-to-Video Reconstruction
Jianxiong Tang, Jian-Huang Lai, Lingxiao Yang et al.
Real-Time Recurrent Reinforcement Learning
Julian Lemmel, Radu Grosu
Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness
Kangrui Du, Yuhang Wu, Shikuang Deng et al.
Transformative or Conservative? Conservation laws for ResNets and Transformers
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling
Mónika Farsang, Radu Grosu
Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Chengting Yu, Xiaochen Zhao, Lei Liu et al.
Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training
Xi Chen, Chang Gao, Zuowen Wang et al.
Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control
Zijie Xu, Tong Bu, Zecheng Hao et al.
Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks
Devon Jarvis, Richard Klein, Benjamin Rosman et al.
Learning long range dependencies through time reversal symmetry breaking
Guillaume Pourcel, Maxence Ernoult
Parallel Sequence Modeling via Generalized Spatial Propagation Network
Hongjun Wang, Wonmin Byeon, Jiarui Xu et al.
Learning In-context $n$-grams with Transformers: Sub-$n$-grams Are Near-Stationary Points
Aditya Vardhan Varre, Gizem Yüce, Nicolas Flammarion
STRAP: Spatio-Temporal Pattern Retrieval for Out-of-Distribution Generalization
Haoyu Zhang, WentaoZhang, Hao Miao et al.
PAC-Bayes Generalisation Bounds for Dynamical Systems including Stable RNNs
Deividas Eringis, John Leth, Zheng-Hua Tan et al.
BARNN: A Bayesian Autoregressive and Recurrent Neural Network
Dario Coscia, Max Welling, Nicola Demo et al.
Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences
Xiwen Chen, Peijie Qiu, Wenhui Zhu et al.
GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL
Lang Qin, Ziming Wang, Runhao Jiang et al.
Learning Spatiotemporal Dynamical Systems from Point Process Observations
Valerii Iakovlev, Harri Lähdesmäki
Dendritic Resonate-and-Fire Neuron for Effective and Efficient Long Sequence Modeling
Dehao Zhang, Malu Zhang, Shuai Wang et al.
Transformers for Mixed-type Event Sequences
Felix Draxler, Yang Meng, Kai Nelson et al.
LOCORE: Image Re-ranking with Long-Context Sequence Modeling
Zilin Xiao, Pavel Suma, Ayush Sachdeva et al.
TS-SNN: Temporal Shift Module for Spiking Neural Networks
Kairong Yu, Tianqing Zhang, Qi Xu et al.
Multiplication-Free Parallelizable Spiking Neurons with Efficient Spatio-Temporal Dynamics
Peng Xue, Wei Fang, Zhengyu Ma et al.
Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)
Yoonsoo Nam, Seok Hyeong Lee, Clémentine Dominé et al.
Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking Neural Networks
Guobin Shen, Dongcheng Zhao, Yiting Dong et al.
Accelerated training through iterative gradient propagation along the residual path
Erwan Fagnou, Paul Caillon, Blaise Delattre et al.
Revisiting Bi-Linear State Transitions in Recurrent Neural Networks
Reza Ebrahimi, Roland Memisevic
HADAMRNN: BINARY AND SPARSE TERNARY ORTHOGONAL RNNS
Armand Foucault, Francois Malgouyres, Franck Mamalet
Martingale Posterior Neural Networks for Fast Sequential Decision Making
Gerardo Duran-Martin, Leandro Sánchez-Betancourt, Alvaro Cartea et al.
Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
Min Chen, Guansong Pang, Wenjun Wang et al.
StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMs
Qijun Luo, Mengqi Li, Lei Zhao et al.
Improving Bilinear RNN with Closed-loop Control
Jiaxi Hu, Yongqi Pan, Jusen Du et al.
Locally Connected Echo State Networks for Time Series Forecasting
Filip Matzner, František Mráz
RHYTHM: Reasoning with Hierarchical Temporal Tokenization for Human Mobility
Haoyu He, Haozheng Luo, Yan Chen et al.
Learning Successor Features with Distributed Hebbian Temporal Memory
Evgenii Dzhivelikian, Petr Kuderov, Aleksandr Panov
Learning Chaos In A Linear Way
Xiaoyuan Cheng, Yi He, Yiming Yang et al.
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Riccardo Grazzi, Julien Siems, Arber Zela et al.
Repetition Improves Language Model Embeddings
Jacob Springer, Suhas Kotha, Daniel Fried et al.
Self-Normalized Resets for Plasticity in Continual Learning
Vivek Farias, Adam Jozefiak
TGB-Seq Benchmark: Challenging Temporal GNNs with Complex Sequential Dynamics
Lu Yi, Jie Peng, Yanping Zheng et al.
Deep Non-Rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling
Hui Deng, Jiawei Shi, Zhen Qin et al.
Iterative Sparse Attention for Long-sequence Recommendation
Guanyu Lin, Jinwei Luo, Yinfeng Li et al.
LS-TGNN: Long and Short-Term Temporal Graph Neural Network for Session-Based Recommendation
Zhonghong Ou, Xiao Zhang, Yifan Zhu et al.
Scalable Trajectory-User Linking with Dual-Stream Representation Networks
Hao Zhang, Wei Chen, Xingyu Zhao et al.
FSTA-SNN:Frequency-Based Spatial-Temporal Attention Module for Spiking Neural Networks
Kairong Yu, Tianqing Zhang, Hongwei Wang et al.
TNPAR: Topological Neural Poisson Auto-Regressive Model for Learning Granger Causal Structure from Event Sequences
Authors: Yuequn Liu, Ruichu Cai, Wei Chen et al.
Temporal Graph Contrastive Learning for Sequential Recommendation
Shengzhe Zhang, Liyi Chen, Chao Wang et al.
Complexity of Neural Network Training and ETR: Extensions with Effectively Continuous Functions
Teemu Hankala, Miika Hannula, Juha Kontinen et al.
On the Expressivity of Recurrent Neural Cascades
Nadezda Knorozova, Alessandro Ronca
6385 Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning
Jiangrong Shen, Wenyao Ni, Qi Xu et al.
Instance-Conditional Timescales of Decay for Nonstationary Learning
Nishant Jain, Pradeep Shenoy
Discovering Sequential Patterns with Predictable Inter-event Delays
Joscha Cüppers, Paul Krieger, Jilles Vreeken
Memory-Efficient Reversible Spiking Neural Networks
Hong Zhang, Yu Zhang
Dr2Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning
Chen Zhao, Shuming Liu, Karttikeya Mangalam et al.
Incremental Residual Concept Bottleneck Models
Chenming Shang, Shiji Zhou, Hengyuan Zhang et al.
Online Stabilization of Spiking Neural Networks
Yaoyu Zhu, Jianhao Ding, Tiejun Huang et al.
Critical Learning Periods Emerge Even in Deep Linear Networks
Michael Kleinman, Alessandro Achille, Stefano Soatto
SineNet: Learning Temporal Dynamics in Time-Dependent Partial Differential Equations
Xuan Zhang, Jacob Helwig, Yuchao Lin et al.
Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control
Neehal Tumma, Mathias Lechner, Noel Loo et al.
Parsing neural dynamics with infinite recurrent switching linear dynamical systems
Victor Geadah, International Brain Laboratory, Jonathan Pillow
On the Markov Property of Neural Algorithmic Reasoning: Analyses and Methods
Montgomery Bohde, Meng Liu, Alexandra Saxton et al.
Learning From Simplicial Data Based on Random Walks and 1D Convolutions
Florian Frantzen, Michael Schaub
Inverse Approximation Theory for Nonlinear Recurrent Neural Networks
Shida Wang, Zhong Li, Qianxiao Li
Successor Heads: Recurring, Interpretable Attention Heads In The Wild
Rhys Gould, Euan Ong, George Ogden et al.
Implicit regularization of deep residual networks towards neural ODEs
Pierre Marion, Yu-Han Wu, Michael Sander et al.
A Progressive Training Framework for Spiking Neural Networks with Learnable Multi-hierarchical Model
Zecheng Hao, Xinyu Shi, Zihan Huang et al.
Function-space Parameterization of Neural Networks for Sequential Learning
Aidan Scannell, Riccardo Mereu, Paul Chang et al.
Scalable Monotonic Neural Networks
Hyunho Kim, Jong-Seok Lee
A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks
Tommaso Salvatori, Yuhang Song, Yordan Yordanov et al.
Complex priors and flexible inference in recurrent circuits with dendritic nonlinearities
Benjamin Lyo, Cristina Savin
Sufficient conditions for offline reactivation in recurrent neural networks
Nanda H Krishna, Colin Bredenberg, Daniel Levenstein et al.
Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN
Biswadeep Chakraborty, Beomseok Kang, Harshit Kumar et al.
ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis
DongHao Luo, Xue Wang
Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings
Ilyass Hammouamri, Ismail Khalfaoui Hassani, Timothée Masquelier