Multimodal Learning

CVPR 2025arXiv:2405.21075

#2

Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

Chaoyou Fu, Yuhan Dai, Yongdong Luo et al.

858

CVPR 2024arXiv:2311.04257

#3

mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

Qinghao Ye, Haiyang Xu, Jiabo Ye et al.

601

CVPR 2024arXiv:2312.13286

#4

Generative Multimodal Models are In-Context Learners

Quan Sun, Yufeng Cui, Xiaosong Zhang et al.

422

CVPR 2024arXiv:2312.14135

#5

Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces

Jihan Yang, Shusheng Yang, Anjali W. Gupta et al.

V?: Guided Visual Search as a Core Mechanism in Multimodal LLMs

Penghao Wu, Saining Xie

327

ICLR 2024arXiv:2309.11499

#7

DreamLLM: Synergistic Multimodal Comprehension and Creation

Runpei Dong, chunrui han, Yuang Peng et al.

275

CVPR 2024arXiv:2305.08275

#8

R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization

yi yang, Xiaoxuan He, Hongkun Pan et al.

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier et al.

ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding

Le Xue, Ning Yu, Shu Zhang et al.

192

CVPR 2024arXiv:2312.00784

#11

ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

Mu Cai, Haotian Liu, Siva Mustikovela et al.

153

CVPR 2024arXiv:2312.06739

#12

SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

Yuzhou Huang, Liangbin Xie, Xintao Wang et al.

139

CVPR 2024arXiv:2312.06968

#13

Hallucination Augmented Contrastive Learning for Multimodal Large Language Model

Chaoya Jiang, Haiyang Xu, Mengfan Dong et al.

116

CVPR 2024arXiv:2401.18084

#14

Binding Touch to Everything: Learning Unified Multimodal Tactile Representations

Fengyu Yang, Chao Feng, Ziyang Chen et al.

109

NeurIPS 2025arXiv:2506.15564

#15

Show-o2: Improved Native Unified Multimodal Models

Jinheng Xie, Zhenheng Yang, Mike Zheng Shou

90

CVPR 2024arXiv:2311.17049

#16

MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training

Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri et al.

84

ICLR 2024arXiv:2308.12038

#17

Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

Jinyi Hu, Yuan Yao, Chongyi Wang et al.

77

AAAI 2025arXiv:2408.15556

#18

Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization

Renjie Pi, Tianyang Han, Wei Xiong et al.

Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models

Wenbin Wang, Liang Ding, Minyan Zeng et al.

73

CVPR 2025arXiv:2410.08202

#20

Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

Luo, Xue Yang, Wenhan Dou et al.

68

ICLR 2025arXiv:2405.17430

#21

Matryoshka Multimodal Models

Mu Cai, Jianwei Yang, Jianfeng Gao et al.

multimodal modelsvisual token granularitytoken efficiency optimizationhigh-resolution image processing+3

57

AAAI 2024arXiv:2309.08912

#22

OmniSat: Self-Supervised Modality Fusion for Earth Observation

Guillaume Astruc, Nicolas Gonthier, Clement Mallet et al.

Delving into Multimodal Prompting for Fine-Grained Visual Classification

Xin Jiang, Hao Tang, Junyao Gao et al.

fine-grained visual classificationvision-language modelsmultimodal promptingcontrastive language-image pre-training+4

55

AAAI 2024arXiv:2403.06197

#24

DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency

Wenfang Yao, Kejing Yin, William Cheung et al.

multi-modal fusionmissing modalitydisentangled representationelectronic health records+4

55

NeurIPS 2025arXiv:2409.15272

#25

OmniBench: Towards The Future of Universal Omni-Language Models

Yizhi Li, Ge Zhang, Yinghao Ma et al.

51

CVPR 2024arXiv:2309.06255

#26

Enhancing Multimodal Cooperation via Sample-level Modality Valuation

Yake Wei, Ruoxuan Feng, Zihe Wang et al.

51

AAAI 2025arXiv:2402.19407

#27

MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation

Jinfeng Xu, Zheyu Chen, Shuo Yang et al.

50

CVPR 2024arXiv:2311.17435

#28

MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning

Chaoyi Zhang, Kevin Lin, Zhengyuan Yang et al.

49

CVPR 2024arXiv:2312.14233

#29

VCoder: Versatile Vision Encoders for Multimodal Large Language Models

Jitesh Jain, Jianwei Yang, Humphrey Shi

48

CVPR 2024arXiv:2312.04521

#30

Multimodal Industrial Anomaly Detection by Crossmodal Feature Mapping

Alex Costanzino, Pierluigi Zama Ramirez, Giuseppe Lisanti et al.

45

CVPR 2024arXiv:2311.17597

#31

BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models

Rizhao Cai, Zirui Song, DAYAN GUAN et al.

Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning

Yiwen Ye, Yutong Xie, Jianpeng Zhang et al.

42

AAAI 2024arXiv:2312.08762

#33

The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

Zhaofeng Wu, Xinyan Yu, Dani Yogatama et al.

Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models

Liqi He, Zuchao Li, Xiantao Cai et al.

chain-of-thought reasoningmulti-modal reasoninglatent space learningdiffusion processes+4

34

AAAI 2024arXiv:2312.14667

#35

Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition

Qianrui Zhou, Hua Xu, Hao Li et al.

multimodal intent recognitioncontrastive learningmodality-aware promptingcross-modality attention+4

33

ICCV 2025arXiv:2503.21979

#36

Harmonizing Visual Representations for Unified Multimodal Understanding and Generation

Size Wu, Wenwei Zhang, Lumin Xu et al.

33

ICLR 2024arXiv:2309.15564

#37

Jointly Training Large Autoregressive Multimodal Models

Emanuele Aiello, Lili Yu, Yixin Nie et al.

32

ICLR 2025arXiv:2409.07402

#38

What to align in multimodal contrastive learning?

Benoit Dufumier, Javiera Castillo Navarro, Devis Tuia et al.

30

ECCV 2024arXiv:2407.11351

#39

Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities

Xu Zheng, Yuanhuiyi Lyu, LIN WANG

modality-agnostic representationsemantic segmentationknowledge distillationmulti-modal vision-language models+3

28

AAAI 2025arXiv:2404.14368

#40

Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning

Hanxun Yu, Wentong Li, Song Wang et al.

DC-NAS: Divide-and-Conquer Neural Architecture Search for Multi-Modal Classification

Xinyan Liang, Pinhan Fu, Qian Guo et al.

Graphic Design with Large Multimodal Model

Yutao Cheng, Zhao Zhang, Maoke Yang et al.

AAAI 2025arXiv:2403.06199

#43

A Comprehensive Overhaul of Multimodal Assistant with Small Language Models

Minjie Zhu, Yichen Zhu, Ning Liu et al.

CVPR 2025arXiv:2501.09757

#44

Distilling Multi-modal Large Language Models for Autonomous Driving

Deepti Hegde, Rajeev Yasarla, Hong Cai et al.

autonomous drivingmotion planningmulti-modal llmsvision-based planner+4

ICLR 2025arXiv:2412.11959

#45

Gramian Multimodal Representation Learning and Alignment

Giordano Cicchetti, Eleonora Grassucci, Luigi Sigillo et al.

multimodal representation learningcontrastive learningmodality alignmentgramian volume minimization+3

ICML 2025arXiv:2412.01441

#46

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations

Anian Ruoss, Fabio Pardo, Harris Chan et al.

CVPR 2024arXiv:2306.08832

#47

Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding

Le Zhang, Rabiul Awal, Aishwarya Agrawal

NeurIPS 2025arXiv:2410.12787

#48

Multimodal Patient Representation Learning with Missing Modalities and Labels

Zhenbang Wu, Anant Dadu, Nicholas Tustison et al.

The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

Sicong Leng, Yun Xing, Zesen Cheng et al.

26

CVPR 2025arXiv:2503.01210

#50

Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond

Guanyao Wu, Haoyu Liu, Hongming Fu et al.

25

CVPR 2024arXiv:2311.05698

#51

Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities

AJ Piergiovanni, Isaac Noble, Dahun Kim et al.

25

CVPR 2025arXiv:2503.20188

#52

Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector

Xiao Guo, Xiufeng Song, Yue Zhang et al.

deepfake detectionvision-language modelsmulti-modal learninginterpretable ai+4

24

ICCV 2025arXiv:2410.16236

#53

From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons

Andrew Szot, Bogdan Mazoure, Omar Attia et al.

LLaVA-KD: A Framework of Distilling Multimodal Large Language Models

Yuxuan Cai, Jiangning Zhang, Haoyang He et al.

knowledge distillationmultimodal large language modelsvision-language understandingmodel compression+4

22

ICCV 2025arXiv:2412.05934

#55

Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models

Ma Teng, Xiaojun Jia, Ranjie Duan et al.

jailbreak attacksmultimodal large language modelsadversarial capabilitiesmultimodal risk distribution+3

21

ICLR 2024arXiv:2404.09403

#56

Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning

Xiongye Xiao, Gengshuo Liu, Gaurav Gupta et al.

20

CVPR 2024arXiv:2403.06946

#57

Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation

Xinyao Li, Yuke Li, Zhekai Du et al.

19

CVPR 2024arXiv:2402.18490

#58

TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding

Zhihao Zhang, Shengcao Cao, Yu-Xiong Wang

18

CVPR 2024arXiv:2312.16279

#59

Cloud-Device Collaborative Learning for Multimodal Large Language Models

Guanqun Wang, Jiaming Liu, Chenxuan Li et al.

18

AAAI 2024arXiv:2402.10002

#60

MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding

HaiTao Yu, Mofei Song

point cloud understandingself-supervised representation learningmulti-modal learningmulti-view 2d information+4

18

CVPR 2024arXiv:2404.11207

#61

Exploring the Transferability of Visual Prompting for Multimodal Large Language Models

Yichi Zhang, Yinpeng Dong, Siyuan Zhang et al.

17

ICLR 2025arXiv:2410.01506

#62

Self-Supervised Multi-Modal Knowledge Graph Contrastive Hashing for Cross-Modal Search

Meiyu Liang, Junping Du, Zhengyang Liang et al.

Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion

Dexuan Ding, Lei Wang, Liyun Zhu et al.

16

ICLR 2025arXiv:2403.09193

#64

Can We Talk Models Into Seeing the World Differently?

Paul Gavrikov, Jovita Lukasik, Steffen Jung et al.

vision language modelstexture vs shape biasmultimodal fusionvisual cue processing+4

AAAI 2024arXiv:2401.03468

#65

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

Qiushi Zhu, Jie Zhang, Yu Gu et al.

self-supervised learningmultichannel speech processingaudio-visual speech recognitioncontrastive learning+4

AAAI 2025arXiv:2408.12763

#66

Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models

Jean Park, Kuk Jin Jang, Basam Alasaly et al.

ICLR 2025arXiv:2402.05889

#67

CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion

Shoubin Yu, Jaehong Yoon, Mohit Bansal

video-language reasoningmultimodal fusionmodular modality integrationparameter-efficient modules+4

CVPR 2025arXiv:2503.06063

#68

Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices

Junyan Lin, Haoran Chen, Yue Fan et al.

CVPR 2025arXiv:2503.16036

#69

Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models

Zhihang Liu, Chen-Wei Xie, Pandeng Li et al.

video token compressionmulti-modal large language modelsinstruction injectionconditional token compression+3

CVPR 2025arXiv:2412.04616

#70

Assessing and Learning Alignment of Unimodal Vision and Language Models

Le Zhang, Qian Yang, Aishwarya Agrawal

vision-language alignmentself-supervised learningzero-shot retrievalmultimodal large language models+4

ICLR 2025arXiv:2502.04263

#71

Docopilot: Improving Multimodal Models for Document-Level Understanding

Yuchen Duan, Zhe Chen, Yusong Hu et al.

Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models

Xin Zhang, Yanzhao Zhang, Wen Xie et al.

Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci et al.

vision-language modelsmodality inversionintra-modal misalignmentcontrastive learning+3

CVPR 2024arXiv:2403.00691

#74

Tri-Modal Motion Retrieval by Learning a Joint Embedding Space

Kangning Yin, Shihao Zou, Yuxuan Ge et al.

ICCV 2025arXiv:2504.08358

#75

LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs

Jiarui Wang, Huiyu Duan, Yu Zhao et al.

ECCV 2024arXiv:2311.12047

#76

Robust Multimodal Learning via Representation Decoupling

Shicai Wei, Yang Luo, Yuji Wang et al.

MultiDelete for Multimodal Machine Unlearning

Jiali Cheng, Hadi Amiri

machine unlearningmultimodal datamodality decouplingmultimodal representation+3

vision-language modelsmultimodal continual learningimage-text matchingzero-shot prediction+3

#78

C-CLIP: Multimodal Continual Learning for Vision-Language Model

Wenzhuo Liu, Fei Zhu, Longhui Wei et al.

ICLR 2025

AAAI 2025arXiv:2412.11248

#79

Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing

Pengcheng Zhao, Jinxing Zhou, Yang Zhao et al.

ICML 2025arXiv:2502.09356

#80

Galileo: Learning Global & Local Features of Many Remote Sensing Modalities

Gabriel Tseng, Anthony Fuller, Marlena Reil et al.

12

CVPR 2025arXiv:2406.10462

#81

CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation

Wei Chen, Lin Li, Yongqi Yang et al.

multimodal large language modelsinterleaved image-text generationmultimodal in-context learningnarrative coherence+4

12

CVPR 2024arXiv:2401.14405

#82

Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities

Yiyuan Zhang, Xiaohan Ding, Kaixiong Gong et al.

11

ICLR 2025arXiv:2405.16869

#83

Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning

Yichi Zhang, Zhuo Chen, Lingbing Guo et al.

10

CVPR 2025arXiv:2408.00754

#84

Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering

Federico Cocchi, Nicholas Moratelli, Marcella Cornia et al.

Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs

Yikang Zhou, Tao Zhang, Shilin Xu et al.

Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model

Benlin Liu, Yuhao Dong, Yiqin Wang et al.

multimodal language modelsspatial-temporal reasoningvisual promptingobject correspondences+4

CVPR 2025arXiv:2412.09511

#87

GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistency

Dongyue Lu, Lingdong Kong, Tianxin Huang et al.

ECCV 2024arXiv:2407.13851

#88

X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs

Swetha Sirnam, Jinyu Yang, Tal Neiman et al.

ICML 2025arXiv:2501.02669

#89

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?

Simon Park, Abhishek Panigrahi, Yun Cheng et al.

ICLR 2025arXiv:2410.10167

#90

X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing

Xinyan Chen, Jianfei Yang

ICLR 2025arXiv:2505.20027

#91

Multi-modal brain encoding models for multi-modal stimuli

SUBBA REDDY OOTA, Khushbu Pahwa, mounika marreddy et al.

AAAI 2025arXiv:2412.10840

#92

Attention-Driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models Without Fine-Tuning

Hai-Ming Xu, Qi Chen, Lei Wang et al.

AAAI 2025arXiv:2404.13322

#93

MergeNet: Knowledge Migration Across Heterogeneous Models, Tasks, and Modalities

Kunxi Li, Tianyu Zhan, Kairui Fu et al.

AAAI 2025arXiv:2503.12560

#94

OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities

Lichang Chen, Hexiang Hu, Mingda Zhang et al.

Multi-Granular Multimodal Clue Fusion for Meme Understanding

Li Zheng, Hao Fei, Ting Dai et al.

AAAI 2024arXiv:2305.08372

#96

Hierarchical Aligned Multimodal Learning for NER on Tweet Posts

Peipei Liu, Hong Li, Yimo Ren et al.

multimodal named entity recognitioncross-modal learningattention mechanismimage-text alignment+4

AAAI 2024arXiv:2401.11649

#97

A Multimodal, Multi-Task Adapting Framework for Video Action Recognition

Mengmeng Wang, Jiazheng Xing, Boyuan Jiang et al.

video action recognitionvision-language pretrained modelsparameter-efficient fine-tuningmultimodal adapters+3

ICML 2025arXiv:2505.22483

#98

A Closer Look at Multimodal Representation Collapse

Abhra Chaudhuri, Anjan Dutta, Tu Bui et al.

CVPR 2025arXiv:2504.08851

#99

RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models

Haoran Hao, Jiaming Han, Changsheng Li et al.

Mimic In-Context Learning for Multimodal Tasks

Yuchu Jiang, Jiale Fu, chenduo hao et al.