Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision

The rapid evolution of Multi-modality Large Language Models (MLLMs) has catalyzed a shift in computer vision from specialized models to general-purpose foundation models. Nevertheless, there is still an inadequacy in assessing the abilities of MLLMs onlow-level visual perception and understanding. To address this gap, we presentQ-Bench, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs on three realms: low-level visual perception, low-level visual description, and overall visual quality assessment.a)To evaluate the low-levelperceptionability, we construct theLLVisionQAdataset, consisting of 2,990 diverse-sourced images, each equipped with a human-asked question focusing on its low-level attributes. We then measure the correctness of MLLMs on answering these questions.b)To examine thedescriptionability of MLLMs on low-level information, we propose theLLDescribedataset consisting of long expert-labelledgoldenlow-level text descriptions on 499 images, and a GPT-involved comparison pipeline between outputs of MLLMs and thegoldendescriptions.c)Besides these two tasks, we further measure their visual qualityassessmentability to align with human opinion scores. Specifically, we design a softmax-based strategy that enables MLLMs to predictquantifiablequality scores, and evaluate them on various existing image quality assessment (IQA) datasets. Our evaluation across the three abilities confirms that MLLMs possess preliminary low-level visual skills. However, these skills are still unstable and relatively imprecise, indicating the need for specific enhancements on MLLMs towards these abilities. We hope that our benchmark can encourage the research community to delve deeper to discover and enhance these untapped potentials of MLLMs.