Text-Based Reasoning About Vector Graphics
π Homepage β’ π Paper β’ π€ Data (PVD-160k) β’ π€ Model (PVD-160k-Mistral-7b) β’ π» Code
We observe that current large multimodal models (LMMs) still struggle with seemingly straightforward reasoning tasks that require precise perception of low-level visual details, such as identifying spatial relations or solving simple mazes. In particular, this failure mode persists in question-answering tasks about vector graphicsβimages composed purely of 2D objects and shapes.
To solve this challenge, we propose Visually Descriptive Language Model (VDLM), a visual reasoning framework that operates with intermediate text-based visual descriptionsβSVG representations and learned Primal Visual Description, which can be directly integrated into existing LLMs and LMMs. We demonstrate that VDLM outperforms state-of-the-art large multimodal models, such as GPT-4V, across various multimodal reasoning tasks involving vector graphics. See our paper for more details.
- Downloads last month
- 20