Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptions

Wan Ju Kang, Eunki Kim, Na Min An, Sangryul Kim, Haemin Choi, Ki Hoon Kwak, James Thorne

📄 Paper     💻 Code

Hello, we are a team of researchers based in KAIST AI working on accessible visualization. In specific, we compiled a diagram description dataset for the blind and low-vision individuals. We worked in close cooperation with two schools for the blind, as well as over 30 sighted annotators, and we are grateful for their contribution. Check out our preprint [coming soon], and feel free to contact us at [email protected].

Abstract

Often, the needs and visual abilities differ between the annotator group and the end user group. Generating detailed diagram descriptions for blind and low-vision (BLV) users is one such challenging domain. Sighted annotators could describe visuals with ease, but existing studies have shown that direct generations by them are costly, bias-prone, and somewhat lacking by BLV standards. In this study, we ask sighted individuals to assess—rather than produce—diagram descriptions generated by vision-language models (VLM) that have been guided with latent supervision via a multi-pass inference. The sighted assessments prove effective and useful to professional educators who are themselves BLV and teach visually impaired learners. We release SIGHTATION, a collection of diagram description datasets spanning 5k diagrams and 137k samples for completion, preference, retrieval, question answering, and reasoning training purposes and demonstrate their fine-tuning potential in various downstream tasks.

Sightation Collection

• SightationCompletions
• SightationPreference
• SightationRetrieval
• SightationVQA
• SightationReasoning

The key benefit of utilizing sighted user feedback lies in their assessments that are based on solid visual grounding. The compiled assessments prove an effective training substance for steering VLMs towards more accessible descriptions. The description qualities assessed by their respective evaluator groups.

Results

Tuning VLMs on Sightation enhanced various qualities of the diagram descriptions, evaluated by BLV educators, and shown here as normalized ratings averaged in each aspect. The capability of the dataset is most strongly pronounced with Qwen2-VL-2B model, shown above.

BibTeX

@inproceedings{
}