--- base_model: mistral-community/pixtral-12b library_name: peft license: cc-by-4.0 datasets: - daniel3303/GroundCap language: - en metrics: - bleu - meteor - cider - spice - f1 - recall - precision - gmeteor - rouge model-index: - name: PixtralGroundCap results: - task: type: image-captioning subtype: grounded-image-captioning dataset: name: daniel3303/GroundCap type: grounded-image-captioning split: test metrics: - name: Precision type: grounding-precision value: 0.58 - name: Recall type: grounding-recall value: 0.96 - name: F1 type: grounding-f1 value: 0.69 - name: BLEU-4 type: bleu-4 value: 0.19 - name: METEOR type: meteor value: 0.23 - name: CIDEr type: cider value: 0.51 - name: SPICE type: spice value: 0.30 - name: gMETEOR type: gmeteor value: 0.35 --- # Model Card for PixtralGroundCap This model is a fine-tuned version of Pixtral-12B on the GroundCap dataset for grounded image captioning. It generates detailed image descriptions with explicit grounding tags that link textual descriptions to specific visual elements in the image. The model was trained on the GroundCap dataset and uses a novel tag system to ground objects (``), actions (``), and locations (``) to specific regions in images. ## Model Details ### Model Description - **Developed by:** Daniel A. P. Oliveira, Lourenço Teodoro, and David Martins de Matos (INESC-ID Lisboa and Instituto Superior Técnico, Universidade de Lisboa) - **Model type:** Fine-tuned Pixtral-12B model for grounded image captioning - **Language(s):** English - **License:** Creative Commons Attribution 4.0 - **Finetuned from model:** mistral-community/pixtral-12b ### Model Sources - **Paper:** https://arxiv.org/abs/2502.13898 - **Dataset:** https://huggingface.co/datasets/daniel3303/GroundCap ## Uses ### Direct Use The model is designed for generating grounded image captions that explicitly link textual descriptions to visual elements using three types of grounding tags: - `` for objects - `` for actions - `` for locations Each tag maintains object identity through unique IDs, enabling consistent reference tracking throughout the caption. ### Downstream Use The model can be integrated into: - Accessibility applications requiring detailed image descriptions - Content management systems needing verifiable image captions - Visual question answering systems - Image retrieval systems ### Out-of-Scope Use The model is not designed for: - General image classification - Object detection (requires separate object detection pipeline) - Video captioning - Non-English language captioning ## How to Get Started with the Model ### Input Format The model expects input in the following format: ``` You are an AI assistant that can see and understand images. I will provide you with an image and the detected objects in it along with their positions and dimensions in the format [id, x,y,width,height]. [DETECTIONS] [sky-0: 0.41,0.00,0.20,0.15] [sky-1: 0.62,0.00,0.26,0.10] [wall-0: 0.01,0.02,0.35,0.86] [person-0: 0.38,0.35,0.12,0.40] [person-1: 0.45,0.35,0.08,0.39] [wall-1: 0.39,0.10,0.35,0.48] [person-2: 0.71,0.29,0.20,0.51] [wall-2: 0.75,0.03,0.24,0.88] [person-3: 0.00,0.57,0.22,0.42] [handbag-0: 0.21,0.75,0.11,0.23] [person-4: 0.26,0.48,0.20,0.52] [floor-wood-0: 0.40,0.59,0.60,0.41] [/DETECTIONS] [IMG] ``` ### Example Output The model will generate a grounded caption using three types of tags: - `````` for objects - `````` for actions - `````` for locations Example output: ``` In this scene, a group of individuals is gathered in what appears to be a rugged, makeshift shelter. The walls are constructed from rough materials, giving the space a temporary and utilitarian feel. The sky is visible in the background, suggesting that the shelter is partially open or lacks a complete roof. Several people are present in the scene, each engaged in their own activities. One individual sits on the ground, while another person is seated nearby. Another person is also sitting on the ground, and a fourth individual is seated as well. An additional person is sitting close by. The handbag is placed on the ground near one of the individuals, suggesting they might have brought some personal belongings with them. The overall atmosphere of the scene is one of simplicity and resilience, with the individuals making the best of their surroundings in this temporary shelter. ``` ## Bias, Risks, and Limitations - The model was trained on movie scenes from MovieNet, which may introduce biases in terms of scene composition, lighting, and camera angles - Performance may vary for real-world images that differ significantly from movie scenes - The model relies on pre-detected objects and their bounding boxes, Mask2Former was used for object detection in the original paper ### Recommendations - Use in conjunction with a robust object detection system - Verify grounding accuracy for critical applications - Consider the movie-centric nature of the training data when applying to other domains ## Training Details ### Training Data The model was trained on the GroundCap dataset, which contains: - 52,016 images from 77 movies - 344 human-annotated captions - 52,016 automatically generated captions - ### Training Procedure The training followed a two-stage approach: #### Stage 1: - Training on 52,016 automatically generated captions - Learning rate: 2×10^-4 - Epochs: 2 - Batch size: 64 (with gradient accumulation) #### Stage 2: - Fine-tuning on 344 human-refined captions - Learning rate: 2×10^-6 - Epochs: 2 - Batch size: 32 (with gradient accumulation) #### Training Hyperparameters - **LoRA Configuration:** - Rank: 16 - Alpha: 32 - Targeted layers: Self-attention (query, key, value, output) and MLP (gate, up, down) - **Optimizer:** AdamW - **Weight decay:** 0.01 - **Precision:** bfloat16 - **Hardware:** 2x NVIDIA A100 (80GB) - **Training time:** 1 day ## Evaluation ### Testing Data, Factors & Metrics The model was evaluated on: - 10,000 test images from GroundCap from which 70 are human-annotated test cases ### Metrics - **Grounding metrics:** - Precision (P): Correctly grounded objects / Total objects mentioned in caption - Recall (R): Correctly grounded objects / Total detected objects - F1 score: Harmonic mean of precision and recall - **Caption quality metrics:** - BLEU-4: N-gram overlap with reference captions - METEOR: Semantic similarity with reference captions - CIDEr: Consensus-based image description evaluation - SPICE: Semantic propositional image caption evaluation - ROUGE-L: Longest common subsequence based evaluation - **Combined metric:** - gMETEOR: Harmonic mean of METEOR and grounding F1 score, combining language quality with grounding accuracy - **Human evaluation:** (5-point Likert scale) - Object precision: Accuracy of object grounding and tag classification - Grounding recall: Coverage of detected objects in captions - Description accuracy: Correctness of described actions and relationships - Language quality: Grammar, readability, and coherence - Overall quality: Assessment of caption effectiveness - **ChatGPT-4o evaluation:** (5-point Likert scale) - Uses same criteria as human evaluation - Correlations with human judgments: - Object Precision: 0.81 (Pearson), 0.73 (Spearman) - Grounding Recall: 0.76 (Pearson), 0.67 (Spearman) - Description Accuracy: 0.79 (Pearson), 0.77 (Spearman) - Language Quality: 0.59 (Pearson), 0.44 (Spearman) - Overall Quality: 0.78 (Pearson), 0.68 (Spearman) ### Results Automatic metrics on test set for PixtralGroundCap: - Precision: 0.58 - Recall: 0.96 - F1 Score: 0.69 - BLEU-4: 0.19 - METEOR: 0.23 - CIDEr: 0.51 - SPICE: 0.30 - ROUGE-L: 0.37 - gMETEOR: 0.35 Human evaluation results (scale 1-5): - Object Precision: 4.22 - Grounding Recall: 4.19 - Description Accuracy: 4.08 - Language Quality: 4.91 - Overall Quality: 4.22 ChatGPT-4o evaluation results (scale 1-5): - Object Precision: 4.21 - Grounding Recall: 4.13 - Description Accuracy: 4.01 - Language Quality: 4.90 - Overall Quality: 4.19 ## Environmental Impact - **Hardware Type:** 2x NVIDIA A100 GPUs - **Hours used:** 24 hours - **Cloud Provider:** INESC-ID - **Compute Region:** Lisbon, Portugal ## Paper [ArXiv link](https://arxiv.org/abs/2502.13898). ## Citation **BibTeX:** ```bash @article{Oliveira2025GroundCapAV, title={GroundCap: A Visually Grounded Image Captioning Dataset}, author={Daniel A. P. Oliveira and Louren{ç}o Teodoro and David Martins de Matos}, year={2025}, url={https://api.semanticscholar.org/CorpusID:276450057} } ``` ## Model Card Authors Daniel A. P. Oliveira, Lourenço Teodoro, and David Martins de Matos ## Model Card Contact daniel.oliveira@inesc-id.pt ### Framework versions - PEFT 0.13.2