|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- microsoft/Florence-2-large |
|
|
pipeline_tag: robotics |
|
|
tags: |
|
|
- VLA |
|
|
- LIBERO |
|
|
- Robotics |
|
|
- Flow |
|
|
--- |
|
|
# FlowerVLA - Vision-Language-Action Flow Model finetuned on LIBERO Spatial |
|
|
|
|
|
This is a pretrained FlowerVLA model for robotic manipulation trained on the LIBERO Spatial dataset. |
|
|
Flower is an efficient Vision-Language-Action Flow policy for robot learning that only contains 1B parameters. |
|
|
|
|
|
## Model Description |
|
|
|
|
|
FlowerVLA is a novel architecture that: |
|
|
- Uses half of Florence-2 for multi-modal vision-language encoding |
|
|
- Employs an novel transformer-based flow matching architecture |
|
|
- Provides an efficient, versatile VLA policy with only ~1B parameters |
|
|
|
|
|
## Model Performance |
|
|
|
|
|
This checkpoint contains weights for the LIBERO Spatial challenge and achieves these results: |
|
|
|
|
|
avg_seq_len success rate 0.9681089520454407 |
|
|
pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate with success 0.9791666666666666 |
|
|
pick_up_the_black_bowl_next_to_the_ramekin_and_place_it_on_the_plate with success 0.9807692307692308 |
|
|
pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate with success 0.9807692307692308 |
|
|
pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate with success 1.0 |
|
|
pick_up_the_black_bowl_in_the_top_drawer_of_the_wooden_cabinet_and_place_it_on_the_plate with success 1.0 |
|
|
pick_up_the_black_bowl_on_the_ramekin_and_place_it_on_the_plate with success 0.8621794871794872 |
|
|
pick_up_the_black_bowl_next_to_the_cookie_box_and_place_it_on_the_plate with success 1.0 |
|
|
pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate with success 1.0 |
|
|
pick_up_the_black_bowl_next_to_the_plate_and_place_it_on_the_plate with success 0.9166666666666666 |
|
|
pick_up_the_black_bowl_on_the_wooden_cabinet_and_place_it_on_the_plate with success 0.9615384615384616 |
|
|
|
|
|
|
|
|
### Input/Output Specifications |
|
|
|
|
|
#### Inputs |
|
|
- RGB Static Camera: `(B, T, 3, H, W)` tensor |
|
|
- RGB Gripper Camera: `(B, T, 3, H, W)` tensor |
|
|
- Language Instructions: Text strings |
|
|
|
|
|
#### Outputs |
|
|
- Action Space: `(B, T, 7)` tensor representing delta EEF actions |
|
|
|
|
|
## Usage |
|
|
|
|
|
Check out our full model implementation on Github [todo]() and follow the instructions in the readme to test the model on one of the environments. |
|
|
|
|
|
```python |
|
|
obs = { |
|
|
"rgb_obs": { |
|
|
"rgb_static": static_image, |
|
|
"rgb_gripper": gripper_image |
|
|
} |
|
|
} |
|
|
goal = {"lang_text": "pick up the blue cube"} |
|
|
action = model.step(obs, goal) |
|
|
``` |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Configuration |
|
|
- **Optimizer**: AdamW |
|
|
- **Learning Rate**: 2e-5 |
|
|
- **Weight Decay**: 0.05 |
|
|
|
|
|
|
|
|
@inproceedings{ |
|
|
reuss2025flower, |
|
|
# Add citation when available |
|
|
} |
|
|
|
|
|
|
|
|
## License |
|
|
This model is released under the MIT license. |
