SAM 3.1, ONNX export for EyePop
ONNX conversion of Meta's SAM 3.1 image detector (sam3.1_multiplex.pt), the graphs
EyePop's open-vocabulary eyepop.sam3.detect and eyepop.sam3.segment abilities run.
No retraining and no fine-tuning: the weights are Meta's, rounded to float16.
Attribution and licence
- Original work: SAM 3 / SAM 3.1, Meta Platforms, https://github.com/facebookresearch/sam3
- Licence: the SAM License; a verbatim copy ships in this repository as
LICENSE. Your use of these files is governed by it, exactly as the original weights are.
Files
| file | inputs | outputs |
|---|---|---|
sam3_image_encoder.onnx |
image uint8 [3, 1008, 1008], RGB, the frame scaled to fill |
backbone_fpn_0..2 |
sam3_language_encoder.onnx |
tokens int64 [Q, 32], CLIP BPE (tokenizer.json), zero padded |
language_mask bool [Q, 32], language_features f32 [32, Q, 256] |
sam3_decoder_detect.onnx |
backbone_fpn_2, the language features of Q prompts, score_threshold f32 [1] |
scores [N], boxes [N, 4] (x1, y1, x2, y2 in 0..1), prompt_index int64 [N] |
sam3_decoder_segment.onnx |
the same, with backbone_fpn_0 and backbone_fpn_1 |
the same and masks uint8 [N, 288, 288] over the whole frame |
The detector's position encoding depends only on the fixed input size, so each decoder holds it as a constant rather than taking it from the image encoder.
A detection's score is sigmoid(logit) * sigmoid(presence logit), as Meta's Sam3Processor
computes it. Q prompts are decoded in one run; EyePop allows up to 8.
How they were made
Exported with torch.onnx.export (opset 19) by export_sam3.py in
eyepop-pipeline (examples/abilities/sam3/export).
The patches it applies change how the math is written, not what it computes; each is checked
against Meta's code:
- the fused bf16
addmm_actruns asactivation(linear(x))in fp32; - RoPE runs in real arithmetic, as
x * cos + (x @ P) * sin; - an empty geometric prompt encodes only its CLS token, without zero-length tensors;
- the decoder's attention runs in chunks of 648 queries, to bound its memory;
- each image encoder block merges its attention heads with a 4D transpose instead of a 5D one (same values), which TensorRT 10.16 computes correctly in fp16;
- the image encoder's absolute position embedding is one precomputed constant for the fixed 1008x1008 input, which onnxruntime 1.27 hands TensorRT correctly.
The encoders store their weights as float16, each followed by a Cast to float32, so they compute in float32 on every execution provider. Detections match the PyTorch reference: box IoU >= 0.998, score delta <= 0.0004 on CUDA.
Model tree for eyepop-ai/sam3.1-onnx
Base model
facebook/sam3.1