SAM 3.1, ONNX export for EyePop

ONNX conversion of Meta's SAM 3.1 image detector (sam3.1_multiplex.pt), the graphs EyePop's open-vocabulary eyepop.sam3.detect and eyepop.sam3.segment abilities run. No retraining and no fine-tuning: the weights are Meta's, rounded to float16.

Attribution and licence

  • Original work: SAM 3 / SAM 3.1, Meta Platforms, https://github.com/facebookresearch/sam3
  • Licence: the SAM License; a verbatim copy ships in this repository as LICENSE. Your use of these files is governed by it, exactly as the original weights are.

Files

file inputs outputs
sam3_image_encoder.onnx image uint8 [3, 1008, 1008], RGB, the frame scaled to fill backbone_fpn_0..2
sam3_language_encoder.onnx tokens int64 [Q, 32], CLIP BPE (tokenizer.json), zero padded language_mask bool [Q, 32], language_features f32 [32, Q, 256]
sam3_decoder_detect.onnx backbone_fpn_2, the language features of Q prompts, score_threshold f32 [1] scores [N], boxes [N, 4] (x1, y1, x2, y2 in 0..1), prompt_index int64 [N]
sam3_decoder_segment.onnx the same, with backbone_fpn_0 and backbone_fpn_1 the same and masks uint8 [N, 288, 288] over the whole frame

The detector's position encoding depends only on the fixed input size, so each decoder holds it as a constant rather than taking it from the image encoder.

A detection's score is sigmoid(logit) * sigmoid(presence logit), as Meta's Sam3Processor computes it. Q prompts are decoded in one run; EyePop allows up to 8.

How they were made

Exported with torch.onnx.export (opset 19) by export_sam3.py in eyepop-pipeline (examples/abilities/sam3/export). The patches it applies change how the math is written, not what it computes; each is checked against Meta's code:

  • the fused bf16 addmm_act runs as activation(linear(x)) in fp32;
  • RoPE runs in real arithmetic, as x * cos + (x @ P) * sin;
  • an empty geometric prompt encodes only its CLS token, without zero-length tensors;
  • the decoder's attention runs in chunks of 648 queries, to bound its memory;
  • each image encoder block merges its attention heads with a 4D transpose instead of a 5D one (same values), which TensorRT 10.16 computes correctly in fp16;
  • the image encoder's absolute position embedding is one precomputed constant for the fixed 1008x1008 input, which onnxruntime 1.27 hands TensorRT correctly.

The encoders store their weights as float16, each followed by a Cast to float32, so they compute in float32 on every execution provider. Detections match the PyTorch reference: box IoU >= 0.998, score delta <= 0.0004 on CUDA.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for eyepop-ai/sam3.1-onnx

Base model

facebook/sam3.1
Quantized
(9)
this model