itazap/blt-1b-hf

Byte Latent Transformer (BLT)

Model Description

BLT (Byte Latent Transformer) is a tokenizer-free transformer architecture that operates directly on raw byte sequences. Instead of processing text token by token, BLT dynamically groups bytes into entropy-based patches, enabling more efficient and scalable processing for byte-level tasks.

Key components:

• Local Encoder → Latent Transformer → Local Decoder architecture.
• Entropy-based patcher (BltPatcher): scans byte streams and creates patches when entropy thresholds are met.
• Hash n-gram embeddings: maintain contextual information over neighboring bytes.

BLT achieves competitive performance compared to traditional token-based transformers, supporting multilingual, noisy, or mixed-script input.

Paper: Byte Latent Transformer: Patches Scale Better Than Tokens (FAIR @ Meta)

Original FAIR checkpoint: https://huggingface.co/facebook/blt-1b

---

How to Use

from transformers import BltForCausalLM, AutoTokenizer

model = BltForCausalLM.from_pretrained("itazap/blt-1b-hf", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("itazap/blt-1b-hf")

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=200, do_sample=False, use_cache=False)
output_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True)