anlp-a3-transformer

transformer decoder-only sequence model trained for ANLP Monsoon 2026 Assignment 3 (Part A), implemented from scratch in PyTorch (no mamba-ssm / fla).

architecture	transformer
parameters	4,038,912
d_model / layers	256 / 5
max sequence length	256
analytic FLOPs / 256-token sequence	2,402,025,472
best validation token accuracy	0.7069
learning rate	0.001
cipher conditioning	real

Trained as a prefix LM on [BOS] ciphertext [SEP] plaintext [EOS] with the loss on the plaintext span only.

Note on the task. The released cipher is a per-position one-time pad: deleting the 18 punctuation null symbols aligns ciphertext to plaintext 1:1, and on the resulting aligned positions the cipher symbol is statistically independent of the plaintext character it encodes. Only the plaintext length is recoverable. These checkpoints therefore behave as character language models with an exact length cue, not as decryptors.

Use

from huggingface_hub import snapshot_download
from part1.hf_utils import load_checkpoint
model, tok, meta = load_checkpoint(snapshot_download("vipenl26/anlp-a3-transformer"), device="cuda")