ReactiveAI/sSQAT-mm
Text Generation
•
0.0B
•
Updated
Sparse Query Attention (SQA) Research by Reactive AI
Experimental models with Sparse Query Attention layers. Reducing training time/cost by ~3-10% compared to GQA & MQA, with the same level performance
-
-
ReactiveAI/SQAT-mm
Text Generation • 0.0B • Updated -
ReactiveAI/xSQAT-mm
Text Generation • 0.0B • Updated -
ReactiveAI/GQA-Ref-Micro
Text Generation • 0.0B • Updated -
ReactiveAI/MQA-Ref-Micro
Text Generation • 0.0B • Updated -
ReactiveAI/SQAT-m
Text Generation • 0.0B • Updated -
ReactiveAI/xSQAT-m
Text Generation • 0.0B • Updated -
ReactiveAI/sSQAT-m
Text Generation • 0.0B • Updated -
ReactiveAI/xSMQAT-m
Text Generation • 0.0B • Updated -
Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
Paper • 2510.01817 • Published • 12