TheSyx-V1-7B-Base

Introduction

TheSyx-V1-7B-Base is the first LLM released by thehosy.

Features:

• Type: Causal Language Models
• Training Stage: Pretraining
• Architecture: Qwen3 MoE
• Number of Parameters: 7.52B
• Number of Layers: 28
• Number of Attention Heads (GQA): 24 for Q and 4 for KV
• Context Length: Full 16384 tokens and generation 8192 tokens

We do not recommend using base language models for conversations. Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., on this model.

Requirements

The code of TheSyx-V1-7B-Base has been in the latest Hugging face transformers and i advise you to use the latest version of transformers.

Citation

...