Hugging Face's logo

locuslab
/
base-smollm2-1.7b-score0_60p_123rephrase_mild_45ref_45web_ref6x-600B-step-270000

PyTorch
llama
model
transformer
smollm2
Model card Files
xet
Community
base-smollm2-1.7b-score0_60p_123rephrase_mild_45ref_45web_ref6x-600B-step-270000 / README.md
pratyushmaini's picture
pratyushmaini
Upload model version: final
5abac3b verified
preview code
|
raw
history blame
1.18 kB
metadata
version: final
family: smollm2-1.7b
model_name: score0_60p_123rephrase_mild_45ref_45web_ref6x-600B-step-270000
license: mit
tags:
  - model
  - transformer
  - smollm2

SmolLM2 score0_60p_123rephrase_mild_45ref_45web_ref6x-600B-step-270000 (Version: final)

Model Details

  • Architecture: SmolLM2
  • Parameters: 1.7B

Training Configuration 

attention_logit_softcapping: null
attention_scores_scalar: null
attn_bias: false
bias: false
block_size: 8192
final_logit_softcapping: null
gelu_approximate: none
head_size: 64
hf_config:
  name: SmolLM2-1.7B
  org: HuggingFaceTB
intermediate_size: 8192
lm_head_bias: false
mlp_class_name: LLaMAMLP
n_embd: 2048
n_expert: 0
n_expert_per_token: 0
n_head: 32
n_layer: 24
n_query_groups: 32
name: SmolLM2-1.7B
norm_class_name: RMSNorm
norm_eps: 1.0e-05
norm_qk: false
padded_vocab_size: 49152
padding_multiple: 512
parallel_residual: false
post_attention_norm: false
post_mlp_norm: false
rope_adjustments: null
rope_base: 130000
rope_condense_ratio: 1
rotary_percentage: 1.0
scale_embeddings: false
shared_attention_norm: false
sliding_window_layer_placing: null
sliding_window_size: null
vocab_size: 49152