jo-mengr/mmcontext-cg_3_5k-nla-biomodern-None-text_only_50-feat_cs

SentenceTransformer based on thomas-sounack/BioClinical-ModernBERT-base

This is a sentence-transformers model finetuned from thomas-sounack/BioClinical-ModernBERT-base on the cellxgene_pseudo_bulk_3_5k_multiplets_natural_language_annotation_cs50 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: thomas-sounack/BioClinical-ModernBERT-base
• Maximum Sequence Length: None tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • cellxgene_pseudo_bulk_3_5k_multiplets_natural_language_annotation_cs50
• Language: code

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): MMContextEncoder(
    (text_encoder): ModernBertModel(
      (embeddings): ModernBertEmbeddings(
        (tok_embeddings): Embedding(50368, 768, padding_idx=50283)
        (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (drop): Dropout(p=0.0, inplace=False)
      )
      (layers): ModuleList(
        (0): ModernBertEncoderLayer(
          (attn_norm): Identity()
          (attn): ModernBertAttention(
            (Wqkv): Linear(in_features=768, out_features=2304, bias=False)
            (rotary_emb): ModernBertRotaryEmbedding()
            (Wo): Linear(in_features=768, out_features=768, bias=False)
            (out_drop): Identity()
          )
          (mlp_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (mlp): ModernBertMLP(
            (Wi): Linear(in_features=768, out_features=2304, bias=False)
            (act): GELUActivation()
            (drop): Dropout(p=0.0, inplace=False)
            (Wo): Linear(in_features=1152, out_features=768, bias=False)
          )
        )
        (1-21): 21 x ModernBertEncoderLayer(
          (attn_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (attn): ModernBertAttention(
            (Wqkv): Linear(in_features=768, out_features=2304, bias=False)
            (rotary_emb): ModernBertRotaryEmbedding()
            (Wo): Linear(in_features=768, out_features=768, bias=False)
            (out_drop): Identity()
          )
          (mlp_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (mlp): ModernBertMLP(
            (Wi): Linear(in_features=768, out_features=2304, bias=False)
            (act): GELUActivation()
            (drop): Dropout(p=0.0, inplace=False)
            (Wo): Linear(in_features=1152, out_features=768, bias=False)
          )
        )
      )
      (final_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    )
    (pooling): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("jo-mengr/mmcontext-cg_3_5k-nla-biomodern-None-text_only_50-feat_cs")
# Run inference
sentences = [
    'S100A6 S100A11 TPT1 TMSB10 ACTB MT2A FTH1 JUNB MT-CO3 SAT1 PTMA RPL36A HSPA5 HSPB1 PTGES3 RTN4 SPCS2 PRMT1 RPS29 FTL LGALS1 NFKBIA PDCD5 DNPH1 RPS4Y1 FKBP11 MYC SDCBP HSPD1 S100A10 PHF20 AP2S1 NDUFB4 KLF6 RTRAF PAPOLA NUDC CMTM6 NAPA MRPL3 HSPE1 GORASP2 VPS4A MAP7 INPP1 ZFP36L2 ATP5PF PDIA3 SLC3A2 MACROD2',
    "This measurement was conducted with 10x 3' v3. Basal cell from the transition zone of prostate epithelium, derived from a 73-year-old male.",
    "This measurement was conducted with 10x 3' v2. Basal cell of prostate epithelium from the peripheral zone of a 31-year-old European male.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9957, 0.9955],
#         [0.9957, 1.0000, 1.0000],
#         [0.9955, 1.0000, 1.0000]])

Training Details

Training Dataset

cellxgene_pseudo_bulk_3_5k_multiplets_natural_language_annotation_cs50

• Dataset: cellxgene_pseudo_bulk_3_5k_multiplets_natural_language_annotation_cs50 at e6bf2fd
• Size: 2,331 training samples
• Columns: anchor, positive, negative_1, and negative_2
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative_1	negative_2
  type	string	string	string	string
  details	min: 260 characters mean: 296.2 characters max: 343 characters	min: 90 characters mean: 211.32 characters max: 613 characters	min: 90 characters mean: 212.11 characters max: 811 characters	min: 260 characters mean: 295.95 characters max: 335 characters

• Samples:

  anchor	positive	negative_1	negative_2
  TAGLN MT2A ADIRF ACTA2 MYL9 MT-ND1 IGFBP7 FOS VIM S100A6 MT1E JUN HES4 CALD1 TPM1 FTL SPARCL1 IFITM3 RERGL IGFBP5 DUSP1 RGS5 FRZB PPP1R14A S100A4 CD9 TIMP3 CST3 CCN2 JCHAIN GLUL CXCL14 GSN NEAT1 FKBP5 RGCC CRYAB ZBTB16 SPARC CLU APOE NR4A2 A2M NDUFA4L2 TFPI LAMP2 TAC1 MGST1 IL32 ANLN	This measurement was conducted with 10x 3' v2. Colon pericytes from a male individual in his eighth decade, characterized as Pericytes RERGL NTRK2.	This measurement was conducted with 10x 3' v3. Colon fibroblast cells derived from the lamina propria of mucosa of a male individual in his fourth decade.	CXCL14 FOS TAGLN VIM S100A6 MT-ND1 APOE ACTA2 TPM1 JUN CALD1 IGFBP7 FTL HHIP COL1A2 MYL9 NEAT1 DCN FN1 SPARCL1 A2M S100A4 SPARC IGFBP5 CST3 IFITM3 GSN VCAN COL4A2 ADIRF CD9 FBLN1 PMP22 DUSP1 PPP1R14A HSPA6 MT-ND4L SELENOP TIMP1 LAMA4 RGS10 SYNPO2 TIMP3 CPED1 CCN2 GNG11 APCDD1 NFIA IFI16 CLIC4
  PLP1 ST18 MBP MAN2A1 RNF220 MOBP LINC00609 PDE1A PIP4K2A CDH20 TMEM144 NCKAP5 TF GAB1 FRMD4B PLEKHH1 PTGDS C10orf90 CDK18 ENPP2 DOCK5 LMCD1-AS1 LINC01505 BCAS1 FAM107B SYNJ2 ATP10B PDE1C UGT8 FBXL7 GRM3 NEAT1 ANLN VRK2 ARAP2 CLDND1 SCD POLR2F CHN2 TLE4 PCSK6 ABCA8 SHROOM4 CERCAM RAD51B KCNH8 LPAR1 DBNDD2 LINC01170 LINC00639	This measurement was conducted with 10x 3' v3. Oligodendrocyte cells from the hippocampal formation of a 29-year-old male, specifically from the Head of hippocampus (HiH) - Uncal DG-CA4 region, with a region of interest at Human DGU-CA4Upy.	This measurement was conducted with 10x 3' v3. Neuron cell type from a 42-year-old European male, specifically from the hippocampal formation within the Head of hippocampus (HiH) - Uncal DG-CA4 region, which belongs to the Hippocampal dentate gyrus.	PLXDC2 APBB1IP APOE P2RY12 CD74 ARHGAP15 SLC1A3 SYNDIG1 LPAR6 ADAM28 ST6GAL1 FYB1 CSGALNACT1 C1QC DISC1 DLEU1 MAML2 FTL SGK3 LY86 C3 SAT1 DOCK5 CD14 C1QA NCKAP5 LAMP2 IGF1 ANKRD44 LPCAT2 NRP1 CST3 ATP8B4 DOCK8 PALD1 LIPA CD69 ITPR2 PREX1 ATG4C FOXP2 ARHGAP22 NHSL1 SORL1 ARHGAP24 ZFHX3 TMEM163 JAM2 RUNX1 TGFBR2
  DPP10 LINC01088 NFIA SOX6 PARD3B RMST CASC15 AGBL1 HPSE2 MAML2 KANK1 GLIS3 ADGRV1 WLS ITGA2 LMCD1-AS1 LINC02055 TCF7L2 GRAMD2B ROR1 SLC1A3 EYA4 CLU FOXP2 GREB1L ANKFN1 MYO1E COLEC12 ZFHX4 PLCH1 CD36 PLCE1 ARHGAP18 CHD7 RAD51B RNF220 NEAT1 PREX2 PCSK5 CDH20 SLC1A2 ACSS3 RFX4 ITPR2 ENOSF1 IQGAP2 CREB5 GJA1 UTRN RGS12	This measurement was conducted with 10x 3' v3. Ependymal cell from the thalamic complex (thalamus (THM) - medial nuclear complex of thalamus (MNC) - mediodorsal nucleus of thalamus + reuniens nucleus (medioventral nucleus) of thalamus - MD + Re) of a 50-year-old male with European ethnicity.	This measurement was conducted with 10x 3' v3. Neuron cell type from a 50-year-old human thalamic complex, specifically from the thalamus (THM) - medial nuclear complex of thalamus (MNC) - mediodorsal nucleus of thalamus + reuniens nucleus (medioventral nucleus) of thalamus - MD + Re region, with European self-reported ethnicity and male sex, classified under the Thalamic excitatory supercluster.	ADARB2 NXPH1 FBXL7 GRIK1 TOX SPARCL1 GAD2 VWC2 NKAIN3 PRELID2 MT-ND1 EYA4 TACR1 KCNH5 PREX2 DPP10 GRIN3A PCDH15 CCK KIT MAF ZMAT4 ARAP2 ANKRD44 RERG UBASH3B NOS1 CLU SV2C EGFR ARL4C DLX6-AS1 GHR SLC35F4 ADAMTS9-AS2 PCSK5 PTPRB PDGFD ROR1 ARHGAP6 SPHKAP NFIA GOLIM4 AP1S2 SLC24A3 ANK1 SYNDIG1 ZBTB16 COL4A2 FBN2

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

cellxgene_pseudo_bulk_3_5k_multiplets_natural_language_annotation_cs50

• Dataset: cellxgene_pseudo_bulk_3_5k_multiplets_natural_language_annotation_cs50 at e6bf2fd
• Size: 59 evaluation samples
• Columns: anchor, positive, negative_1, and negative_2
• Approximate statistics based on the first 59 samples:

  	anchor	positive	negative_1	negative_2
  type	string	string	string	string
  details	min: 286 characters mean: 303.12 characters max: 319 characters	min: 101 characters mean: 210.83 characters max: 573 characters	min: 101 characters mean: 207.69 characters max: 573 characters	min: 286 characters mean: 302.88 characters max: 319 characters

• Samples:

  anchor	positive	negative_1	negative_2
  RPS29 ACTB PTMA TPT1 BTG1 DUSP1 TMSB10 KLF2 JUN FTH1 TMSB4X FTL ZFP36L2 HLA-DRB5 CD79A CXCR4 TSC22D3 ZNF331 MS4A1 STK17A NR4A2 RPL36A VIM HERPUD1 MAP3K8 CD69 CD83 SBDS CD52 JUNB CRIP1 STK17B VPREB3 OSER1 FOS BASP1 FNBP1 H4C3 MBP PHF20 FAM107B KLF6 SLC44A1 PAPOLA MAST3 STK4 POLR1F C12orf57 TFEB HSPE1	This measurement was conducted with 10x 5' v1. Plasmablast cells derived from the tonsil tissue of a 9-year-old female patient with recurrent tonsillitis. The cells exhibit IGH + IGK chains, with IGH being functional, in-frame, and having IGHJ602 and IGHV2-7001 genes, as well as IGKC, IGKV3-11, IGKJ2 genes with productive status, and IgG2 isotype.	This measurement was conducted with 10x 5' v1. Memory B cell sample taken from a 5-year-old female human tonsil tissue, with FCRL2/3high MBC phenotype, and medical history of obstructive sleep apnea and recurrent tonsillitis.	ACTB TMSB4X RPS29 TPT1 FTH1 FTL PTMA CD52 MS4A1 TMSB10 CD79A BTG1 HLA-DRB5 SNHG7 C12orf57 VIM H4C3 SEC62 ANKRD44 ATP2B1 MBD4 MARCKSL1 HCLS1 NDUFB4 ETFB PTGES3 CDKN1B ELF1 SERPINB6 THOC2 SAT1 CD164 NHP2 ZFP36L2 DNAJB14 PSIP1 TAF10 S100A6 RPL36A CD22 RTF2 AP2S1 ZNF800 FAM107B SMARCA2 PSEN1 CD82 LAT2 SNX5 TASP1
  TMSB4X RPS29 TMSB10 PTMA FTL ACTB SAT1 FTH1 MT-CO3 TSC22D3 CD52 ISG15 BTG1 OAS1 ELF1 CXCR4 TPT1 JUNB CARD16 RPL36A HERPUD1 ENO1 PAPOLA CD79A PTGES3 DUSP1 HSPD1 ATP5MJ ARPC5 STK17A ZEB2 CDC42SE1 S100A10 S100A6 MAD1L1 ICA1 CFLAR FKBP4 RBM6 LAMP2 IFRD1 JARID2 MED24 SEC62 RNF216 CD22 CLK1 ZMYND11 PHLDB1 RUNX3	This measurement was conducted with 10x 3' v2. B cell sample taken from blood tissue of a 49-year old European female with managed systemic lupus erythematosus (SLE). The cell type was enriched from peripheral blood mononuclear cells.	This measurement was conducted with 10x 3' v2. Sample is a conventional dendritic cell derived from a 29-year old European male with managed systemic lupus erythematosus (SLE). The cell was obtained from blood tissue and enriched as part of peripheral blood mononuclear cells.	TMSB4X TMSB10 ACTB FTH1 FTL RPS29 S100A4 MT-CO3 PTMA S100A6 LGALS1 TPT1 FOS VIM MT2A SAT1 S100A10 TYMP GSTP1 IFITM3 S100A11 JUNB RPL36A ATP5MJ CD52 ISG15 HLA-DRB5 AP2S1 HERPUD1 ENO1 NFKBIA PLEK RGS2 DUSP1 HCST BTG1 C1orf162 ATP5PF ARPC5 CARD16 MAD1L1 ICA1 CFLAR FKBP4 RBM6 LAMP2 IFRD1 JARID2 MED24 SEC62
  ACTB TMSB4X TMSB10 CD79A PTMA RPS29 TPT1 FTH1 MS4A1 HLA-DRB5 CD52 MARCKSL1 CD22 VPREB3 PDIA3 RPL36A MAD1L1 ARAP2 SRRT CD69 XRN1 CXCR4 SOX4 XPO4 BTG1 LARS1 TIAL1 CNOT8 ATP5MJ BASP1 NT5DC1 MRPL14 FAM133B ADSL SEC62 UBA6 ZNF800 HERPUD1 NDUFB4 FRYL STK17B GSTP1 LAT2 FTL RTRAF MRPS33 NUDC CMTM6 NFKBIA GID8	This measurement was conducted with 10x 5' v1. Naive B cell sample taken from a 5-year-old female with obstructive sleep apnea and recurrent tonsillitis, originating from the tonsil tissue.	This measurement was conducted with 10x 5' v1. Naive B cell sample taken from a 5-year-old female individual with obstructive sleep apnea and recurrent tonsillitis, isolated from tonsil tissue.	TMSB4X CD79A PTMA ACTB CD83 HLA-DRB5 CD52 RPS29 TPT1 TMSB10 FTL HSPA9 MS4A1 SMG1 FTH1 CD22 CLPTM1 SMC3 C12orf57 BTG1 NHP2 DNAJB14 JUN HCLS1 TYMP ZNF800 IPO5 NDUFB4 ISOC1 FRYL GSTP1 USP48 MKNK2 LMF2 TXN2 ZC3H7B NFKBIA RIOK3 STAG2 DNAJC3 UBE4A RSRC2 SUDS3 MCM3 C6orf62 LMNB1 NPHP3 FAM162A RNF7 NFE2L2

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 0.05
• num_train_epochs: 1
• warmup_ratio: 0.1

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Framework Versions

• Python: 3.12.9
• Sentence Transformers: 5.0.0
• Transformers: 4.52.3
• PyTorch: 2.7.0
• Accelerate: 1.7.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}