Vinit3241/clinical_trials_all-MiniLM-L6-v2

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'open label trial safety efficacy sym001 treatment immune thrombocytopenic purpura itp. If your serious vaccine-induced adverse event has been entered in the CDC Vaccine Adverse Event Reporting System (VAERS) we are interested in enrolling you for this study in order to log your symptoms.\n\nThe primary goal of this study is to create a national database and gather vaccine-associated serious adverse events/injury data from newly vaccinated individuals in the US in order to identify the possible underlying causal relationships and plausible underlying biological mechanisms. The project aims to identify the genetic determinants of vaccine-induced adverse response by studying host genetics. We plan to use whole genome sequencing to identify single nucleotide polymorphisms associated with cardiovascular, neurological, gastrointestinal, musculoskeletal and immunological symptoms induced by vaccine administration.\n\nThe secondary goal is to establish criteria that enable classification of vaccine-induced adverse events/injuries compare data from our database with the official Vaccine Injury Table National Vaccine Injury Compensation Program on or after March 21, 2017.\n\nThe tertiary goal is to establish a database to gather detailed long-term adverse reaction data from subjects enrolled in FDA Emergency Use Authorized vaccine clinical trials.',
    'Immune Thrombocytopenic Purpura inclusion criterion confirm presence thrombocytopenia platelet count   30000mm3 predose visit history isolated itp rhdpositive serology previous treatment response line therapy itpexclusion criterion know clinical picture suggestive cause thrombocytopenia especially systematic lupus erythematosusantiphospholipid syndrome evans syndrome immunodeficiency state lymphoproliferative disorder liver diseaseingestion drug quinidinequinine heparin sulfonamide hereditary thrombocytopenia confirm relevant laboratory finding suspect infection hiv hepatitis c h pylori clinical splenomegaly history abnormal bone marrow examination ongoing haemorrhage correspond grade 3 4 bleeding scale underlie haemolytic condition history splenectomy subject pregnant breast feeding intend pregnant incidence severity adverse event aes include adverse event saes 6 week post dose measurement platelet count day 1 week 6',
    'Multiple System Atrophy inclusion criteriadiagnostic1 participant diagnosis possible probable msa modify gilman et al 2008 diagnostic criteria2 participant onset msa symptom occur 4 year screen assess investigator3 evidence msa specific symptom deficit measure umsars scaleexclusion criteriamedical history1 participant contraindication study proceduresdiagnostic assessments1 presence confound diagnosis andor condition affect participant safety study investigator judgement2 participant participation previous study diseasemodifye therapy prove receipt active treatment compromise interpretability datum present study consultation medical monitor designeeother1 participant participate study investigate active passive immunization αsynuclein αsyn progressive disease pd msa immunoglobulin g therapy 6 month screen change baseline modify unified multiple system atrophy rating scale umsar week 52 umsar historical review 11item scale adapt unify parkinson disease rating scale updrs assess activity relate motor disability relate autonomic dysfunction item score 0 normal 3 severe total score sum score domain range 0 33 high score mean poor health 52 week change baseline 11item umsar week 52 11 item umsar include 11 item ii assess motor autonomic disability umsar historical review assess activity relate motor disability autonomic dysfunction umsar ii motor examination measure functional impairment specific parkinsonian cerebellar feature item score 0 normal 4 severe total score sum score domain range 0 44 high score mean poor health 52 weekschange baseline umsar total score umsar   ii week 52 umsar total scale consist item umsars part ii umsar historical review 12item scale assess activity relate motor disability autonomic dysfunction item score 0 normal 4 severe umsar ii motor examination 14item scale measure functional impairment eg speech rapid alternate movement hand finger tap leg agility select complex movement specific parkinsonian tremor rest cerebellar ocular motor dysfunction heelshin test feature item score 0 normal 4 severe 52 weekschange baseline umsars week 52 umsar historical review modified 11item scale adapt updrs assess activity relate motor disability 8 item 4 novel item relate autonomic dysfunction item score 0 normal 4 severe total score sum score item range 0 44 high score mean poor health 52 weekschange baseline umsars ii week 52 umsar ii motor examination 14item scale item eg speech rapid alternate movement hand finger tap leg agility measure functional impairment select complex movement item directly refer specific parkinsonian tremor rest cerebellar ocular motor dysfunction heelshin test feature motor examination section umsar base modify updrsiii item addition novel item heelkneeshin ataxia item score 0 normal 4 severe total score sum score item range 0 56 high score mean poor health 52 weeksclinical global impressionseverity cgis score cgis assess clinicians impression participant clinical condition clinician use total clinical experience participant population rate current severity participant illness 7point scale range 1 normal ill 7 extremely ill participant high score mean well health 52 weekschange baseline scale outcome parkinson disease   autonomic dysfunction scopaaut total score scopaaut patientreported outcome assess autonomic function autonomic function critical symptom domain msa scale selfcomplete participant consist 25 item assess follow domain gastrointestinal 7 item urinary 6 item cardiovascular 3 item thermoregulatory 4 item pupillomotor 1 item sexual 2 item man 2 item woman score item range 0 experience symptom 3 experience symptom total composite score include domain report score range 0 symptom 69 high burden symptom 52 weeksoverall survival os os define time day study drug administration death cause 52 weekschange baseline level cerebrospinal fluid csf free alphasynuclein αsyn 52 weekscmax maximum observe serum concentration tak341 predose day 1 29 57 85 169 253 337 multiple timepoint 24 hour postdose day 1 57 85 169 337 anytime day 365 427 early termination day 57 applicable early pk cohortstmax time occurrence cmax serum tak341 predose day 1 29 57 85 169 253 337 multiple timepoint 24 hour postdose day 1 57 85 169 337 anytime day 365 427 early termination day 57 applicable early pk cohortsaucτ area concentrationtime curve dose interval serum tak341 predose day 1 29 57 85 169 253 337 multiple timepoint 24 hour postdose day 1 57 85 169 337 anytime day 365 427 early termination day 57 applicable early pk cohortscsf concentration tak341 lumbar puncture csf sampling perform predose day 1 85 applicable early pk cohort 365number participant adverse event ae adverse event ae define untoward medical occurrence participant administer pharmaceutical product untoward medical occurrence necessarily causal relationship treatment datum report number participant analyze safety parameter include clinically significant abnormal value clinical laboratory evaluation vital sign ecg parameters physical examination neurological examination columbiasuicide severity rating scale cssrs 52 weeksnumber participant antidrug antibody 52 week',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 92,934 training samples
• Columns: Text1 and Text2
• Approximate statistics based on the first 1000 samples:

  	Text1	Text2
  type	string	string
  details	min: 29 tokens mean: 104.36 tokens max: 256 tokens	min: 8 tokens mean: 227.98 tokens max: 256 tokens

• Samples:

  Text1	Text2
  study people normal kidney function people reduce kidney function test bi 1467335 process body. The primary objective of the current study is to investigate the influence of moderate renal impairment on the pharmacokinetics of multiple doses in comparison to a matched control group with normal renal function.	Renal Insufficiency
  16w interventional study titration doseefficacy assessment exelon chinese alzheimers disease patient. To investigate the efficacy of Exelon capsule at maximal tolerated dose in mild to moderate Chinese AD patients via dosage titration from 3mg/d to 12mg/d in a 16 weeks duration	Alzheimer's Disease key inclusion criterion diagnosis dementia alzheimers type accord dsmiv criterion clinical diagnosis probable ad accord nincdsadrda criteria mmse score 10 26 treatment naïve patient stop donepezil galantamine huperzine memantine 2 week stable medical condition sign inform consent form patient hisher legal guardiankey exclusion criterion severe ad patient history cerebrovascular disease active uncontrolled epilepsy active hypothyroidism asthma cns infection neurodegenerative disorder advanced severe progressive unstable medical condition attend clinical trial take clinical trial drug score 4 modify hachinski ischemic scale mhis patient achei memantine mean change baseline alzheimer disease assessment scale cognitive subscale adascog alzheimer disease assessment scale cognitive subscale adascog measure change cognitive function alzheimer disease assessment scale adas scale measure specific cognitive behavior disorder alzheimer disease ad patient alzheimer di...
  case series saneso 360 gastroscope. To confirm the procedural performance of the Saneso 360° gastroscope in Esophago-gastro-duodenoscopy (EGD) procedures.	EGD Procedure inclusion criterion 18 74 year age willing able comply study procedure provide write inform consent participate study schedule clinically indicate routine egd procedure asa class 13exclusion criterion alter esophageal gastric duodenal anatomy pregnant woman child 18 year age adult 75 year age subject routine endoscopic procedure contraindicate comorbid medical condition patient currently enrol investigational study directly interfere current study prior write approval sponsor asa class 45 successful egd procedure success assess end procedure 1 procedure success define successful intubation portion duodenum photograph portion duodenum take 24 hour study day endoscopist qualitative rating saneso 360 gastroscope endoscopist rate experience saneso 360 gastroscope immediately follow completion study procedure 1 fivepoint likert scale 5 excellent 4 good 3 acceptable 2 difficult 1unacceptable 24 hour study dayendoscopist qualitative rating saneso 360 gastroscope compare past...

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          384,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss
0.0861	500	2.1564
0.1721	1000	1.6731
0.2582	1500	1.3615
0.3443	2000	1.331
0.4304	2500	1.2666
0.5164	3000	1.1645
0.6025	3500	1.159
0.6886	4000	1.0752
0.7747	4500	1.0458
0.8607	5000	1.0803
0.9468	5500	1.0237

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.3.1
• Transformers: 4.48.1
• PyTorch: 2.1.0a0+32f93b1
• Accelerate: 1.3.0
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}