TatvaRA/clip-skin-embeddings

SentenceTransformer based on sentence-transformers/clip-ViT-B-32

This is a sentence-transformers model finetuned from sentence-transformers/clip-ViT-B-32 on the skin-positive-negative-pairs dataset. It maps sentences & paragraphs to a None-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/clip-ViT-B-32
• Maximum Sequence Length: 77 tokens
• Output Dimensionality: None dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • skin-positive-negative-pairs

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): CLIPModel()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("TatvaRA/clip-skin-embeddings")
# Run inference
sentences = [
    'The image depicts the abdominal region with a reticulated, or net-like, erythematous pattern on the skin. The area shows a faint, reddish-brown discoloration with a lacy appearance, indicative of erythema ab igne. The borders of the discoloration are irregular but well-demarcated. No blistering or open lesions are visible, and the skin surface appears intact without signs of scaling or crusting.',
    'The image depicts multiple small, soft, flesh-colored to slightly hyperpigmented skin lesions located on the eyelid and surrounding skin. These lesions are pedunculated with a smooth surface, typical of skin tags (acrochordons). Some are more prominent and raised, while others are flatter against the skin surface. The surrounding skin appears slightly wrinkled, consistent with normal aging changes.',
    'The image shows an elderly person with erythematous patches and papules prominently located on the central face, including the cheeks, nose, and forehead. The skin findings display a symmetrical distribution. The nose is slightly bulbous, suggesting possible rhinophyma changes. The lesions do not have well-defined borders and there is no visible scaling or pustular formations. The background skin appears sun-damaged with fine wrinkling, but without significant telangiectasia visible in this image.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3884, 0.3253],
#         [0.3884, 1.0000, 0.5531],
#         [0.3253, 0.5531, 1.0000]])

Evaluation

Metrics

Triplet

• Datasets: skin-dataset-train and skin-dataset-valid
• Evaluated with TripletEvaluator

Metric	skin-dataset-train	skin-dataset-valid
cosine_accuracy	1.0	1.0

Training Details

Training Dataset

skin-positive-negative-pairs

• Dataset: skin-positive-negative-pairs at a6e13e3
• Size: 105 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 105 samples:

  	anchor	positive	negative
  type	PIL.PngImagePlugin.PngImageFile	string	string
  details		min: 46 tokens mean: 75.37 tokens max: 77 tokens	min: 46 tokens mean: 75.55 tokens max: 77 tokens

• Samples:

  anchor	positive	negative
  	The image depicts a melanocytic nevus located on the skin. It is characterized by a well-defined, round brown lesion centrally located on a lighter skin background. The pigmentation appears uniform with a slightly darker speckled pattern distributed evenly throughout the nevus. The borders are smooth and clearly demarcated from the surrounding skin. There is no apparent erythema, ulceration, or significant secondary changes. Sparse hair is visible around the lesion, and the overall lesion is flat, without palpable elevation.	The image shows the upper lip, featuring a cluster of small, grouped vesicles on an erythematous base. The vesicles are clear, filled with fluid, and some are beginning to coalesce. There is minimal surrounding erythema, and the lesions are located at the vermillion border of the lip. The borders of the vesicles are well-defined, and there are no signs of crusting at this stage.
  	The image shows a well-defined, erythematous plaque on the skin with a rough, scaly surface. The lesion is located on a hand and exhibits significant inflammation and lichenification. There is evidence of excoriation and possible oozing, suggesting chronic irritation and scratching. The borders of the lesion are distinct, and the surrounding skin appears dry.	The image displays multiple open comedones, appearing as black or dark brown plugs, located on the periocular region of the face. These lesions have a characteristic distribution, clustering under the eye and on the upper cheek area. The surrounding skin exhibits signs of chronic sun damage, including fine wrinkles and a mildly rough texture. The borders of the comedones are well-defined, and there is no evidence of erythema or secondary infection around the lesions.
  	The image shows a close-up view of the skin with an irregularly shaped lesion displaying a mixture of colors, including pink, brown, and some erythematous areas. The lesion has uneven borders and appears slightly raised with scaly, crusted surfaces. There are areas of keratinization, with the lesion being distributed in a somewhat asymmetrical pattern. Surrounding skin appears less affected, with some visible telangiectasia.	The image shows the back of an individual with multiple vesicular lesions. These lesions are scattered across the skin surface and vary in size. They appear as small, red, blister-like eruptions that are slightly raised. Some lesions exhibit a clear vesicular head, while others are crusted over, indicating different stages of development. The distribution is widespread across the back, with a noticeable absence of clustering. The lesions have well-defined borders, and the surrounding skin appears otherwise normal without significant erythema or swelling.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

skin-positive-negative-pairs

• Dataset: skin-positive-negative-pairs at a6e13e3
• Size: 22 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 22 samples:

  	anchor	positive	negative
  type	PIL.PngImagePlugin.PngImageFile	string	string
  details		min: 74 tokens mean: 76.59 tokens max: 77 tokens	min: 60 tokens mean: 75.59 tokens max: 77 tokens

• Samples:

  anchor	positive	negative
  	The image shows an area of skin with multiple open comedones, commonly known as blackheads, scattered across the affected area. These comedones are predominantly located on the cheek and temple of the face. The skin appears weathered and sallow, with some areas exhibiting signs of solar damage, including wrinkling and yellowish discoloration. The borders of the lesions are irregular, and there are multiple large, open comedones. There are secondary changes, such as mild erythema and induration in some areas, suggesting chronic sun exposure.	The image displays toenails with signs of fungal infection. The nails appear thickened and discolored, predominantly yellowish-brown. There is noticeable onycholysis, especially in the larger toenail, with crumbling and irregular borders. The surface of the nails looks rough and uneven, and there is subungual debris. The surrounding skin appears intact without visible erythema or scaling.
  	The image shows a circular, red lesion on the skin, located on the cheek. It features a central elevated red spot with radiating capillaries, resembling a spider's web. The surrounding skin appears slightly erythematous. The lesion has well-defined borders and is isolated with no other lesions visible in the immediate vicinity.	The image shows an area of skin with multiple erythematous, excoriated papules and plaques, some of which are coalescing. The lesions are distributed on the flexural aspect of the extremity. The skin appears dry and lichenified with some areas showing slight scaling. Borders of the individual lesions are irregular, and there is no clear demarcation from the surrounding skin.
  	The image shows an elderly person with erythematous patches and papules prominently located on the central face, including the cheeks, nose, and forehead. The skin findings display a symmetrical distribution. The nose is slightly bulbous, suggesting possible rhinophyma changes. The lesions do not have well-defined borders and there is no visible scaling or pustular formations. The background skin appears sun-damaged with fine wrinkling, but without significant telangiectasia visible in this image.	The image shows a round, erythematous (red) lesion with a well-defined, slightly raised border. The center of the lesion appears to be clearer with minor scaling, typical of a ringworm (tinea corporis) infection. The lesion is located on skin likely from a limb or torso. The margins are distinct with some subtle scaling toward the periphery.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 0.0001
• num_train_epochs: 7

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0001
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 7
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	skin-dataset-train_cosine_accuracy	skin-dataset-valid_cosine_accuracy
-1	-1	-	-	0.9619	1.0
0.1429	1	2.7596	-	-	-
0.2857	2	2.5414	-	-	-
0.4286	3	2.6204	-	-	-
0.5714	4	2.4667	-	-	-
0.7143	5	2.4014	-	-	-
0.8571	6	2.5579	-	-	-
1.0	7	2.0899	2.2113	-	-
1.1429	8	2.1459	-	-	-
1.2857	9	2.0529	-	-	-
1.4286	10	1.8848	-	-	-
1.5714	11	2.0387	-	-	-
1.7143	12	2.0944	-	-	-
1.8571	13	2.1068	-	-	-
2.0	14	1.6342	2.1221	-	-
0.1429	1	1.7895	-	-	-
0.2857	2	1.6297	-	-	-
0.4286	3	1.9204	-	-	-
0.5714	4	1.9449	-	-	-
0.7143	5	1.8608	-	-	-
0.8571	6	2.186	-	-	-
1.0	7	1.522	1.8561	-	-
1.1429	8	1.5326	-	-	-
1.2857	9	1.3086	-	-	-
1.4286	10	1.1582	-	-	-
1.5714	11	1.4275	-	-	-
1.7143	12	1.3807	-	-	-
1.8571	13	1.4184	-	-	-
2.0	14	0.909	1.7285	-	-
2.1429	15	1.2727	-	-	-
2.2857	16	1.3549	-	-	-
2.4286	17	1.0448	-	-	-
2.5714	18	1.0773	-	-	-
2.7143	19	1.1036	-	-	-
2.8571	20	0.9444	-	-	-
3.0	21	0.5148	1.6311	-	-
3.1429	22	0.8764	-	-	-
3.2857	23	0.809	-	-	-
3.4286	24	0.8724	-	-	-
3.5714	25	0.8276	-	-	-
3.7143	26	0.8923	-	-	-
3.8571	27	0.797	-	-	-
4.0	28	0.5056	1.5989	-	-
4.1429	29	0.7625	-	-	-
4.2857	30	0.6356	-	-	-
4.4286	31	0.8352	-	-	-
4.5714	32	0.6936	-	-	-
4.7143	33	0.6338	-	-	-
4.8571	34	0.5392	-	-	-
5.0	35	0.3826	1.5942	-	-
5.1429	36	0.6252	-	-	-
5.2857	37	0.5969	-	-	-
5.4286	38	0.4948	-	-	-
5.5714	39	0.7617	-	-	-
5.7143	40	0.6775	-	-	-
5.8571	41	0.6289	-	-	-
6.0	42	0.2195	1.5698	-	-
6.1429	43	0.5786	-	-	-
6.2857	44	0.6049	-	-	-
6.4286	45	0.5162	-	-	-
6.5714	46	0.5841	-	-	-
6.7143	47	0.6579	-	-	-
6.8571	48	0.6702	-	-	-
7.0	49	0.2975	1.5598	-	-
-1	-1	-	-	1.0	1.0

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.0.0
• Transformers: 4.53.2
• PyTorch: 2.7.1+cu126
• Accelerate: 1.9.0
• Datasets: 4.0.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}