shawhin/distilroberta-ai-job-embeddings

SentenceTransformer based on sentence-transformers/all-distilroberta-v1

This is a sentence-transformers model finetuned from sentence-transformers/all-distilroberta-v1 on the ai-job-embedding-finetuning dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-distilroberta-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • ai-job-embedding-finetuning

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("shawhin/distilroberta-ai-job-embeddings")
# Run inference
sentences = [
    'AI architecture design, technical leadership in AI/ML solutions, cybersecurity integration.',
    "experience.• Research, develop, and implement complex AI/ML solutions for internal initiatives and customer programs.• Lead the creation of AI architectures with varying frameworks and deployment models to devise and execute AI architecture strategies.• Be the key liaison between data scientists, data engineers, developers, operations (DevOps, DataOps, and MLOPs) and business leaders to govern and scale AI initiatives.• Lead key captures and their technical solutions (RFI & RFP responses) and program execution to ensure delivery of differentiated capabilities.• Lead, participant in, and propose R&D and solution development efforts related to AI/ML solutions and associated technologies for successful deployment• Work with program teams, Division leadership, and CTO organization to identify technology and solution roadmaps to improve mission enterprise capabilities and broad Leidos contribution in mission attainment resulting in increased contract growth and improved customer satisfaction• Follow, understand, and implement latest advancements in AI/ML and analytics.Basic Qualifications:• BS Degree in Computer Science, Data Science, Engineering, or related field and 12-15 years of prior relevant experience or Master's degree with 10-13 years of prior relevant experience. Additional years experience may be used in lieu of a degree.• US Citizenship and an active SECRET security clearance with ability to attain a TOP SECRET/SCI security clearance.• Demonstratable data science and advanced analytics experience including expert knowledge of advanced analytic tools such as SAS, R, and Python along with applied mathematics.• Effective communicator, able to capture action items and status clearly and concisely.• Multitask and manage competing priorities.• Work independently, engaging SMEs when needed, to drive work to closure on-time.• Experience with MS Excel, MS Word, MS SharePoint.• Significant hands-on work in the area of AI/ML may be considered in lieu of degree.Preferred Qualifications:• Top Secret Clearance• Certified Artificial Intelligence Consultant (CAIC ) certification or similar industry-recognized AI/ML certification• Recognized expert technologist with 10+ years' experience in AI/ML leading-edge technologies• Defense Industry experienceOriginal Posting Date:2024-11-12While subject to change based on business needs, Leidos reasonably anticipates that this job requisition will remain open for at least 3 days with an anticipated close date of no earlier than 3 days after the original posting date as listed above.Pay Range:Pay Range $126,100.00 - $227,950.00The Leidos pay range for this job level is a general guideline only and not a guarantee of compensation or salary. Additional factors considered in extending an offer include (but are not limited to) responsibilities of the job, education, experience, knowledge, skills, and abilities, as well as internal equity, alignment with market data, applicable bargaining agreement (if any), or other law.#Remote",
    "experience in data engineering. You have a solid understanding of the software development lifecycle and enjoy working in a team of engineers. The ideal candidate shows aptitude to evaluate and incorporate new technologies. You thrive in a work environment that requires creative problem-solving skills, independent self-direction, open communication and attention to details. You are a self-starter, comfortable with ambiguity and working in a fast-paced, ever-changing environment. You are passionate about bringing value to clients.As member of the Regulatory Tech team, you will:Work with engineers, project managers, technical leads, business owners and analysts throughout the whole SDLCDesign and implement new features in our core product's data platform and suspicious activity identifying mechanismBe brave enough to come up with ideas to improve resiliency, stability and performance of our platformParticipate in setting coding standards and guidelines, identify and document standard methodologiesDesired Skills and Experience:Hands-on experience with Python and SQLExperience with Snowflake databaseExperience with AirflowThorough knowledge of GIT, CI/CD and unit/end-to-end testingInterest in data engineeringSolid written and verbal communication skillsNice to have:Experience with DBT, Great Expectations frameworksExperience with Big Data technologies (Spark, Sqoop, HDFS, YARN)Experience with Agile developmentOur benefitsTo help you stay energized, engaged and inspired, we offer a wide range of employee benefits including: retirement investment and tools designed to help you in building a sound financial future; access to education reimbursement; comprehensive resources to support your physical health and emotional well-being; family support programs; and Flexible Time Off (FTO) so you can relax, recharge and be there for the people you care about.Our hybrid work modelBlackRock's hybrid work model is designed to enable a culture of collaboration and apprenticeship that enriches the experience of our employees, while supporting flexibility for all. Employees are currently required to work at least 4 days in the office per week, with the flexibility to work from home 1 day a week. Some business groups may require more time in the office due to their roles and responsibilities. We remain focused on increasing the impactful moments that arise when we work together in person - aligned with our commitment to performance and innovation. As a new joiner, you can count on this hybrid model to accelerate your learning and onboarding experience here at BlackRock.About BlackRockAt BlackRock, we are all connected by one mission: to help more and more people experience financial well-being. Our clients, and the people they serve, are saving for retirement, paying for their children's educations, buying homes and starting businesses. Their investments also help to strengthen the global economy: support businesses small and large; finance infrastructure projects that connect and power cities; and facilitate innovations that drive progress.This mission would not be possible without our smartest investment - the one we make in our employees. It's why we're dedicated to creating an environment where our colleagues feel welcomed, valued and supported with networks, benefits and development opportunities to help them thrive.For additional information on BlackRock, please visit @blackrock | Twitter: @blackrock | LinkedIn: www.linkedin.com/company/blackrockBlackRock is proud to be",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Datasets: ai-job-validation and ai-job-test
• Evaluated with TripletEvaluator

Metric	ai-job-validation	ai-job-test
cosine_accuracy	0.9878	0.9643

Training Details

Training Dataset

ai-job-embedding-finetuning

• Dataset: ai-job-embedding-finetuning at e501322
• Size: 660 training samples
• Columns: query, job_description_pos, and job_description_neg
• Approximate statistics based on the first 660 samples:

  	query	job_description_pos	job_description_neg
  type	string	string	string
  details	min: 8 tokens mean: 15.28 tokens max: 40 tokens	min: 47 tokens mean: 455.32 tokens max: 512 tokens	min: 47 tokens mean: 454.42 tokens max: 512 tokens

• Samples:

  query	job_description_pos	job_description_neg
  Data pipeline management, advanced SQL techniques, ETL/ELT processes.	skills and experiences. Even if you don't meet every requirement, we'd love to hear from you-you might be just what we're looking for, whether in this role or another.✨ Let's give businesses more time for what matters.Make your impact within a rapidly growing Fintech CompanyBuild and manage robust data pipelines that support scalable and efficient operations across various data platforms.Work closely with different teams to translate business requirements into sustainable technical solutions, facilitating effective data usage and integrationParticipate in designing, implementing, and refining data models and database schemas to effectively support business functionalities and operations.Collaborate on migration projects to modern data platforms like Trino, using Iceberg as the table format, and enhance data flow and architecture to improve data reliability, efficiency, and quality.Engage in continuous optimization of data models and pipelines, contributing to infrastructure migrations ...	requirements and expectations. In order to execute this mission, EUV Source Performance acts as the critical enabler of knowledge acquisition and transfer throughout the products development process, via:Substantive participation in the technology development and system definition process for products in definition and early development stage.Ensuring quality of the module to sub-system to system-level performance testing and validation for the products in integration phase.Driving technology improvements enabling new products field release and refinement of system specifications based upon deep understanding of factory and field performance.The creation and delivery of analytic tooling, action plans and procedures to drive maximum solving power of our customer support organizations for sustaining products.Collaborate to design and implement end-to-end AI and data science lifecycle, from data collection and preprocessing to model deployment and monitoring, providing technical leadershi...
  Data integrity, PowerBI reporting, SQL data management	requirements. Supporting initiatives for data integrity and normalization. Assessing tests and implementing new or upgraded software. Assisting with strategic decisions on new systems. Generating reports from single or multiple systems. Troubleshooting the reporting database environment and reports. Evaluating changes and updates to source production systems. Training end users on new reports and dashboards. Providing technical expertise on data storage structures, data mining, and data cleansing. Basic Qualifications: 6+ years of technical experience working as a Data Analyst. 5+ years of SQL experience (TSQL, Spark SQL, MYSQL, and/or PLSQL). 5+ years of PowerBI experience. 5+ year of experience with Microsoft Office Applications including Word, Excel, Access, and PowerPoint. 5+ years of experience assessing complex data sets and performing root cause analysis. Strong experience building data for claims analysis. Bachelor's Degree required. Preferred Skills: Advanced Degree in a quant...	experience in developing one or more of those models and applying them to solve a problem of interest. ResponsibilitiesConduct original research in developing, extending, and/or applying deep learning methods for solving an engineering design problemArchitect and implement a data generation and model training/testing pipelinePublish the outcome at a top-tier AI conferenceCreate a prototype/demo to showcase the solution and present the results to internal audiencesMinimum QualificationsCurrently pursuing a PhD degree in Computer Science, Mathematics, Physics, Computer and Electrical Engineering, Mechanical and Industrial Engineering, Aerospace Science and Engineering, Civil Engineering, or other related disciplinesExceptional Master's degree candidates will also be consideredDemonstrated success in solving difficult problems with deep learning methods (e.g., publication at top-tier conferences)Experience working with PyTorch and other data science and machine learning libraries in Pytho...
  data analysis, statistical techniques, data pipeline automation	skillset necessary for extracting value from a collection of novel data sets. In addition to being a highly proficient data analyst, this candidate is expected to foster a data pipeline by engineering analytics that lend themselves to automation and scale. As a function of engaging with stakeholders and taking ownership of outcomes, the candidate can comprehend the general problem space and is confident in socializing, demonstrating, and presenting insights generated from the data. This position requires work 100% on-site located at Linthicum, MD and/or Annapolis Junction, MD. Your requirements-analysis and stake-holder engagement. * Strong problem-solving skills and root-cause analysis to ensure that curated data is accurate, repeatable, and explainable. * Solid communication skills (written, verbal, and visual) * Ability to communicate data and engineering analysis regarding conclusions that were reached. * Ability to create and present data reports to support investigations. * Excel...	experiences. Our investments in technology infrastructure and world-class talent - along with our deep experience in machine learning - position us to be at the forefront of enterprises leveraging AI. From informing customers about unusual charges to answering their questions in real time, our applications of AI & ML are bringing humanity and simplicity to banking. We are committed to continuing to build world-class applied science and engineering teams to deliver our industry leading capabilities with breakthrough product experiences and scalable, high-performance AI infrastructure. At Capital One, you will help bring the transformative power of emerging AI capabilities to reimagine how we serve our customers and businesses who have come to love the products and services we build.Team Description:The Intelligent Foundations and Experiences (IFX) team is at the center of bringing our vision for AI at Capital One to life. We work hand-in-hand with our partners across the company to adva...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

ai-job-embedding-finetuning

• Dataset: ai-job-embedding-finetuning at e501322
• Size: 82 evaluation samples
• Columns: query, job_description_pos, and job_description_neg
• Approximate statistics based on the first 82 samples:

  	query	job_description_pos	job_description_neg
  type	string	string	string
  details	min: 9 tokens mean: 15.4 tokens max: 24 tokens	min: 79 tokens mean: 446.1 tokens max: 512 tokens	min: 105 tokens mean: 439.83 tokens max: 512 tokens

• Samples:

  query	job_description_pos	job_description_neg
  ML pipeline design, GenAI application development, RAG-based pipeline frameworks	experiences and belief systems-the ones that comprise us as individuals, shape who we are and make us unique. We believe your personal interests, identities, and desire to learn are part of your success here. Learn more about our diversity, equity, and inclusion efforts and the networks ZS supports to assist our ZSers in cultivating community spaces, obtaining the resources they need to thrive, and sharing the messages they are passionate about. Architecture & Engineering Specialist - ML EngineeringZS's India Capability & Expertise Center (CEC) houses more than 60% of ZS people across three offices in New Delhi, Pune and Bengaluru. Our teams work with colleagues across North America, Europe and East Asia to create and deliver real world solutions to the clients who drive our business. The CEC maintains standards of analytical, operational and technological excellence across our capability groups. Together, our collective knowledge enables each ZS team to deliver superior results to ou...	skills, able to prioritize asks, and move requests from the point of curiosity and into realized data products, insights, and solutions. Additional Description Your Skills & Abilities (Required Qualifications) Bachelor's degree in computer science, Data Science, Applied Mathematics, or related quantitative field, or equivalent combination of education and recent, relevant work experience. 5+ years of experience in full stack software development, machine learning, data science, or quantitative insights, and with data structures/algorithms. Strong programming skills in Python, Spark are necessary for implementing machine learning algorithms, data pipelines, and model development. Expertise in Machine Learning Algorithms and Techniques: Comprehensive understanding of diverse machine learning algorithms, spanning supervised and unsupervised learning, deep learning, reinforcement learning, and ensemble techniques. 2+ years of experience successfully leading technical teams or wo...
  large-scale system design, performance analysis, accessible technology development	qualifications: Bachelor's degree or equivalent practical experience. 2 years of experience with software development in one or more programming languages, or 1 year of experience with an advanced degree in an industry setting. 2 years of experience with data structures or algorithms in either an academic or industry setting. 2 years of experience with machine learning algorithms and tools (e.g., TensorFlow), artificial intelligence, deep learning or natural language processing. Preferred qualifications: Master's degree or PhD in Computer Science or related technical field. 2 years of experience with performance, large scale systems data analysis, visualization tools, or debugging. Experience developing accessible technologies. Proficiency in code and system health, diagnosis and resolution, and software test engineering. About the jobGoogle's software engineers develop the next-generation technologies that change how billions of users connect, explore, and interact with inform...	experience and products, using statistics, Artificial Intelligence and lots of creativity to predict our customers' behaviors. Our team strives for cutting edge model development techniques, from Machine Learning to Reinforcement Learning and beyond. We're partnering with business and technology to make speed of thought decisions possible.PrinciplesAs a Chapter, we identified a set of principles that guide behaviors we admire in each other and we pursue as a team:Diverse ensembles don't overfit;We reinforce each other's learning;Mind the person behind the data point;We share the same objective function;We trust our confidence interval;Pursue the global maxima.We believe in:Good team chemistry;Enthusiasm for building and delivering new features and products;Passion for learning new things while constantly improving what we are already good at;Collaborating efficiently to ship a great product experience to our customers.The positionWe are looking for a Senior Data Scientists to integrate...
  Cloud ecosystems, distributed systems, Large Language Models	experience with Golang and LLMs is not required, but will be helpful.CrowdStrike is a computer security company, but we do not require candidates for this role to have prior security industry experience. We will mentor and train in security topics as needed. We do expect a strong interest in CrowdStrike's mission and a willingness to engage with the needs of our product teams and customers.About the Product: CrowdStrike has pioneered the use of artificial intelligence (AI) since we first introduced AI-powered protection to replace signature-based antivirus over 10 years ago, and we've continued to deeply integrate it across our platform since.Using the recent advances in Large Language Models technologies, CrowdStrike introduced Charlotte AI, a new generative AI security analyst. It uses the world's highest-fidelity security data and it is seamlessly integrated with CrowdStrike's industry-leading threat hunters, managed detection and response operators, and incident response experts. C...	skills, a deep understanding of database technology and data use within analytic platforms, and they must have strong programming skills with SQL and/or Python.We expect our Senior Lead Data Engineers to have demonstrated fluency and competency in both the technology language of IT and the analyst language of Supply Chain analysts. They have experience developing and implementing data models for analytic use and a moderate level of experience with cloud database architecture. They will need to be fast learners with a keen eye for detail, systems thinking, and process design. They must be team players who work steadfastly to create impactful change. Our Flexible Future model offers a healthy mix of working in person and remotely, strengthening key elements of the Chick-fil-A culture by fostering collaboration and community. Responsibilities The person who fills this role will be expected to do the following as a part of their regular work requirements and expectations Help design patt...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	ai-job-validation_cosine_accuracy	ai-job-test_cosine_accuracy
0	0	0.9390	-
1.0	42	0.9878	0.9643

Framework Versions

• Python: 3.12.8
• Sentence Transformers: 3.3.1
• Transformers: 4.48.0
• PyTorch: 2.5.1
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}