README.md

---
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
base_model: Shuu12121/CodeModernBERT-Owl
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
- code_eval
model-index:
- name: SentenceTransformer based on Shuu12121/CodeModernBERT-Owl
  results:
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: code docstring dev
      type: code-docstring-dev
    metrics:
    - type: pearson_cosine
      value: null
      name: Pearson Cosine
    - type: spearman_cosine
      value: null
      name: Spearman Cosine
license: apache-2.0
datasets:
- code-search-net/code_search_net
- Shuu12121/java-codesearch-dataset-open
- Shuu12121/rust-codesearch-dataset-open
- google/code_x_glue_ct_code_to_text
language:
- en
---





# SentenceTransformer based on Shuu12121/CodeModernBERT-Owl🦉



This model is a **sentence-transformers** model fine-tuned from **[Shuu12121/CodeModernBERT-Owl](https://huggingface.co/Shuu12121/CodeModernBERT-Owl)**, which is a **ModernBERT model specifically designed for code, pre-trained from scratch by me**.  
**It is specifically designed for code search and efficiently calculates semantic similarity between code snippets and documentation.**  
One of the key features of this model is its **maximum sequence length of 2048 tokens**, which allows it to handle moderately long code snippets and documentation.  
Despite being a relatively small model with about **150 million parameters**, it demonstrates remarkable performance in code search tasks.  

---

このモデルは、**私が一から事前学習を行ったコード特化のModernBERTモデルである [Shuu12121/CodeModernBERT-Owl](https://huggingface.co/Shuu12121/CodeModernBERT-Owl)** をベースにファインチューニングされた **[sentence-transformers](https://www.SBERT.net)** モデルです。  
**特にコードサーチに特化しており、コード片やドキュメントから効果的に意味的類似性を計算できる** ように設計されています。  
本モデルの特徴として、**最大シーケンス長が2048トークン**に対応しており、**中程度の長さのコード片やドキュメントにも対応可能**です。  
**150M程度と比較的小さいモデル**ながらも、コード検索タスクにおいて高い性能を発揮します。


---

### Model Evaluation / モデル評価

#### CoIR Evaluation Results / CoIRにおける評価結果

Despite being a relatively small model with around **150M parameters**, this model achieved an impressive **76.89** on the **CodeSearchNet** benchmark, demonstrating its high performance in code search tasks.  
Since this model is specialized for code search, it does not support other tasks, and thus evaluation scores for other tasks are not provided.  
In the CodeSearchNet task, this model outperforms many well-known models, as shown in the comparison table below.  

このモデルは、**150M程度と比較的小さいモデル**ながら、**コードサーチタスクにおける評価指標である CodeSearchNet で 76.89** を達成しました。  
他のタスクには対応していないため、評価値は提供されていません。  
CodeSearchNetタスクにおける評価値としては、他の有名なモデルと比較しても高いパフォーマンスを示しています。  

| Model Name                                    | CodeSearchNet Score |
|-----------------------------------------------|----------------------|
| **Shuu12121/CodeModernBERT-Owl**                | **76.89**             |
| Salesforce/SFR-Embedding-Code-2B_R              | 73.5                  |
| CodeSage-large-v2                              | 94.26                 |
| Salesforce/SFR-Embedding-Code-400M_R             | 72.53                 |
| CodeSage-large                                 | 90.58                 |
| Voyage-Code-002                                | 81.79                 |
| E5-Mistral                                     | 54.25                 |
| E5-Base-v2                                     | 67.99                 |
| OpenAI-Ada-002                                 | 74.21                 |
| BGE-Base-en-v1.5                               | 69.6                  |
| BGE-M3                                         | 43.23                 |
| UniXcoder                                      | 60.2                  |
| GTE-Base-en-v1.5                               | 43.35                 |
| Contriever                                     | 34.72                 |

---

### Model Details / モデル詳細

- **Model Type / モデルタイプ:** Sentence Transformer  
- **Base Model / ベースモデル:** [Shuu12121/CodeModernBERT-Owl](https://huggingface.co/Shuu12121/CodeModernBERT-Owl)  
- **Maximum Sequence Length / 最大シーケンス長:** 2048 tokens  
- **Output Dimensions / 出力次元:** 768 dimensions  
- **Similarity Function / 類似度関数:** Cosine Similarity  
- **License / ライセンス:** Apache-2.0  

---

### Usage / 使用方法

#### Installation / インストール

To install Sentence Transformers, run the following command:  
Sentence Transformers をインストールするには、以下のコマンドを実行します。  

```bash
pip install -U sentence-transformers
```

#### Model Loading and Inference / モデルのロードと推論

```python
from sentence_transformers import SentenceTransformer

# Load the model / モデルをダウンロードしてロード
model = SentenceTransformer("Shuu12121/CodeSearch-ModernBERT-Owl")

# Example sentences for inference / 推論用の文リスト
sentences = [
    'Encrypts the zip file',
    'def freeze_encrypt(dest_dir, zip_filename, config, opt):\n    \n    pgp_keys = grok_keys(config)\n    icefile_prefix = "aomi-%s" % \\\n                     os.path.basename(os.path.dirname(opt.secretfile))\n    if opt.icefile_prefix:\n        icefile_prefix = opt.icefile_prefix\n\n    timestamp = time.strftime("%H%M%S-%m-%d-%Y",\n                              datetime.datetime.now().timetuple())\n    ice_file = "%s/%s-%s.ice" % (dest_dir, icefile_prefix, timestamp)\n    if not encrypt(zip_filename, ice_file, pgp_keys):\n        raise aomi.exceptions.GPG("Unable to encrypt zipfile")\n\n    return ice_file',
    'def transform(self, sents):\n        \n\n        def convert(tokens):\n            return torch.tensor([self.vocab.stoi[t] for t in tokens], dtype=torch.long)\n\n        if self.vocab is None:\n            raise Exception(\n                "Must run .fit() for .fit_transform() before " "calling .transform()."\n            )\n\n        seqs = sorted([convert(s) for s in sents], key=lambda x: -len(x))\n        X = torch.LongTensor(pad_sequence(seqs, batch_first=True))\n        return X',
]

# Generate embeddings / 埋め込みベクトルの生成
embeddings = model.encode(sentences)
print(embeddings.shape)  # Output: [3, 768]

# Calculate similarity scores / 類似度スコアの計算
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)  # Output: [3, 3]
```

---

### Library Versions / ライブラリバージョン

- Python: 3.11.11  
- Sentence Transformers: 3.4.1  
- Transformers: 4.50.0  
- PyTorch: 2.6.0+cu124  
- Accelerate: 1.5.2  
- Datasets: 3.4.1  
- Tokenizers: 0.21.1  

---

### Citation / 引用情報

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```