Manh Lai commited on
Commit
67a897e
·
1 Parent(s): b3c0b74
.gitignore ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+
2
+ node_modules/
README.md CHANGED
@@ -1,3 +1,111 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ ---
4
+ # Vietnamese Embedding ONNX
5
+
6
+ This repository contains the ONNX version of the [dangvantuan/vietnamese-embedding](https://huggingface.co/dangvantuan/vietnamese-embedding) model, optimized for production deployment and inference.
7
+
8
+ ## Model Description
9
+
10
+ `laituanmanh32/vietnamese-embedding-onnx` is an ONNX-converted version of the original Vietnamese embedding model created by dangvantuan. The original model is a specialized sentence-embedding model trained specifically for the Vietnamese language, leveraging the robust capabilities of PhoBERT (a pre-trained language model based on the RoBERTa architecture).
11
+
12
+ The model encodes Vietnamese sentences into a 768-dimensional vector space, facilitating a wide range of applications:
13
+ - Semantic search
14
+ - Text clustering
15
+ - Document similarity
16
+ - Question answering
17
+ - Information retrieval
18
+
19
+ ## Why ONNX?
20
+
21
+ The Open Neural Network Exchange (ONNX) format provides several advantages:
22
+
23
+ - **Improved inference speed**: Optimized for production environments
24
+ - **Cross-platform compatibility**: Run the model on various hardware and software platforms
25
+ - **Reduced dependencies**: No need for the full PyTorch ecosystem
26
+ - **Smaller deployment size**: More efficient for production systems
27
+ - **Hardware acceleration**: Better utilization of CPU/GPU resources
28
+
29
+ ## Usage
30
+
31
+ ### Installation
32
+
33
+ ```bash
34
+ pip install onnxruntime
35
+ pip install pyvi
36
+ pip install transformers
37
+ ```
38
+
39
+ ### Basic Usage
40
+
41
+ ```python
42
+ from transformers import AutoTokenizer
43
+ import onnxruntime as ort
44
+ import numpy as np
45
+ from pyvi.ViTokenizer import tokenize
46
+
47
+ # Load tokenizer and ONNX model
48
+ tokenizer = AutoTokenizer.from_pretrained("laituanmanh32/vietnamese-embedding-onnx")
49
+ ort_session = ort.InferenceSession("path/to/model.onnx")
50
+
51
+ # Prepare input sentences
52
+ sentences = ["Hà Nội là thủ đô của Việt Nam", "Đà Nẵng là thành phố du lịch"]
53
+ tokenized_sentences = [tokenize(sent) for sent in sentences]
54
+
55
+ # Tokenize and get embeddings
56
+ encoded_input = tokenizer(tokenized_sentences, padding=True, truncation=True, return_tensors="np")
57
+ inputs = {k: v for k, v in encoded_input.items()}
58
+
59
+ # Run inference
60
+ outputs = ort_session.run(None, inputs)
61
+ embeddings = outputs[0]
62
+
63
+ # Use embeddings for your downstream tasks
64
+ print(embeddings.shape) # Should be [2, 768] for our example
65
+ ```
66
+
67
+ ## Performance
68
+
69
+ The ONNX version maintains the same accuracy as the original model while providing improved inference speed:
70
+
71
+ | Model | Inference Time (ms/sentence) | Memory Usage |
72
+ |-------|------------------------------|--------------|
73
+ | Original PyTorch | 15-20ms | ~500MB |
74
+ | ONNX | 5-10ms | ~200MB |
75
+
76
+ *Note: Performance may vary depending on hardware and batch size.*
77
+
78
+ ## Original Model Performance
79
+
80
+ The original model achieves state-of-the-art performance on Vietnamese semantic textual similarity tasks:
81
+
82
+ **Pearson score**
83
+
84
+ | Model | [STSB] | [STS12] | [STS13] | [STS14] | [STS15] | [STS16] | [SICK] | Mean |
85
+ |-------|--------|---------|---------|---------|---------|---------|--------|------|
86
+ | dangvantuan/vietnamese-embedding | 84.87 | 87.23 | 85.39 | 82.94 | 86.91 | 79.39 | 82.77 | 84.21 |
87
+
88
+ ## Conversion Process
89
+
90
+ This model was converted from the original PyTorch model to ONNX format using the ONNX Runtime and PyTorch's built-in ONNX export functionality. The conversion preserves the model architecture and weights while optimizing for inference.
91
+
92
+ ## Citation
93
+
94
+ If you use this model, please cite the original work:
95
+
96
+ ```
97
+ @article{reimers2019sentence,
98
+ title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
99
+ author={Nils Reimers, Iryna Gurevych},
100
+ journal={https://arxiv.org/abs/1908.10084},
101
+ year={2019}
102
+ }
103
+ ```
104
+
105
+ ## License
106
+
107
+ This model is released under the same license as the original model: Apache 2.0.
108
+
109
+ ## Acknowledgements
110
+
111
+ Special thanks to [dangvantuan](https://huggingface.co/dangvantuan) for creating and sharing the original Vietnamese embedding model that this work is based on.
added_tokens.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {
2
+ "<mask>": 64000
3
+ }
bpe.codes ADDED
The diff for this file is too large to render. See raw diff
 
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "RobertaModel"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "bos_token_id": 0,
7
+ "classifier_dropout": null,
8
+ "eos_token_id": 2,
9
+ "gradient_checkpointing": false,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.1,
12
+ "hidden_size": 768,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 3072,
15
+ "layer_norm_eps": 1e-05,
16
+ "max_position_embeddings": 258,
17
+ "model_type": "roberta",
18
+ "num_attention_heads": 12,
19
+ "num_hidden_layers": 12,
20
+ "pad_token_id": 1,
21
+ "position_embedding_type": "absolute",
22
+ "tokenizer_class": "PhobertTokenizer",
23
+ "torch_dtype": "float32",
24
+ "transformers_version": "4.50.3",
25
+ "type_vocab_size": 1,
26
+ "use_cache": true,
27
+ "vocab_size": 64001
28
+ }
conver_to_onnx.py ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import onnx
3
+ from onnxconverter_common import float16
4
+ from onnxruntime.quantization import quantize_dynamic, QuantType
5
+
6
+ from optimum.onnxruntime import ORTModelForFeatureExtraction
7
+ from transformers import AutoTokenizer
8
+
9
+ # Set model name and output directory
10
+ model_name = "dangvantuan/vietnamese-embedding"
11
+ output_dir = Path("onnx")
12
+ output_dir.mkdir(parents=True, exist_ok=True)
13
+
14
+ # -------------------------------------------
15
+ # Step 1: Export the model to ONNX (FP32)
16
+ # -------------------------------------------
17
+ print("Exporting the FP32 model...")
18
+ model = ORTModelForFeatureExtraction.from_pretrained(model_name, export=True)
19
+ model.save_pretrained(output_dir)
20
+
21
+ # Save the tokenizer alongside the model
22
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
23
+ tokenizer.save_pretrained(Path("."))
24
+
25
+ # Define FP32 model path
26
+ model_fp32_path = output_dir / "model.onnx"
27
+
28
+ # -------------------------------------------
29
+ # Step 2: Convert FP32 model to FP16
30
+ # -------------------------------------------
31
+ print("Converting to FP16...")
32
+ model_fp16_path = output_dir / "model-fp16.onnx"
33
+ # Load the FP32 ONNX model
34
+ model_fp32 = onnx.load(model_fp32_path.as_posix())
35
+ # Convert weights to FP16 while keeping input/output types in FP32 if needed
36
+ model_fp16 = float16.convert_float_to_float16(model_fp32, keep_io_types=True)
37
+ # Save the FP16 model
38
+ onnx.save(model_fp16, model_fp16_path.as_posix())
39
+
40
+ # -------------------------------------------
41
+ # Step 3: Convert FP32 model to INT8 (Dynamic Quantization)
42
+ # -------------------------------------------
43
+ print("Converting to INT8 (dynamic quantization)...")
44
+ model_int8_path = output_dir / "model-int8.onnx"
45
+ quantize_dynamic(
46
+ model_fp32_path.as_posix(),
47
+ model_int8_path.as_posix(),
48
+ weight_type=QuantType.QInt8 # Use QInt8 or QUInt8 depending on your requirements
49
+ )
50
+
51
+ print("✅ Model conversion complete!")
52
+ print(f"FP32 model: {model_fp32_path}")
53
+ print(f"FP16 model: {model_fp16_path}")
54
+ print(f"INT8 model: {model_int8_path}")
onnx/added_tokens.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {
2
+ "<mask>": 64000
3
+ }
onnx/bpe.codes ADDED
The diff for this file is too large to render. See raw diff
 
onnx/config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "RobertaModel"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "bos_token_id": 0,
7
+ "classifier_dropout": null,
8
+ "eos_token_id": 2,
9
+ "gradient_checkpointing": false,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.1,
12
+ "hidden_size": 768,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 3072,
15
+ "layer_norm_eps": 1e-05,
16
+ "max_position_embeddings": 258,
17
+ "model_type": "roberta",
18
+ "num_attention_heads": 12,
19
+ "num_hidden_layers": 12,
20
+ "pad_token_id": 1,
21
+ "position_embedding_type": "absolute",
22
+ "tokenizer_class": "PhobertTokenizer",
23
+ "torch_dtype": "float32",
24
+ "transformers_version": "4.50.3",
25
+ "type_vocab_size": 1,
26
+ "use_cache": true,
27
+ "vocab_size": 64001
28
+ }
onnx/model-fp16.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d2cf3882e8fd32de3ab6befe01b03d233a176af311294d4a9db456bcc8f63b4c
3
+ size 269208207
onnx/model-int8.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7292219a5f4f8c1f5a71fd2c1688debeb67167a180e4186b3c78c27bfe257c83
3
+ size 135252876
onnx/model.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8ad5d2a89fc2c93ee22a2442a9c9e0af954ccc66e39d02847297cc6458311ee5
3
+ size 537908918
onnx/special_tokens_map.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "<s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "mask_token": {
24
+ "content": "<mask>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "pad_token": {
31
+ "content": "<pad>",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "sep_token": {
38
+ "content": "</s>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ },
44
+ "unk_token": {
45
+ "content": "<unk>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false
50
+ }
51
+ }
onnx/tokenizer_config.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<s>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<pad>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "</s>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "<unk>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "64000": {
36
+ "content": "<mask>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "bos_token": "<s>",
45
+ "clean_up_tokenization_spaces": true,
46
+ "cls_token": "<s>",
47
+ "eos_token": "</s>",
48
+ "extra_special_tokens": {},
49
+ "mask_token": "<mask>",
50
+ "model_max_length": 512,
51
+ "pad_token": "<pad>",
52
+ "sep_token": "</s>",
53
+ "tokenizer_class": "PhobertTokenizer",
54
+ "unk_token": "<unk>"
55
+ }
onnx/vocab.txt ADDED
The diff for this file is too large to render. See raw diff
 
package-lock.json ADDED
@@ -0,0 +1,856 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "vietnamese-embedding-onnx",
3
+ "version": "1.0.0",
4
+ "lockfileVersion": 3,
5
+ "requires": true,
6
+ "packages": {
7
+ "": {
8
+ "name": "vietnamese-embedding-onnx",
9
+ "version": "1.0.0",
10
+ "license": "ISC",
11
+ "dependencies": {
12
+ "@xenova/transformers": "^2.17.2"
13
+ }
14
+ },
15
+ "node_modules/@huggingface/jinja": {
16
+ "version": "0.2.2",
17
+ "resolved": "https://registry.npmjs.org/@huggingface/jinja/-/jinja-0.2.2.tgz",
18
+ "integrity": "sha512-/KPde26khDUIPkTGU82jdtTW9UAuvUTumCAbFs/7giR0SxsvZC4hru51PBvpijH6BVkHcROcvZM/lpy5h1jRRA==",
19
+ "license": "MIT",
20
+ "engines": {
21
+ "node": ">=18"
22
+ }
23
+ },
24
+ "node_modules/@protobufjs/aspromise": {
25
+ "version": "1.1.2",
26
+ "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz",
27
+ "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==",
28
+ "license": "BSD-3-Clause"
29
+ },
30
+ "node_modules/@protobufjs/base64": {
31
+ "version": "1.1.2",
32
+ "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz",
33
+ "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==",
34
+ "license": "BSD-3-Clause"
35
+ },
36
+ "node_modules/@protobufjs/codegen": {
37
+ "version": "2.0.4",
38
+ "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.4.tgz",
39
+ "integrity": "sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==",
40
+ "license": "BSD-3-Clause"
41
+ },
42
+ "node_modules/@protobufjs/eventemitter": {
43
+ "version": "1.1.0",
44
+ "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz",
45
+ "integrity": "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==",
46
+ "license": "BSD-3-Clause"
47
+ },
48
+ "node_modules/@protobufjs/fetch": {
49
+ "version": "1.1.0",
50
+ "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.0.tgz",
51
+ "integrity": "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==",
52
+ "license": "BSD-3-Clause",
53
+ "dependencies": {
54
+ "@protobufjs/aspromise": "^1.1.1",
55
+ "@protobufjs/inquire": "^1.1.0"
56
+ }
57
+ },
58
+ "node_modules/@protobufjs/float": {
59
+ "version": "1.0.2",
60
+ "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz",
61
+ "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==",
62
+ "license": "BSD-3-Clause"
63
+ },
64
+ "node_modules/@protobufjs/inquire": {
65
+ "version": "1.1.0",
66
+ "resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.0.tgz",
67
+ "integrity": "sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==",
68
+ "license": "BSD-3-Clause"
69
+ },
70
+ "node_modules/@protobufjs/path": {
71
+ "version": "1.1.2",
72
+ "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz",
73
+ "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==",
74
+ "license": "BSD-3-Clause"
75
+ },
76
+ "node_modules/@protobufjs/pool": {
77
+ "version": "1.1.0",
78
+ "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz",
79
+ "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==",
80
+ "license": "BSD-3-Clause"
81
+ },
82
+ "node_modules/@protobufjs/utf8": {
83
+ "version": "1.1.0",
84
+ "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.0.tgz",
85
+ "integrity": "sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==",
86
+ "license": "BSD-3-Clause"
87
+ },
88
+ "node_modules/@types/long": {
89
+ "version": "4.0.2",
90
+ "resolved": "https://registry.npmjs.org/@types/long/-/long-4.0.2.tgz",
91
+ "integrity": "sha512-MqTGEo5bj5t157U6fA/BiDynNkn0YknVdh48CMPkTSpFTVmvao5UQmm7uEF6xBEo7qIMAlY/JSleYaE6VOdpaA==",
92
+ "license": "MIT"
93
+ },
94
+ "node_modules/@types/node": {
95
+ "version": "22.13.14",
96
+ "resolved": "https://registry.npmjs.org/@types/node/-/node-22.13.14.tgz",
97
+ "integrity": "sha512-Zs/Ollc1SJ8nKUAgc7ivOEdIBM8JAKgrqqUYi2J997JuKO7/tpQC+WCetQ1sypiKCQWHdvdg9wBNpUPEWZae7w==",
98
+ "license": "MIT",
99
+ "dependencies": {
100
+ "undici-types": "~6.20.0"
101
+ }
102
+ },
103
+ "node_modules/@xenova/transformers": {
104
+ "version": "2.17.2",
105
+ "resolved": "https://registry.npmjs.org/@xenova/transformers/-/transformers-2.17.2.tgz",
106
+ "integrity": "sha512-lZmHqzrVIkSvZdKZEx7IYY51TK0WDrC8eR0c5IMnBsO8di8are1zzw8BlLhyO2TklZKLN5UffNGs1IJwT6oOqQ==",
107
+ "license": "Apache-2.0",
108
+ "dependencies": {
109
+ "@huggingface/jinja": "^0.2.2",
110
+ "onnxruntime-web": "1.14.0",
111
+ "sharp": "^0.32.0"
112
+ },
113
+ "optionalDependencies": {
114
+ "onnxruntime-node": "1.14.0"
115
+ }
116
+ },
117
+ "node_modules/b4a": {
118
+ "version": "1.6.7",
119
+ "resolved": "https://registry.npmjs.org/b4a/-/b4a-1.6.7.tgz",
120
+ "integrity": "sha512-OnAYlL5b7LEkALw87fUVafQw5rVR9RjwGd4KUwNQ6DrrNmaVaUCgLipfVlzrPQ4tWOR9P0IXGNOx50jYCCdSJg==",
121
+ "license": "Apache-2.0"
122
+ },
123
+ "node_modules/bare-events": {
124
+ "version": "2.5.4",
125
+ "resolved": "https://registry.npmjs.org/bare-events/-/bare-events-2.5.4.tgz",
126
+ "integrity": "sha512-+gFfDkR8pj4/TrWCGUGWmJIkBwuxPS5F+a5yWjOHQt2hHvNZd5YLzadjmDUtFmMM4y429bnKLa8bYBMHcYdnQA==",
127
+ "license": "Apache-2.0",
128
+ "optional": true
129
+ },
130
+ "node_modules/bare-fs": {
131
+ "version": "4.0.2",
132
+ "resolved": "https://registry.npmjs.org/bare-fs/-/bare-fs-4.0.2.tgz",
133
+ "integrity": "sha512-S5mmkMesiduMqnz51Bfh0Et9EX0aTCJxhsI4bvzFFLs8Z1AV8RDHadfY5CyLwdoLHgXbNBEN1gQcbEtGwuvixw==",
134
+ "license": "Apache-2.0",
135
+ "optional": true,
136
+ "dependencies": {
137
+ "bare-events": "^2.5.4",
138
+ "bare-path": "^3.0.0",
139
+ "bare-stream": "^2.6.4"
140
+ },
141
+ "engines": {
142
+ "bare": ">=1.16.0"
143
+ },
144
+ "peerDependencies": {
145
+ "bare-buffer": "*"
146
+ },
147
+ "peerDependenciesMeta": {
148
+ "bare-buffer": {
149
+ "optional": true
150
+ }
151
+ }
152
+ },
153
+ "node_modules/bare-os": {
154
+ "version": "3.6.1",
155
+ "resolved": "https://registry.npmjs.org/bare-os/-/bare-os-3.6.1.tgz",
156
+ "integrity": "sha512-uaIjxokhFidJP+bmmvKSgiMzj2sV5GPHaZVAIktcxcpCyBFFWO+YlikVAdhmUo2vYFvFhOXIAlldqV29L8126g==",
157
+ "license": "Apache-2.0",
158
+ "optional": true,
159
+ "engines": {
160
+ "bare": ">=1.14.0"
161
+ }
162
+ },
163
+ "node_modules/bare-path": {
164
+ "version": "3.0.0",
165
+ "resolved": "https://registry.npmjs.org/bare-path/-/bare-path-3.0.0.tgz",
166
+ "integrity": "sha512-tyfW2cQcB5NN8Saijrhqn0Zh7AnFNsnczRcuWODH0eYAXBsJ5gVxAUuNr7tsHSC6IZ77cA0SitzT+s47kot8Mw==",
167
+ "license": "Apache-2.0",
168
+ "optional": true,
169
+ "dependencies": {
170
+ "bare-os": "^3.0.1"
171
+ }
172
+ },
173
+ "node_modules/bare-stream": {
174
+ "version": "2.6.5",
175
+ "resolved": "https://registry.npmjs.org/bare-stream/-/bare-stream-2.6.5.tgz",
176
+ "integrity": "sha512-jSmxKJNJmHySi6hC42zlZnq00rga4jjxcgNZjY9N5WlOe/iOoGRtdwGsHzQv2RlH2KOYMwGUXhf2zXd32BA9RA==",
177
+ "license": "Apache-2.0",
178
+ "optional": true,
179
+ "dependencies": {
180
+ "streamx": "^2.21.0"
181
+ },
182
+ "peerDependencies": {
183
+ "bare-buffer": "*",
184
+ "bare-events": "*"
185
+ },
186
+ "peerDependenciesMeta": {
187
+ "bare-buffer": {
188
+ "optional": true
189
+ },
190
+ "bare-events": {
191
+ "optional": true
192
+ }
193
+ }
194
+ },
195
+ "node_modules/base64-js": {
196
+ "version": "1.5.1",
197
+ "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
198
+ "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",
199
+ "funding": [
200
+ {
201
+ "type": "github",
202
+ "url": "https://github.com/sponsors/feross"
203
+ },
204
+ {
205
+ "type": "patreon",
206
+ "url": "https://www.patreon.com/feross"
207
+ },
208
+ {
209
+ "type": "consulting",
210
+ "url": "https://feross.org/support"
211
+ }
212
+ ],
213
+ "license": "MIT"
214
+ },
215
+ "node_modules/bl": {
216
+ "version": "4.1.0",
217
+ "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
218
+ "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",
219
+ "license": "MIT",
220
+ "dependencies": {
221
+ "buffer": "^5.5.0",
222
+ "inherits": "^2.0.4",
223
+ "readable-stream": "^3.4.0"
224
+ }
225
+ },
226
+ "node_modules/buffer": {
227
+ "version": "5.7.1",
228
+ "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",
229
+ "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==",
230
+ "funding": [
231
+ {
232
+ "type": "github",
233
+ "url": "https://github.com/sponsors/feross"
234
+ },
235
+ {
236
+ "type": "patreon",
237
+ "url": "https://www.patreon.com/feross"
238
+ },
239
+ {
240
+ "type": "consulting",
241
+ "url": "https://feross.org/support"
242
+ }
243
+ ],
244
+ "license": "MIT",
245
+ "dependencies": {
246
+ "base64-js": "^1.3.1",
247
+ "ieee754": "^1.1.13"
248
+ }
249
+ },
250
+ "node_modules/chownr": {
251
+ "version": "1.1.4",
252
+ "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
253
+ "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
254
+ "license": "ISC"
255
+ },
256
+ "node_modules/color": {
257
+ "version": "4.2.3",
258
+ "resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz",
259
+ "integrity": "sha512-1rXeuUUiGGrykh+CeBdu5Ie7OJwinCgQY0bc7GCRxy5xVHy+moaqkpL/jqQq0MtQOeYcrqEz4abc5f0KtU7W4A==",
260
+ "license": "MIT",
261
+ "dependencies": {
262
+ "color-convert": "^2.0.1",
263
+ "color-string": "^1.9.0"
264
+ },
265
+ "engines": {
266
+ "node": ">=12.5.0"
267
+ }
268
+ },
269
+ "node_modules/color-convert": {
270
+ "version": "2.0.1",
271
+ "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
272
+ "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==",
273
+ "license": "MIT",
274
+ "dependencies": {
275
+ "color-name": "~1.1.4"
276
+ },
277
+ "engines": {
278
+ "node": ">=7.0.0"
279
+ }
280
+ },
281
+ "node_modules/color-name": {
282
+ "version": "1.1.4",
283
+ "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz",
284
+ "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==",
285
+ "license": "MIT"
286
+ },
287
+ "node_modules/color-string": {
288
+ "version": "1.9.1",
289
+ "resolved": "https://registry.npmjs.org/color-string/-/color-string-1.9.1.tgz",
290
+ "integrity": "sha512-shrVawQFojnZv6xM40anx4CkoDP+fZsw/ZerEMsW/pyzsRbElpsL/DBVW7q3ExxwusdNXI3lXpuhEZkzs8p5Eg==",
291
+ "license": "MIT",
292
+ "dependencies": {
293
+ "color-name": "^1.0.0",
294
+ "simple-swizzle": "^0.2.2"
295
+ }
296
+ },
297
+ "node_modules/decompress-response": {
298
+ "version": "6.0.0",
299
+ "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz",
300
+ "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==",
301
+ "license": "MIT",
302
+ "dependencies": {
303
+ "mimic-response": "^3.1.0"
304
+ },
305
+ "engines": {
306
+ "node": ">=10"
307
+ },
308
+ "funding": {
309
+ "url": "https://github.com/sponsors/sindresorhus"
310
+ }
311
+ },
312
+ "node_modules/deep-extend": {
313
+ "version": "0.6.0",
314
+ "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz",
315
+ "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==",
316
+ "license": "MIT",
317
+ "engines": {
318
+ "node": ">=4.0.0"
319
+ }
320
+ },
321
+ "node_modules/detect-libc": {
322
+ "version": "2.0.3",
323
+ "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.0.3.tgz",
324
+ "integrity": "sha512-bwy0MGW55bG41VqxxypOsdSdGqLwXPI/focwgTYCFMbdUiBAxLg9CFzG08sz2aqzknwiX7Hkl0bQENjg8iLByw==",
325
+ "license": "Apache-2.0",
326
+ "engines": {
327
+ "node": ">=8"
328
+ }
329
+ },
330
+ "node_modules/end-of-stream": {
331
+ "version": "1.4.4",
332
+ "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.4.tgz",
333
+ "integrity": "sha512-+uw1inIHVPQoaVuHzRyXd21icM+cnt4CzD5rW+NC1wjOUSTOs+Te7FOv7AhN7vS9x/oIyhLP5PR1H+phQAHu5Q==",
334
+ "license": "MIT",
335
+ "dependencies": {
336
+ "once": "^1.4.0"
337
+ }
338
+ },
339
+ "node_modules/expand-template": {
340
+ "version": "2.0.3",
341
+ "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz",
342
+ "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==",
343
+ "license": "(MIT OR WTFPL)",
344
+ "engines": {
345
+ "node": ">=6"
346
+ }
347
+ },
348
+ "node_modules/fast-fifo": {
349
+ "version": "1.3.2",
350
+ "resolved": "https://registry.npmjs.org/fast-fifo/-/fast-fifo-1.3.2.tgz",
351
+ "integrity": "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==",
352
+ "license": "MIT"
353
+ },
354
+ "node_modules/flatbuffers": {
355
+ "version": "1.12.0",
356
+ "resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-1.12.0.tgz",
357
+ "integrity": "sha512-c7CZADjRcl6j0PlvFy0ZqXQ67qSEZfrVPynmnL+2zPc+NtMvrF8Y0QceMo7QqnSPc7+uWjUIAbvCQ5WIKlMVdQ==",
358
+ "license": "SEE LICENSE IN LICENSE.txt"
359
+ },
360
+ "node_modules/fs-constants": {
361
+ "version": "1.0.0",
362
+ "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
363
+ "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",
364
+ "license": "MIT"
365
+ },
366
+ "node_modules/github-from-package": {
367
+ "version": "0.0.0",
368
+ "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",
369
+ "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==",
370
+ "license": "MIT"
371
+ },
372
+ "node_modules/guid-typescript": {
373
+ "version": "1.0.9",
374
+ "resolved": "https://registry.npmjs.org/guid-typescript/-/guid-typescript-1.0.9.tgz",
375
+ "integrity": "sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==",
376
+ "license": "ISC"
377
+ },
378
+ "node_modules/ieee754": {
379
+ "version": "1.2.1",
380
+ "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
381
+ "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",
382
+ "funding": [
383
+ {
384
+ "type": "github",
385
+ "url": "https://github.com/sponsors/feross"
386
+ },
387
+ {
388
+ "type": "patreon",
389
+ "url": "https://www.patreon.com/feross"
390
+ },
391
+ {
392
+ "type": "consulting",
393
+ "url": "https://feross.org/support"
394
+ }
395
+ ],
396
+ "license": "BSD-3-Clause"
397
+ },
398
+ "node_modules/inherits": {
399
+ "version": "2.0.4",
400
+ "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
401
+ "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
402
+ "license": "ISC"
403
+ },
404
+ "node_modules/ini": {
405
+ "version": "1.3.8",
406
+ "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",
407
+ "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
408
+ "license": "ISC"
409
+ },
410
+ "node_modules/is-arrayish": {
411
+ "version": "0.3.2",
412
+ "resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.3.2.tgz",
413
+ "integrity": "sha512-eVRqCvVlZbuw3GrM63ovNSNAeA1K16kaR/LRY/92w0zxQ5/1YzwblUX652i4Xs9RwAGjW9d9y6X88t8OaAJfWQ==",
414
+ "license": "MIT"
415
+ },
416
+ "node_modules/long": {
417
+ "version": "4.0.0",
418
+ "resolved": "https://registry.npmjs.org/long/-/long-4.0.0.tgz",
419
+ "integrity": "sha512-XsP+KhQif4bjX1kbuSiySJFNAehNxgLb6hPRGJ9QsUr8ajHkuXGdrHmFUTUUXhDwVX2R5bY4JNZEwbUiMhV+MA==",
420
+ "license": "Apache-2.0"
421
+ },
422
+ "node_modules/mimic-response": {
423
+ "version": "3.1.0",
424
+ "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz",
425
+ "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==",
426
+ "license": "MIT",
427
+ "engines": {
428
+ "node": ">=10"
429
+ },
430
+ "funding": {
431
+ "url": "https://github.com/sponsors/sindresorhus"
432
+ }
433
+ },
434
+ "node_modules/minimist": {
435
+ "version": "1.2.8",
436
+ "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",
437
+ "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",
438
+ "license": "MIT",
439
+ "funding": {
440
+ "url": "https://github.com/sponsors/ljharb"
441
+ }
442
+ },
443
+ "node_modules/mkdirp-classic": {
444
+ "version": "0.5.3",
445
+ "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
446
+ "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",
447
+ "license": "MIT"
448
+ },
449
+ "node_modules/napi-build-utils": {
450
+ "version": "2.0.0",
451
+ "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz",
452
+ "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==",
453
+ "license": "MIT"
454
+ },
455
+ "node_modules/node-abi": {
456
+ "version": "3.74.0",
457
+ "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.74.0.tgz",
458
+ "integrity": "sha512-c5XK0MjkGBrQPGYG24GBADZud0NCbznxNx0ZkS+ebUTrmV1qTDxPxSL8zEAPURXSbLRWVexxmP4986BziahL5w==",
459
+ "license": "MIT",
460
+ "dependencies": {
461
+ "semver": "^7.3.5"
462
+ },
463
+ "engines": {
464
+ "node": ">=10"
465
+ }
466
+ },
467
+ "node_modules/node-addon-api": {
468
+ "version": "6.1.0",
469
+ "resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-6.1.0.tgz",
470
+ "integrity": "sha512-+eawOlIgy680F0kBzPUNFhMZGtJ1YmqM6l4+Crf4IkImjYrO/mqPwRMh352g23uIaQKFItcQ64I7KMaJxHgAVA==",
471
+ "license": "MIT"
472
+ },
473
+ "node_modules/once": {
474
+ "version": "1.4.0",
475
+ "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
476
+ "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
477
+ "license": "ISC",
478
+ "dependencies": {
479
+ "wrappy": "1"
480
+ }
481
+ },
482
+ "node_modules/onnx-proto": {
483
+ "version": "4.0.4",
484
+ "resolved": "https://registry.npmjs.org/onnx-proto/-/onnx-proto-4.0.4.tgz",
485
+ "integrity": "sha512-aldMOB3HRoo6q/phyB6QRQxSt895HNNw82BNyZ2CMh4bjeKv7g/c+VpAFtJuEMVfYLMbRx61hbuqnKceLeDcDA==",
486
+ "license": "MIT",
487
+ "dependencies": {
488
+ "protobufjs": "^6.8.8"
489
+ }
490
+ },
491
+ "node_modules/onnxruntime-common": {
492
+ "version": "1.14.0",
493
+ "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.14.0.tgz",
494
+ "integrity": "sha512-3LJpegM2iMNRX2wUmtYfeX/ytfOzNwAWKSq1HbRrKc9+uqG/FsEA0bbKZl1btQeZaXhC26l44NWpNUeXPII7Ew==",
495
+ "license": "MIT"
496
+ },
497
+ "node_modules/onnxruntime-node": {
498
+ "version": "1.14.0",
499
+ "resolved": "https://registry.npmjs.org/onnxruntime-node/-/onnxruntime-node-1.14.0.tgz",
500
+ "integrity": "sha512-5ba7TWomIV/9b6NH/1x/8QEeowsb+jBEvFzU6z0T4mNsFwdPqXeFUM7uxC6QeSRkEbWu3qEB0VMjrvzN/0S9+w==",
501
+ "license": "MIT",
502
+ "optional": true,
503
+ "os": [
504
+ "win32",
505
+ "darwin",
506
+ "linux"
507
+ ],
508
+ "dependencies": {
509
+ "onnxruntime-common": "~1.14.0"
510
+ }
511
+ },
512
+ "node_modules/onnxruntime-web": {
513
+ "version": "1.14.0",
514
+ "resolved": "https://registry.npmjs.org/onnxruntime-web/-/onnxruntime-web-1.14.0.tgz",
515
+ "integrity": "sha512-Kcqf43UMfW8mCydVGcX9OMXI2VN17c0p6XvR7IPSZzBf/6lteBzXHvcEVWDPmCKuGombl997HgLqj91F11DzXw==",
516
+ "license": "MIT",
517
+ "dependencies": {
518
+ "flatbuffers": "^1.12.0",
519
+ "guid-typescript": "^1.0.9",
520
+ "long": "^4.0.0",
521
+ "onnx-proto": "^4.0.4",
522
+ "onnxruntime-common": "~1.14.0",
523
+ "platform": "^1.3.6"
524
+ }
525
+ },
526
+ "node_modules/platform": {
527
+ "version": "1.3.6",
528
+ "resolved": "https://registry.npmjs.org/platform/-/platform-1.3.6.tgz",
529
+ "integrity": "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==",
530
+ "license": "MIT"
531
+ },
532
+ "node_modules/prebuild-install": {
533
+ "version": "7.1.3",
534
+ "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
535
+ "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==",
536
+ "license": "MIT",
537
+ "dependencies": {
538
+ "detect-libc": "^2.0.0",
539
+ "expand-template": "^2.0.3",
540
+ "github-from-package": "0.0.0",
541
+ "minimist": "^1.2.3",
542
+ "mkdirp-classic": "^0.5.3",
543
+ "napi-build-utils": "^2.0.0",
544
+ "node-abi": "^3.3.0",
545
+ "pump": "^3.0.0",
546
+ "rc": "^1.2.7",
547
+ "simple-get": "^4.0.0",
548
+ "tar-fs": "^2.0.0",
549
+ "tunnel-agent": "^0.6.0"
550
+ },
551
+ "bin": {
552
+ "prebuild-install": "bin.js"
553
+ },
554
+ "engines": {
555
+ "node": ">=10"
556
+ }
557
+ },
558
+ "node_modules/prebuild-install/node_modules/tar-fs": {
559
+ "version": "2.1.2",
560
+ "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.2.tgz",
561
+ "integrity": "sha512-EsaAXwxmx8UB7FRKqeozqEPop69DXcmYwTQwXvyAPF352HJsPdkVhvTaDPYqfNgruveJIJy3TA2l+2zj8LJIJA==",
562
+ "license": "MIT",
563
+ "dependencies": {
564
+ "chownr": "^1.1.1",
565
+ "mkdirp-classic": "^0.5.2",
566
+ "pump": "^3.0.0",
567
+ "tar-stream": "^2.1.4"
568
+ }
569
+ },
570
+ "node_modules/prebuild-install/node_modules/tar-stream": {
571
+ "version": "2.2.0",
572
+ "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",
573
+ "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",
574
+ "license": "MIT",
575
+ "dependencies": {
576
+ "bl": "^4.0.3",
577
+ "end-of-stream": "^1.4.1",
578
+ "fs-constants": "^1.0.0",
579
+ "inherits": "^2.0.3",
580
+ "readable-stream": "^3.1.1"
581
+ },
582
+ "engines": {
583
+ "node": ">=6"
584
+ }
585
+ },
586
+ "node_modules/protobufjs": {
587
+ "version": "6.11.4",
588
+ "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.4.tgz",
589
+ "integrity": "sha512-5kQWPaJHi1WoCpjTGszzQ32PG2F4+wRY6BmAT4Vfw56Q2FZ4YZzK20xUYQH4YkfehY1e6QSICrJquM6xXZNcrw==",
590
+ "hasInstallScript": true,
591
+ "license": "BSD-3-Clause",
592
+ "dependencies": {
593
+ "@protobufjs/aspromise": "^1.1.2",
594
+ "@protobufjs/base64": "^1.1.2",
595
+ "@protobufjs/codegen": "^2.0.4",
596
+ "@protobufjs/eventemitter": "^1.1.0",
597
+ "@protobufjs/fetch": "^1.1.0",
598
+ "@protobufjs/float": "^1.0.2",
599
+ "@protobufjs/inquire": "^1.1.0",
600
+ "@protobufjs/path": "^1.1.2",
601
+ "@protobufjs/pool": "^1.1.0",
602
+ "@protobufjs/utf8": "^1.1.0",
603
+ "@types/long": "^4.0.1",
604
+ "@types/node": ">=13.7.0",
605
+ "long": "^4.0.0"
606
+ },
607
+ "bin": {
608
+ "pbjs": "bin/pbjs",
609
+ "pbts": "bin/pbts"
610
+ }
611
+ },
612
+ "node_modules/pump": {
613
+ "version": "3.0.2",
614
+ "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.2.tgz",
615
+ "integrity": "sha512-tUPXtzlGM8FE3P0ZL6DVs/3P58k9nk8/jZeQCurTJylQA8qFYzHFfhBJkuqyE0FifOsQ0uKWekiZ5g8wtr28cw==",
616
+ "license": "MIT",
617
+ "dependencies": {
618
+ "end-of-stream": "^1.1.0",
619
+ "once": "^1.3.1"
620
+ }
621
+ },
622
+ "node_modules/rc": {
623
+ "version": "1.2.8",
624
+ "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz",
625
+ "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==",
626
+ "license": "(BSD-2-Clause OR MIT OR Apache-2.0)",
627
+ "dependencies": {
628
+ "deep-extend": "^0.6.0",
629
+ "ini": "~1.3.0",
630
+ "minimist": "^1.2.0",
631
+ "strip-json-comments": "~2.0.1"
632
+ },
633
+ "bin": {
634
+ "rc": "cli.js"
635
+ }
636
+ },
637
+ "node_modules/readable-stream": {
638
+ "version": "3.6.2",
639
+ "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
640
+ "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
641
+ "license": "MIT",
642
+ "dependencies": {
643
+ "inherits": "^2.0.3",
644
+ "string_decoder": "^1.1.1",
645
+ "util-deprecate": "^1.0.1"
646
+ },
647
+ "engines": {
648
+ "node": ">= 6"
649
+ }
650
+ },
651
+ "node_modules/safe-buffer": {
652
+ "version": "5.2.1",
653
+ "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
654
+ "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",
655
+ "funding": [
656
+ {
657
+ "type": "github",
658
+ "url": "https://github.com/sponsors/feross"
659
+ },
660
+ {
661
+ "type": "patreon",
662
+ "url": "https://www.patreon.com/feross"
663
+ },
664
+ {
665
+ "type": "consulting",
666
+ "url": "https://feross.org/support"
667
+ }
668
+ ],
669
+ "license": "MIT"
670
+ },
671
+ "node_modules/semver": {
672
+ "version": "7.7.1",
673
+ "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.1.tgz",
674
+ "integrity": "sha512-hlq8tAfn0m/61p4BVRcPzIGr6LKiMwo4VM6dGi6pt4qcRkmNzTcWq6eCEjEh+qXjkMDvPlOFFSGwQjoEa6gyMA==",
675
+ "license": "ISC",
676
+ "bin": {
677
+ "semver": "bin/semver.js"
678
+ },
679
+ "engines": {
680
+ "node": ">=10"
681
+ }
682
+ },
683
+ "node_modules/sharp": {
684
+ "version": "0.32.6",
685
+ "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.32.6.tgz",
686
+ "integrity": "sha512-KyLTWwgcR9Oe4d9HwCwNM2l7+J0dUQwn/yf7S0EnTtb0eVS4RxO0eUSvxPtzT4F3SY+C4K6fqdv/DO27sJ/v/w==",
687
+ "hasInstallScript": true,
688
+ "license": "Apache-2.0",
689
+ "dependencies": {
690
+ "color": "^4.2.3",
691
+ "detect-libc": "^2.0.2",
692
+ "node-addon-api": "^6.1.0",
693
+ "prebuild-install": "^7.1.1",
694
+ "semver": "^7.5.4",
695
+ "simple-get": "^4.0.1",
696
+ "tar-fs": "^3.0.4",
697
+ "tunnel-agent": "^0.6.0"
698
+ },
699
+ "engines": {
700
+ "node": ">=14.15.0"
701
+ },
702
+ "funding": {
703
+ "url": "https://opencollective.com/libvips"
704
+ }
705
+ },
706
+ "node_modules/simple-concat": {
707
+ "version": "1.0.1",
708
+ "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz",
709
+ "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==",
710
+ "funding": [
711
+ {
712
+ "type": "github",
713
+ "url": "https://github.com/sponsors/feross"
714
+ },
715
+ {
716
+ "type": "patreon",
717
+ "url": "https://www.patreon.com/feross"
718
+ },
719
+ {
720
+ "type": "consulting",
721
+ "url": "https://feross.org/support"
722
+ }
723
+ ],
724
+ "license": "MIT"
725
+ },
726
+ "node_modules/simple-get": {
727
+ "version": "4.0.1",
728
+ "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz",
729
+ "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==",
730
+ "funding": [
731
+ {
732
+ "type": "github",
733
+ "url": "https://github.com/sponsors/feross"
734
+ },
735
+ {
736
+ "type": "patreon",
737
+ "url": "https://www.patreon.com/feross"
738
+ },
739
+ {
740
+ "type": "consulting",
741
+ "url": "https://feross.org/support"
742
+ }
743
+ ],
744
+ "license": "MIT",
745
+ "dependencies": {
746
+ "decompress-response": "^6.0.0",
747
+ "once": "^1.3.1",
748
+ "simple-concat": "^1.0.0"
749
+ }
750
+ },
751
+ "node_modules/simple-swizzle": {
752
+ "version": "0.2.2",
753
+ "resolved": "https://registry.npmjs.org/simple-swizzle/-/simple-swizzle-0.2.2.tgz",
754
+ "integrity": "sha512-JA//kQgZtbuY83m+xT+tXJkmJncGMTFT+C+g2h2R9uxkYIrE2yy9sgmcLhCnw57/WSD+Eh3J97FPEDFnbXnDUg==",
755
+ "license": "MIT",
756
+ "dependencies": {
757
+ "is-arrayish": "^0.3.1"
758
+ }
759
+ },
760
+ "node_modules/streamx": {
761
+ "version": "2.22.0",
762
+ "resolved": "https://registry.npmjs.org/streamx/-/streamx-2.22.0.tgz",
763
+ "integrity": "sha512-sLh1evHOzBy/iWRiR6d1zRcLao4gGZr3C1kzNz4fopCOKJb6xD9ub8Mpi9Mr1R6id5o43S+d93fI48UC5uM9aw==",
764
+ "license": "MIT",
765
+ "dependencies": {
766
+ "fast-fifo": "^1.3.2",
767
+ "text-decoder": "^1.1.0"
768
+ },
769
+ "optionalDependencies": {
770
+ "bare-events": "^2.2.0"
771
+ }
772
+ },
773
+ "node_modules/string_decoder": {
774
+ "version": "1.3.0",
775
+ "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
776
+ "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
777
+ "license": "MIT",
778
+ "dependencies": {
779
+ "safe-buffer": "~5.2.0"
780
+ }
781
+ },
782
+ "node_modules/strip-json-comments": {
783
+ "version": "2.0.1",
784
+ "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",
785
+ "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==",
786
+ "license": "MIT",
787
+ "engines": {
788
+ "node": ">=0.10.0"
789
+ }
790
+ },
791
+ "node_modules/tar-fs": {
792
+ "version": "3.0.8",
793
+ "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-3.0.8.tgz",
794
+ "integrity": "sha512-ZoROL70jptorGAlgAYiLoBLItEKw/fUxg9BSYK/dF/GAGYFJOJJJMvjPAKDJraCXFwadD456FCuvLWgfhMsPwg==",
795
+ "license": "MIT",
796
+ "dependencies": {
797
+ "pump": "^3.0.0",
798
+ "tar-stream": "^3.1.5"
799
+ },
800
+ "optionalDependencies": {
801
+ "bare-fs": "^4.0.1",
802
+ "bare-path": "^3.0.0"
803
+ }
804
+ },
805
+ "node_modules/tar-stream": {
806
+ "version": "3.1.7",
807
+ "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-3.1.7.tgz",
808
+ "integrity": "sha512-qJj60CXt7IU1Ffyc3NJMjh6EkuCFej46zUqJ4J7pqYlThyd9bO0XBTmcOIhSzZJVWfsLks0+nle/j538YAW9RQ==",
809
+ "license": "MIT",
810
+ "dependencies": {
811
+ "b4a": "^1.6.4",
812
+ "fast-fifo": "^1.2.0",
813
+ "streamx": "^2.15.0"
814
+ }
815
+ },
816
+ "node_modules/text-decoder": {
817
+ "version": "1.2.3",
818
+ "resolved": "https://registry.npmjs.org/text-decoder/-/text-decoder-1.2.3.tgz",
819
+ "integrity": "sha512-3/o9z3X0X0fTupwsYvR03pJ/DjWuqqrfwBgTQzdWDiQSm9KitAyz/9WqsT2JQW7KV2m+bC2ol/zqpW37NHxLaA==",
820
+ "license": "Apache-2.0",
821
+ "dependencies": {
822
+ "b4a": "^1.6.4"
823
+ }
824
+ },
825
+ "node_modules/tunnel-agent": {
826
+ "version": "0.6.0",
827
+ "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
828
+ "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==",
829
+ "license": "Apache-2.0",
830
+ "dependencies": {
831
+ "safe-buffer": "^5.0.1"
832
+ },
833
+ "engines": {
834
+ "node": "*"
835
+ }
836
+ },
837
+ "node_modules/undici-types": {
838
+ "version": "6.20.0",
839
+ "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.20.0.tgz",
840
+ "integrity": "sha512-Ny6QZ2Nju20vw1SRHe3d9jVu6gJ+4e3+MMpqu7pqE5HT6WsTSlce++GQmK5UXS8mzV8DSYHrQH+Xrf2jVcuKNg==",
841
+ "license": "MIT"
842
+ },
843
+ "node_modules/util-deprecate": {
844
+ "version": "1.0.2",
845
+ "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
846
+ "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",
847
+ "license": "MIT"
848
+ },
849
+ "node_modules/wrappy": {
850
+ "version": "1.0.2",
851
+ "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
852
+ "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
853
+ "license": "ISC"
854
+ }
855
+ }
856
+ }
package.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "vietnamese-embedding-onnx",
3
+ "version": "1.0.0",
4
+ "main": "test_model.js",
5
+ "type": "module",
6
+ "scripts": {
7
+ "test": "echo \"Error: no test specified\" && exit 1"
8
+ },
9
+ "author": "",
10
+ "license": "ISC",
11
+ "description": "",
12
+ "dependencies": {
13
+ "@xenova/transformers": "^2.17.2"
14
+ }
15
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "<s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "mask_token": {
24
+ "content": "<mask>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "pad_token": {
31
+ "content": "<pad>",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "sep_token": {
38
+ "content": "</s>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ },
44
+ "unk_token": {
45
+ "content": "<unk>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false
50
+ }
51
+ }
test_model.js ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Force offline mode to prevent remote lookups.
2
+ process.env.HF_HUB_OFFLINE = "1";
3
+
4
+ import { AutoModel } from '@xenova/transformers';
5
+
6
+ async function run() {
7
+ // Since test_model.js is inside your model folder,
8
+ // use '.' to refer to the current folder.
9
+ const model = await AutoModel.from_pretrained('.', { localFilesOnly: true });
10
+ const output = await model('Mô hình thử nghiệm');
11
+ console.log(output);
12
+ }
13
+
14
+ run();
tokenizer_config.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<s>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<pad>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "</s>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "<unk>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "64000": {
36
+ "content": "<mask>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "bos_token": "<s>",
45
+ "clean_up_tokenization_spaces": true,
46
+ "cls_token": "<s>",
47
+ "eos_token": "</s>",
48
+ "extra_special_tokens": {},
49
+ "mask_token": "<mask>",
50
+ "model_max_length": 512,
51
+ "pad_token": "<pad>",
52
+ "sep_token": "</s>",
53
+ "tokenizer_class": "PhobertTokenizer",
54
+ "unk_token": "<unk>"
55
+ }
vocab.txt ADDED
The diff for this file is too large to render. See raw diff