Upload tokenizer

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

special_tokens_map.json

@@ -0,0 +1,27 @@
+{
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,1159 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 256,
+      "content": "<bos>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 257,
+      "content": "<end_of_turn>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 258,
+      "content": "<eos>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 259,
+      "content": "<pad>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 260,
+      "content": "<start_of_turn>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 263,
+      "content": "<unk>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "ByteLevel",
+    "add_prefix_space": false,
+    "trim_offsets": true,
+    "use_regex": false
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {}
+  },
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "Unigram",
+    "unk_id": null,
+    "vocab": [
+      [
+        "Ā",
+        0.0
+      ],
+      [
+        "ā",
+        0.0
+      ],
+      [
+        "Ă",
+        0.0
+      ],
+      [
+        "ă",
+        0.0
+      ],
+      [
+        "Ą",
+        0.0
+      ],
+      [
+        "ą",
+        0.0
+      ],
+      [
+        "Ć",
+        0.0
+      ],
+      [
+        "ć",
+        0.0
+      ],
+      [
+        "Ĉ",
+        0.0
+      ],
+      [
+        "ĉ",
+        0.0
+      ],
+      [
+        "Ċ",
+        0.0
+      ],
+      [
+        "ċ",
+        0.0
+      ],
+      [
+        "Č",
+        0.0
+      ],
+      [
+        "č",
+        0.0
+      ],
+      [
+        "Ď",
+        0.0
+      ],
+      [
+        "ď",
+        0.0
+      ],
+      [
+        "Đ",
+        0.0
+      ],
+      [
+        "đ",
+        0.0
+      ],
+      [
+        "Ē",
+        0.0
+      ],
+      [
+        "ē",
+        0.0
+      ],
+      [
+        "Ĕ",
+        0.0
+      ],
+      [
+        "ĕ",
+        0.0
+      ],
+      [
+        "Ė",
+        0.0
+      ],
+      [
+        "ė",
+        0.0
+      ],
+      [
+        "Ę",
+        0.0
+      ],
+      [
+        "ę",
+        0.0
+      ],
+      [
+        "Ě",
+        0.0
+      ],
+      [
+        "ě",
+        0.0
+      ],
+      [
+        "Ĝ",
+        0.0
+      ],
+      [
+        "ĝ",
+        0.0
+      ],
+      [
+        "Ğ",
+        0.0
+      ],
+      [
+        "ğ",
+        0.0
+      ],
+      [
+        "Ġ",
+        0.0
+      ],
+      [
+        "!",
+        0.0
+      ],
+      [
+        "\"",
+        0.0
+      ],
+      [
+        "#",
+        0.0
+      ],
+      [
+        "$",
+        0.0
+      ],
+      [
+        "%",
+        0.0
+      ],
+      [
+        "&",
+        0.0
+      ],
+      [
+        "'",
+        0.0
+      ],
+      [
+        "(",
+        0.0
+      ],
+      [
+        ")",
+        0.0
+      ],
+      [
+        "*",
+        0.0
+      ],
+      [
+        "+",
+        0.0
+      ],
+      [
+        ",",
+        0.0
+      ],
+      [
+        "-",
+        0.0
+      ],
+      [
+        ".",
+        0.0
+      ],
+      [
+        "/",
+        0.0
+      ],
+      [
+        "0",
+        0.0
+      ],
+      [
+        "1",
+        0.0
+      ],
+      [
+        "2",
+        0.0
+      ],
+      [
+        "3",
+        0.0
+      ],
+      [
+        "4",
+        0.0
+      ],
+      [
+        "5",
+        0.0
+      ],
+      [
+        "6",
+        0.0
+      ],
+      [
+        "7",
+        0.0
+      ],
+      [
+        "8",
+        0.0
+      ],
+      [
+        "9",
+        0.0
+      ],
+      [
+        ":",
+        0.0
+      ],
+      [
+        ";",
+        0.0
+      ],
+      [
+        "<",
+        0.0
+      ],
+      [
+        "=",
+        0.0
+      ],
+      [
+        ">",
+        0.0
+      ],
+      [
+        "?",
+        0.0
+      ],
+      [
+        "@",
+        0.0
+      ],
+      [
+        "A",
+        0.0
+      ],
+      [
+        "B",
+        0.0
+      ],
+      [
+        "C",
+        0.0
+      ],
+      [
+        "D",
+        0.0
+      ],
+      [
+        "E",
+        0.0
+      ],
+      [
+        "F",
+        0.0
+      ],
+      [
+        "G",
+        0.0
+      ],
+      [
+        "H",
+        0.0
+      ],
+      [
+        "I",
+        0.0
+      ],
+      [
+        "J",
+        0.0
+      ],
+      [
+        "K",
+        0.0
+      ],
+      [
+        "L",
+        0.0
+      ],
+      [
+        "M",
+        0.0
+      ],
+      [
+        "N",
+        0.0
+      ],
+      [
+        "O",
+        0.0
+      ],
+      [
+        "P",
+        0.0
+      ],
+      [
+        "Q",
+        0.0
+      ],
+      [
+        "R",
+        0.0
+      ],
+      [
+        "S",
+        0.0
+      ],
+      [
+        "T",
+        0.0
+      ],
+      [
+        "U",
+        0.0
+      ],
+      [
+        "V",
+        0.0
+      ],
+      [
+        "W",
+        0.0
+      ],
+      [
+        "X",
+        0.0
+      ],
+      [
+        "Y",
+        0.0
+      ],
+      [
+        "Z",
+        0.0
+      ],
+      [
+        "[",
+        0.0
+      ],
+      [
+        "\\",
+        0.0
+      ],
+      [
+        "]",
+        0.0
+      ],
+      [
+        "^",
+        0.0
+      ],
+      [
+        "_",
+        0.0
+      ],
+      [
+        "`",
+        0.0
+      ],
+      [
+        "a",
+        0.0
+      ],
+      [
+        "b",
+        0.0
+      ],
+      [
+        "c",
+        0.0
+      ],
+      [
+        "d",
+        0.0
+      ],
+      [
+        "e",
+        0.0
+      ],
+      [
+        "f",
+        0.0
+      ],
+      [
+        "g",
+        0.0
+      ],
+      [
+        "h",
+        0.0
+      ],
+      [
+        "i",
+        0.0
+      ],
+      [
+        "j",
+        0.0
+      ],
+      [
+        "k",
+        0.0
+      ],
+      [
+        "l",
+        0.0
+      ],
+      [
+        "m",
+        0.0
+      ],
+      [
+        "n",
+        0.0
+      ],
+      [
+        "o",
+        0.0
+      ],
+      [
+        "p",
+        0.0
+      ],
+      [
+        "q",
+        0.0
+      ],
+      [
+        "r",
+        0.0
+      ],
+      [
+        "s",
+        0.0
+      ],
+      [
+        "t",
+        0.0
+      ],
+      [
+        "u",
+        0.0
+      ],
+      [
+        "v",
+        0.0
+      ],
+      [
+        "w",
+        0.0
+      ],
+      [
+        "x",
+        0.0
+      ],
+      [
+        "y",
+        0.0
+      ],
+      [
+        "z",
+        0.0
+      ],
+      [
+        "{",
+        0.0
+      ],
+      [
+        "|",
+        0.0
+      ],
+      [
+        "}",
+        0.0
+      ],
+      [
+        "~",
+        0.0
+      ],
+      [
+        "ġ",
+        0.0
+      ],
+      [
+        "Ģ",
+        0.0
+      ],
+      [
+        "ģ",
+        0.0
+      ],
+      [
+        "Ĥ",
+        0.0
+      ],
+      [
+        "ĥ",
+        0.0
+      ],
+      [
+        "Ħ",
+        0.0
+      ],
+      [
+        "ħ",
+        0.0
+      ],
+      [
+        "Ĩ",
+        0.0
+      ],
+      [
+        "ĩ",
+        0.0
+      ],
+      [
+        "Ī",
+        0.0
+      ],
+      [
+        "ī",
+        0.0
+      ],
+      [
+        "Ĭ",
+        0.0
+      ],
+      [
+        "ĭ",
+        0.0
+      ],
+      [
+        "Į",
+        0.0
+      ],
+      [
+        "į",
+        0.0
+      ],
+      [
+        "İ",
+        0.0
+      ],
+      [
+        "ı",
+        0.0
+      ],
+      [
+        "IJ",
+        0.0
+      ],
+      [
+        "ij",
+        0.0
+      ],
+      [
+        "Ĵ",
+        0.0
+      ],
+      [
+        "ĵ",
+        0.0
+      ],
+      [
+        "Ķ",
+        0.0
+      ],
+      [
+        "ķ",
+        0.0
+      ],
+      [
+        "ĸ",
+        0.0
+      ],
+      [
+        "Ĺ",
+        0.0
+      ],
+      [
+        "ĺ",
+        0.0
+      ],
+      [
+        "Ļ",
+        0.0
+      ],
+      [
+        "ļ",
+        0.0
+      ],
+      [
+        "Ľ",
+        0.0
+      ],
+      [
+        "ľ",
+        0.0
+      ],
+      [
+        "Ŀ",
+        0.0
+      ],
+      [
+        "ŀ",
+        0.0
+      ],
+      [
+        "Ł",
+        0.0
+      ],
+      [
+        "ł",
+        0.0
+      ],
+      [
+        "¡",
+        0.0
+      ],
+      [
+        "¢",
+        0.0
+      ],
+      [
+        "£",
+        0.0
+      ],
+      [
+        "¤",
+        0.0
+      ],
+      [
+        "¥",
+        0.0
+      ],
+      [
+        "¦",
+        0.0
+      ],
+      [
+        "§",
+        0.0
+      ],
+      [
+        "¨",
+        0.0
+      ],
+      [
+        "©",
+        0.0
+      ],
+      [
+        "ª",
+        0.0
+      ],
+      [
+        "«",
+        0.0
+      ],
+      [
+        "¬",
+        0.0
+      ],
+      [
+        "Ń",
+        0.0
+      ],
+      [
+        "®",
+        0.0
+      ],
+      [
+        "¯",
+        0.0
+      ],
+      [
+        "°",
+        0.0
+      ],
+      [
+        "±",
+        0.0
+      ],
+      [
+        "²",
+        0.0
+      ],
+      [
+        "³",
+        0.0
+      ],
+      [
+        "´",
+        0.0
+      ],
+      [
+        "µ",
+        0.0
+      ],
+      [
+        "¶",
+        0.0
+      ],
+      [
+        "·",
+        0.0
+      ],
+      [
+        "¸",
+        0.0
+      ],
+      [
+        "¹",
+        0.0
+      ],
+      [
+        "º",
+        0.0
+      ],
+      [
+        "»",
+        0.0
+      ],
+      [
+        "¼",
+        0.0
+      ],
+      [
+        "½",
+        0.0
+      ],
+      [
+        "¾",
+        0.0
+      ],
+      [
+        "¿",
+        0.0
+      ],
+      [
+        "À",
+        0.0
+      ],
+      [
+        "Á",
+        0.0
+      ],
+      [
+        "Â",
+        0.0
+      ],
+      [
+        "Ã",
+        0.0
+      ],
+      [
+        "Ä",
+        0.0
+      ],
+      [
+        "Å",
+        0.0
+      ],
+      [
+        "Æ",
+        0.0
+      ],
+      [
+        "Ç",
+        0.0
+      ],
+      [
+        "È",
+        0.0
+      ],
+      [
+        "É",
+        0.0
+      ],
+      [
+        "Ê",
+        0.0
+      ],
+      [
+        "Ë",
+        0.0
+      ],
+      [
+        "Ì",
+        0.0
+      ],
+      [
+        "Í",
+        0.0
+      ],
+      [
+        "Î",
+        0.0
+      ],
+      [
+        "Ï",
+        0.0
+      ],
+      [
+        "Ð",
+        0.0
+      ],
+      [
+        "Ñ",
+        0.0
+      ],
+      [
+        "Ò",
+        0.0
+      ],
+      [
+        "Ó",
+        0.0
+      ],
+      [
+        "Ô",
+        0.0
+      ],
+      [
+        "Õ",
+        0.0
+      ],
+      [
+        "Ö",
+        0.0
+      ],
+      [
+        "×",
+        0.0
+      ],
+      [
+        "Ø",
+        0.0
+      ],
+      [
+        "Ù",
+        0.0
+      ],
+      [
+        "Ú",
+        0.0
+      ],
+      [
+        "Û",
+        0.0
+      ],
+      [
+        "Ü",
+        0.0
+      ],
+      [
+        "Ý",
+        0.0
+      ],
+      [
+        "Þ",
+        0.0
+      ],
+      [
+        "ß",
+        0.0
+      ],
+      [
+        "à",
+        0.0
+      ],
+      [
+        "á",
+        0.0
+      ],
+      [
+        "â",
+        0.0
+      ],
+      [
+        "ã",
+        0.0
+      ],
+      [
+        "ä",
+        0.0
+      ],
+      [
+        "å",
+        0.0
+      ],
+      [
+        "æ",
+        0.0
+      ],
+      [
+        "ç",
+        0.0
+      ],
+      [
+        "è",
+        0.0
+      ],
+      [
+        "é",
+        0.0
+      ],
+      [
+        "ê",
+        0.0
+      ],
+      [
+        "ë",
+        0.0
+      ],
+      [
+        "ì",
+        0.0
+      ],
+      [
+        "í",
+        0.0
+      ],
+      [
+        "î",
+        0.0
+      ],
+      [
+        "ï",
+        0.0
+      ],
+      [
+        "ð",
+        0.0
+      ],
+      [
+        "ñ",
+        0.0
+      ],
+      [
+        "ò",
+        0.0
+      ],
+      [
+        "ó",
+        0.0
+      ],
+      [
+        "ô",
+        0.0
+      ],
+      [
+        "õ",
+        0.0
+      ],
+      [
+        "ö",
+        0.0
+      ],
+      [
+        "÷",
+        0.0
+      ],
+      [
+        "ø",
+        0.0
+      ],
+      [
+        "ù",
+        0.0
+      ],
+      [
+        "ú",
+        0.0
+      ],
+      [
+        "û",
+        0.0
+      ],
+      [
+        "ü",
+        0.0
+      ],
+      [
+        "ý",
+        0.0
+      ],
+      [
+        "þ",
+        0.0
+      ],
+      [
+        "ÿ",
+        0.0
+      ],
+      [
+        "<bos>",
+        0.0
+      ],
+      [
+        "<end_of_turn>",
+        0.0
+      ],
+      [
+        "<eos>",
+        0.0
+      ],
+      [
+        "<pad>",
+        0.0
+      ],
+      [
+        "<start_of_turn>",
+        0.0
+      ],
+      [
+        "model",
+        0.0
+      ],
+      [
+        "user",
+        0.0
+      ]
+    ],
+    "byte_fallback": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,70 @@
+{
+  "add_bos_token": null,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "256": {
+      "content": "<bos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "257": {
+      "content": "<end_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "258": {
+      "content": "<eos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "259": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "260": {
+      "content": "<start_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "263": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": null,
+  "chat_template": "{{ bos_token }}{% if messages[0]['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if (message['role'] == 'assistant') %}{% set role = 'model' %}{% else %}{% set role = message['role'] %}{% endif %}{{ '<start_of_turn>' + role + '\n' + message['content'] | trim + '<end_of_turn>\n' }}{% endfor %}{% if add_generation_prompt %}{{'<start_of_turn>model\n'}}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<eos>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}