Upload tokenizer

README.md

@@ -1,5 +1,4 @@
 ---
----
 license: other
 license_name: trillion
 license_link: LICENSE
@@ -12,49 +11,97 @@ language:
 - ja
 pipeline_tag: text-generation
 library_name: transformers
-extra_gated_prompt: >-
-  **TRILLION LABS AI MODEL LICENSE AGREEMENT**
-  Tri- Model Series Version Effective Date: February 1, 2025
+extra_gated_prompt: '**TRILLION LABS AI MODEL LICENSE AGREEMENT** Tri- Model Series
+  Version Effective Date: February 1, 2025
 
-  "**Agreement**" means the terms and conditions for use, reproduction, distribution and modification of the Trillion Labs AI Model series set forth herein.
+  "**Agreement**" means the terms and conditions for use, reproduction, distribution
+  and modification of the Trillion Labs AI Model series set forth herein.
 
-  "**Documentation**" means the specifications, manuals and documentation accompanying the Tri- Model series distributed by Trillion Labs.
+  "**Documentation**" means the specifications, manuals and documentation accompanying
+  the Tri- Model series distributed by Trillion Labs.
 
-  "**Licensee**" or "**you**" means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity's behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf.
+  "**Licensee**" or "**you**" means you, or your employer or any other person or entity
+  (if you are entering into this Agreement on such person or entity''s behalf), of
+  the age required under applicable laws, rules or regulations to provide legal consent
+  and that has legal authority to bind your employer or such other person or entity
+  if you are entering in this Agreement on their behalf.
 
-  "**Model**" means the artificial intelligence model series provided by Licensor ("Tri-" series), including software, algorithms, machine learning models, and related components provided by Licensor, including all updates, enhancements, improvements, bug fixes, patches, or other modifications.
+  "**Model**" means the artificial intelligence model series provided by Licensor
+  ("Tri-" series), including software, algorithms, machine learning models, and related
+  components provided by Licensor, including all updates, enhancements, improvements,
+  bug fixes, patches, or other modifications.
 
-  "**Trillion Labs**" or "**we**" means Trillion Labs, the owner, developer, and provider of the Model, holding all rights, title, and interest in the Model.
+  "**Trillion Labs**" or "**we**" means Trillion Labs, the owner, developer, and provider
+  of the Model, holding all rights, title, and interest in the Model.
 
-  By clicking "I Accept" below or by using or distributing any portion or element of the Model, you agree to be bound by this Agreement.
+  By clicking "I Accept" below or by using or distributing any portion or element
+  of the Model, you agree to be bound by this Agreement.
 
   1\. **License Grant and Redistribution**.
 
-  a. Grant of Rights. You are granted a limited, non-exclusive, non-transferable, worldwide, revocable license under Trillion Labs' intellectual property or other rights to use, reproduce, distribute, and make modifications to the Model for research purposes.
+  a. Grant of Rights. You are granted a limited, non-exclusive, non-transferable,
+  worldwide, revocable license under Trillion Labs'' intellectual property or other
+  rights to use, reproduce, distribute, and make modifications to the Model for research
+  purposes.
 
   b. Redistribution and Use.
 
-  i. If you distribute or make available the Model (or any derivative works thereof), or a product or service that contains any of them, you shall (A) provide a copy of this Agreement with any such Model; and (B) prominently display "Built with Tri-" on a related website, user interface, blogpost, about page, or product documentation. If you use the Model to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include "Tri-" followed by the original Model version at the beginning of any such AI model name.
-
-  ii. You must retain in all copies of the Model that you distribute the following attribution notice within a "Notice" text file distributed as a part of such copies: "Tri- Model Series is licensed under the Trillion Labs AI Model License Agreement, Copyright © Trillion Labs. All Rights Reserved."
-
-  iii. Your use of the Model must comply with applicable laws and regulations (including trade compliance laws and regulations).
-
-  2\. **Additional Commercial Terms**. If the monthly active users of the products or services made available by or for Licensee, or Licensee's affiliates, is greater than 1 million monthly active users OR Annual Recurring Revenue is greater than $10 million USD, you must request a commercial license from Trillion Labs, and you are not authorized to exercise any commercial rights under this Agreement unless or until Trillion Labs otherwise expressly grants you such rights.
-
-  3\. **Disclaimer of Warranty**. THE MODEL, DERIVATIVES, AND OUTPUT ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, AND TRILLION LABS DISCLAIMS ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
-
-  4\. **Limitation of Liability**. IN NO EVENT WILL TRILLION LABS BE LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS OR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES.
+  i. If you distribute or make available the Model (or any derivative works thereof),
+  or a product or service that contains any of them, you shall (A) provide a copy
+  of this Agreement with any such Model; and (B) prominently display "Built with Tri-"
+  on a related website, user interface, blogpost, about page, or product documentation.
+  If you use the Model to create, train, fine tune, or otherwise improve an AI model,
+  which is distributed or made available, you shall also include "Tri-" followed by
+  the original Model version at the beginning of any such AI model name.
+
+  ii. You must retain in all copies of the Model that you distribute the following
+  attribution notice within a "Notice" text file distributed as a part of such copies:
+  "Tri- Model Series is licensed under the Trillion Labs AI Model License Agreement,
+  Copyright © Trillion Labs. All Rights Reserved."
+
+  iii. Your use of the Model must comply with applicable laws and regulations (including
+  trade compliance laws and regulations).
+
+  2\. **Additional Commercial Terms**. If the monthly active users of the products
+  or services made available by or for Licensee, or Licensee''s affiliates, is greater
+  than 1 million monthly active users OR Annual Recurring Revenue is greater than
+  $10 million USD, you must request a commercial license from Trillion Labs, and you
+  are not authorized to exercise any commercial rights under this Agreement unless
+  or until Trillion Labs otherwise expressly grants you such rights.
+
+  3\. **Disclaimer of Warranty**. THE MODEL, DERIVATIVES, AND OUTPUT ARE PROVIDED
+  ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, AND TRILLION LABS DISCLAIMS
+  ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION,
+  ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR
+  PURPOSE.
+
+  4\. **Limitation of Liability**. IN NO EVENT WILL TRILLION LABS BE LIABLE UNDER
+  ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY,
+  OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS OR ANY INDIRECT,
+  SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES.
 
   5\. **Intellectual Property**.
 
-  a. No trademark licenses are granted under this Agreement, and in connection with the Model, neither Trillion Labs nor Licensee may use any name or mark owned by or associated with the other or any of its affiliates, except as required for reasonable and customary use in describing and redistributing the Model or as set forth in this Section 5(a).
-
-  b. All rights, title, and interest in the Model, including modifications, Derivatives, and documentation, remain exclusively with Trillion Labs.
-
-  6\. **Term and Termination**. The term of this Agreement will commence upon your acceptance of this Agreement or access to the Model and will continue in full force and effect until terminated in accordance with the terms and conditions herein. Trillion Labs may terminate this Agreement if you are in breach of any term or condition of this Agreement. Upon termination of this Agreement, you shall delete and cease use of the Model. Sections 3, 4 and 5 shall survive the termination of this Agreement.
-
-  7\. **Governing Law and Jurisdiction**. This Agreement will be governed and construed under the laws of the State of California without regard to choice of law principles. The courts of California shall have exclusive jurisdiction of any dispute arising out of this Agreement.
+  a. No trademark licenses are granted under this Agreement, and in connection with
+  the Model, neither Trillion Labs nor Licensee may use any name or mark owned by
+  or associated with the other or any of its affiliates, except as required for reasonable
+  and customary use in describing and redistributing the Model or as set forth in
+  this Section 5(a).
+
+  b. All rights, title, and interest in the Model, including modifications, Derivatives,
+  and documentation, remain exclusively with Trillion Labs.
+
+  6\. **Term and Termination**. The term of this Agreement will commence upon your
+  acceptance of this Agreement or access to the Model and will continue in full force
+  and effect until terminated in accordance with the terms and conditions herein.
+  Trillion Labs may terminate this Agreement if you are in breach of any term or condition
+  of this Agreement. Upon termination of this Agreement, you shall delete and cease
+  use of the Model. Sections 3, 4 and 5 shall survive the termination of this Agreement.
+
+  7\. **Governing Law and Jurisdiction**. This Agreement will be governed and construed
+  under the laws of the State of California without regard to choice of law principles.
+  The courts of California shall have exclusive jurisdiction of any dispute arising
+  out of this Agreement.'
 extra_gated_fields:
   First Name: text
   Last Name: text
@@ -71,12 +118,18 @@ extra_gated_fields:
     - Reporter
     - Other
   geo: ip_location
-  By clicking Submit below I accept the terms of the license and acknowledge that the information I provide will be collected stored processed and shared in accordance with the Trillion Labs Privacy Policy: checkbox
-extra_gated_description: >-
-  The information you provide will be collected, stored, processed and shared in
-  accordance with the Trillion Labs Privacy Policy.
+  ? By clicking Submit below I accept the terms of the license and acknowledge that
+    the information I provide will be collected stored processed and shared in accordance
+    with the Trillion Labs Privacy Policy
+  : checkbox
+extra_gated_description: The information you provide will be collected, stored, processed
+  and shared in accordance with the Trillion Labs Privacy Policy.
 extra_gated_button_content: Submit
-extra_gated_heading: "Please be sure to provide your full legal name, date of birth, and full organization name with all corporate identifiers. Avoid the use of acronyms and special characters. Failure to follow these instructions may prevent you from accessing this model and others on Hugging Face. You will not have the ability to edit this form after submission, so please ensure all information is accurate."
+extra_gated_heading: Please be sure to provide your full legal name, date of birth,
+  and full organization name with all corporate identifiers. Avoid the use of acronyms
+  and special characters. Failure to follow these instructions may prevent you from
+  accessing this model and others on Hugging Face. You will not have the ability to
+  edit this form after submission, so please ensure all information is accurate.
 ---
 
 

chat_template.jinja

@@ -16,7 +16,7 @@
     {%- if messages[0]['role'] == 'system' %}
         {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
     {%- else %}
-        {{- '<|im_start|>system\nYou are Trillion, created by TrillionLabs. You are a helpful assistant.<|im_end|>\n' }}
+        {{- '<|im_start|>system\nYou are Trillion, created by TrillionLabs. You a   re a helpful assistant.<|im_end|>\n' }}
     {%- endif %}
 {%- endif %}
 {%- for message in messages %}
@@ -40,7 +40,7 @@
         {{- '<|im_end|>\n' }}
     {%- elif message.role == "tool" %}
         {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
-            {{- '<|im_start|>tool\n' }}
+            {{- '<|im_start|>tool' + '\n' }}
         {%- endif %}
         {% for tool_response in message.content %}
             {{- tool_response | tojson }}