Xenova HF Staff commited on
Commit
9e41188
·
verified ·
1 Parent(s): f7fee9a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +26 -0
README.md CHANGED
@@ -1,3 +1,29 @@
1
  ---
 
 
 
 
2
  license: mit
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ library_name: transformers
3
+ tags:
4
+ - transformers.js
5
+ - tokenizers
6
  license: mit
7
  ---
8
+
9
+ # Claude Tokenizer
10
+
11
+ A 🤗-compatible version of the **Claude tokenizer** (adapted from [anthropics/anthropic-sdk-python](https://github.com/anthropics/anthropic-sdk-python)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
12
+
13
+ ## Example usage:
14
+
15
+ ### Transformers/Tokenizers
16
+ ```py
17
+ from transformers import GPT2TokenizerFast
18
+
19
+ tokenizer = GPT2TokenizerFast.from_pretrained('Xenova/claude-tokenizer')
20
+ assert tokenizer.encode('hello world') == [9381, 2253]
21
+ ```
22
+
23
+ ### Transformers.js
24
+ ```js
25
+ import { AutoTokenizer } from '@xenova/transformers';
26
+
27
+ const tokenizer = await AutoTokenizer.from_pretrained('Xenova/claude-tokenizer');
28
+ const tokens = tokenizer.encode('hello world'); // [9381, 2253]
29
+ ```