whitphx HF Staff commited on
Commit
a6ed142
·
verified ·
1 Parent(s): cae6888

Add/update the quantized ONNX model files and README.md for Transformers.js v3

Browse files
Files changed (1) hide show
  1. README.md +18 -2
README.md CHANGED
@@ -8,7 +8,23 @@ license: mit
8
 
9
  # Claude Tokenizer
10
 
11
- A 🤗-compatible version of the **Claude tokenizer** (adapted from [anthropics/anthropic-sdk-python](https://github.com/anthropics/anthropic-sdk-python)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
  ## Example usage:
14
 
@@ -22,7 +38,7 @@ assert tokenizer.encode('hello world') == [9381, 2253]
22
 
23
  ### Transformers.js
24
  ```js
25
- import { AutoTokenizer } from '@xenova/transformers';
26
 
27
  const tokenizer = await AutoTokenizer.from_pretrained('Xenova/claude-tokenizer');
28
  const tokens = tokenizer.encode('hello world'); // [9381, 2253]
 
8
 
9
  # Claude Tokenizer
10
 
11
+ A 🤗-compatible version of the **Claude tokenizer** (adapted from [anthropics/anthropic-sdk-python](https://github.com/anthropics/anthropic-sdk-python)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/huggingface/transformers.js).
12
+
13
+ ## Usage (Transformers.js)
14
+
15
+ If you haven't already, you can install the [Transformers.js](https://huggingface.co/docs/transformers.js) JavaScript library from [NPM](https://www.npmjs.com/package/@huggingface/transformers) using:
16
+ ```bash
17
+ npm i @huggingface/transformers
18
+ ```
19
+
20
+ **Example:** Tokenize text using Claude tokenizer
21
+
22
+ ```js
23
+ import { AutoTokenizer } from '@huggingface/transformers';
24
+
25
+ const tokenizer = await AutoTokenizer.from_pretrained('Xenova/claude-tokenizer');
26
+ const tokens = tokenizer.encode('hello world'); // [9381, 2253]
27
+ ```
28
 
29
  ## Example usage:
30
 
 
38
 
39
  ### Transformers.js
40
  ```js
41
+ import { AutoTokenizer } from '@huggingface/transformers';
42
 
43
  const tokenizer = await AutoTokenizer.from_pretrained('Xenova/claude-tokenizer');
44
  const tokens = tokenizer.encode('hello world'); // [9381, 2253]