Models
BPE
class tokenizers.models.BPE
( vocab = None merges = None cache_capacity = None dropout = None unk_token = None continuing_subword_prefix = None end_of_word_suffix = None fuse_unk = None byte_fallback = False ignore_merges = False )
Parameters
- vocab (
Dict[str, int], optional) — A dictionary of string keys and their ids{"am": 0,...} - merges (
List[Tuple[str, str]], optional) — A list of pairs of tokens (Tuple[str, str])[("a", "b"),...] - cache_capacity (
int, optional) — The number of words that the BPE cache can contain. The cache allows to speed-up the process by keeping the result of the merge operations for a number of words. - dropout (
float, optional) — A float between 0 and 1 that represents the BPE dropout to use. - unk_token (
str, optional) — The unknown token to be used by the model. - continuing_subword_prefix (
str, optional) — The prefix to attach to subword units that don’t represent a beginning of word. - end_of_word_suffix (
str, optional) — The suffix to attach to subword units that represent an end of word. - fuse_unk (
bool, optional) — Whether to fuse any subsequent unknown tokens into a single one - byte_fallback (
bool, optional) — Whether to use spm byte-fallback trick (defaults to False) - ignore_merges (
bool, optional) — Whether or not to match tokens with the vocab before using merges.
An implementation of the BPE (Byte-Pair Encoding) algorithm
from_file
( vocab merge **kwargs ) → BPE
Parameters
Returns
An instance of BPE loaded from these files
Instantiate a BPE model from the given files.
This method is roughly equivalent to doing:
vocab, merges = BPE.read_file(vocab_filename, merges_filename) bpe = BPE(vocab, merges)
If you don’t need to keep the vocab, merges values lying around,
this method is more optimized than manually calling
read_file() to initialize a BPE
Read a vocab.json and a merges.txt files
This method provides a way to read and parse the content of these files, returning the relevant data structures. If you want to instantiate some BPE models from memory, this method gives you the expected input from the standard files.
Model
Base class for all models
The model represents the actual tokenization algorithm. This is the part that will contain and manage the learned vocabulary.
This class cannot be constructed directly. Please use one of the concrete models.
Get the associated Trainer
Retrieve the Trainer associated to this
Model.
Save the current model
Save the current model in the given folder, using the given prefix for the various files that will get created. Any file with the same name that already exists in this folder will be overwritten.
Unigram
An implementation of the Unigram algorithm
WordLevel
An implementation of the WordLevel algorithm
Most simple tokenizer model based on mapping tokens to their corresponding id.
from_file
( vocab unk_token ) → WordLevel
Parameters
Returns
An instance of WordLevel loaded from file
Instantiate a WordLevel model from the given file
This method is roughly equivalent to doing:
vocab = WordLevel.read_file(vocab_filename) wordlevel = WordLevel(vocab)
If you don’t need to keep the vocab values lying around, this method is
more optimized than manually calling read_file() to
initialize a WordLevel
Read a vocab.json
This method provides a way to read and parse the content of a vocabulary file, returning the relevant data structures. If you want to instantiate some WordLevel models from memory, this method gives you the expected input from the standard files.
WordPiece
An implementation of the WordPiece algorithm
from_file
( vocab **kwargs ) → WordPiece
Parameters
Returns
An instance of WordPiece loaded from file
Instantiate a WordPiece model from the given file
This method is roughly equivalent to doing:
vocab = WordPiece.read_file(vocab_filename) wordpiece = WordPiece(vocab)
If you don’t need to keep the vocab values lying around, this method is
more optimized than manually calling read_file() to
initialize a WordPiece
Read a vocab.txt file
This method provides a way to read and parse the content of a standard vocab.txt file as used by the WordPiece Model, returning the relevant data structures. If you want to instantiate some WordPiece models from memory, this method gives you the expected input from the standard files.