Tokenizers documentation
Visualizer
You are viewing main version, which requires installation from source. If you'd like
regular pip install, checkout the latest stable version (v0.20.3).
Visualizer
Python
Rust
Node
Annotation
EncodingVisualizer
class tokenizers.tools.EncodingVisualizer
< source >( tokenizer: Tokenizer default_to_notebook: bool = True annotation_converter: typing.Optional[typing.Callable[[typing.Any], tokenizers.tools.visualizer.Annotation]] = None )
Parameters
- tokenizer (Tokenizer) — A tokenizer instance
- default_to_notebook (
bool) — Whether to render html output in a notebook by default - annotation_converter (
Callable, optional) — An optional (lambda) function that takes an annotation in any format and returns an Annotation object
Build an EncodingVisualizer
__call__
< source >( text: str annotations: typing.List[tokenizers.tools.visualizer.Annotation] = [] default_to_notebook: typing.Optional[bool] = None )
Parameters
- text (
str) — The text to tokenize - annotations (
List[Annotation], optional) — An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function - default_to_notebook (
bool, optional, defaults to False) — If True, will render the html in a notebook. Otherwise returns an html string.
Build a visualization of the given text