.. index:: single: tokenizer(Representation,Language) .. _tokenizer/2:

.. rst-class:: right

object

tokenizer(Representation,Language)

• Representation - Text representation. Valid values are atom, codes, and chars.
• Language - Object implementing the tokenizer_language_protocol protocol. Tokenizer and sentence splitter parameterized by text representation and language provider.

  | Availability: | logtalk_load(tokenization(loader))

  | Author: Paulo Moura | Version: 1:0:0 | Date: 2026-09-08

  | Compilation flags: | static, context_switching_calls

  | Implements: | public :ref:`tokenizer_protocol <tokenizer_protocol/0>` | public :ref:`sentence_splitter_protocol <sentence_splitter_protocol/0>` | Imports: | public :ref:`options <options/0>` | Uses: | :ref:`list <list/0>` | :ref:`type <type/0>`

  | Remarks: | (none)

  | Inherited public predicates: | Â :ref:`options_protocol/0::check_option/1` Â :ref:`options_protocol/0::check_options/1` Â :ref:`options_protocol/0::default_option/1` Â :ref:`options_protocol/0::default_options/1` Â :ref:`options_protocol/0::option/2` Â :ref:`options_protocol/0::option/3` Â :ref:`sentence_splitter_protocol/0::split_sentences/2` Â :ref:`sentence_splitter_protocol/0::split_sentences/3` Â :ref:`tokenizer_protocol/0::tokenize/2` Â :ref:`tokenizer_protocol/0::tokenize/3` Â :ref:`tokenizer_protocol/0::tokenize_sentences/2` Â :ref:`tokenizer_protocol/0::tokenize_sentences/3` Â :ref:`options_protocol/0::valid_option/1` Â :ref:`options_protocol/0::valid_options/1` Â

  .. contents:: :local: :backlinks: top

Public predicates

(no local declarations; see entity ancestors if any)

Protected predicates

(no local declarations; see entity ancestors if any)

Private predicates

(no local declarations; see entity ancestors if any)

Operators

(none)

.. seealso::

:ref:`tokenizer_protocol <tokenizer_protocol/0>`, :ref:`sentence_splitter_protocol <sentence_splitter_protocol/0>`, :ref:`tokenizer_language_protocol <tokenizer_language_protocol/0>`