object

tokenizer(Representation,Language)

  • Representation - Text representation. Valid values are atom, codes, and chars.

  • Language - Object implementing the tokenizer_language_protocol protocol.

Tokenizer and sentence splitter parameterized by text representation and language provider.

Availability:
logtalk_load(tokenization(loader))
Author: Paulo Moura
Version: 1:0:0
Date: 2026-09-08
Compilation flags:
static, context_switching_calls
Implements:
public tokenizer_protocol
public sentence_splitter_protocol
Imports:
public options
Uses:
list
type
Remarks:
(none)
Inherited public predicates:
 check_option/1  check_options/1  default_option/1  default_options/1  option/2  option/3  split_sentences/2  split_sentences/3  tokenize/2  tokenize/3  tokenize_sentences/2  tokenize_sentences/3  valid_option/1  valid_options/1

Public predicates

(no local declarations; see entity ancestors if any)

Protected predicates

(no local declarations; see entity ancestors if any)

Private predicates

(no local declarations; see entity ancestors if any)

Operators

(none)

See also

tokenizer_protocol, sentence_splitter_protocol, tokenizer_language_protocol