object

english_tokenizer

English tokenization and sentence splitting provider using portable rule-based heuristics and a curated abbreviation inventory.

Availability:
logtalk_load(tokenization(loader))
Author: Paulo Moura
Version: 1:0:0
Date: 2026-09-05
Compilation flags:
static, context_switching_calls
Implements:
public tokenizer_language_protocol
Imports:
public tokenizer_rules
Remarks:
(none)
Inherited public predicates:
 check_option/1  check_options/1  default_option/1  default_options/1  option/2  option/3  split_sentence_codes/3  tokenize_codes/3  valid_option/1  valid_options/1

Public predicates

(no local declarations; see entity ancestors if any)

Protected predicates

(no local declarations; see entity ancestors if any)

Private predicates

(no local declarations; see entity ancestors if any)

Operators

(none)

See also

tokenizer(Representation,Language), tokenizer_rules