protocol

tokenizer_language_protocol

Language provider protocol for tokenization and sentence splitting using canonical character-code lists.

Availability:
logtalk_load(tokenization(loader))
Author: Paulo Moura
Version: 1:0:0
Date: 2026-09-05
Compilation flags:
static
Dependencies:
(none)
Remarks:
(none)
Inherited public predicates:
(none)

Public predicates

tokenize_codes/3

Tokenizes a character-code list using validated, merged facade options.

Compilation flags:
static
Template:
tokenize_codes(Codes,Tokens,Options)
Mode and number of proofs:
tokenize_codes(+list(character_code),-list(list(character_code)),+list(compound)) - one

split_sentence_codes/3

Splits a character-code list into sentence character-code lists using validated, merged facade options.

Compilation flags:
static
Template:
split_sentence_codes(Codes,Sentences,Options)
Mode and number of proofs:
split_sentence_codes(+list(character_code),-list(list(character_code)),+list(compound)) - one

Protected predicates

(none)

Private predicates

(none)

Operators

(none)

See also

tokenizer_protocol, sentence_splitter_protocol, tokenizer(Representation,Language)