| Did you know ... | Search Documentation: |
 | Pack logtalk -- logtalk-3.102.0/docs/apis/_sources/tokenizer_protocol_0.rst.txt |
- HOME
- DOWNLOAD
- DOCUMENTATION
- TUTORIALS
- Beginner▶
- Advanced▶
- Web applications▶
- Semantic web▶
- Graphics▶
- Machine learning▶
- External collections▶
- For packagers▶
- COMMUNITY
- COMMERCIAL
- WIKI
.. index:: single: tokenizer_protocol .. _tokenizer_protocol/0:
.. rst-class:: right
protocol
tokenizer_protocol
Tokenization protocol for text represented as atoms, lists of characters, or lists of character codes.
| Availability:
| logtalk_load(tokenization(loader))
| Author: Paulo Moura | Version: 1:0:0 | Date: 2026-09-08
| Compilation flags:
| static
| Dependencies: | (none)
| Remarks: | (none)
| Inherited public predicates: | (none)
.. contents:: :local: :backlinks: top
Public predicates
.. index:: tokenize/2 .. _tokenizer_protocol/0::tokenize/2:
tokenize/2 ^^^^^^^^^^^^^^
Tokenizes text using the default options.
| Compilation flags:
| static
| Template:
| tokenize(Text,Tokens)
| Mode and number of proofs:
| tokenize(+text,-list(text)) - one_or_error
| Exceptions:
| The Representation parameter is a variable:
| instantiation_error
| The Representation parameter is neither a variable nor atom, chars, or codes:
| domain_error(text_representation,Representation)
| The Language parameter is a variable:
| instantiation_error
| The Language parameter neither a variable nor an object identifier:
| type_error(object_identifier,Language)
| The Language parameter is an object identifier but not an object conforming to tokenizer_language_protocol:
| domain_error(tokenizer_language_protocol,Language)
| Text is not ground:
| instantiation_error
| The Representation parameter is atom but Text is not an atom:
| type_error(atom,Text)
| The Representation parameter is chars but Text is not a list of characters:
| type_error(chars,Text)
| The Representation parameter is codes but Text is not a list of character codes:
| type_error(codes,Text)
.. index:: tokenize/3 .. _tokenizer_protocol/0::tokenize/3:
tokenize/3 ^^^^^^^^^^^^^^
Tokenizes text using the given options. Recognized options are keep_punctuation(Boolean), lowercase(Boolean), normalize_quotes(Boolean), and normalize_dashes(Boolean).
| Compilation flags:
| static
| Template:
| tokenize(Text,Tokens,Options)
| Mode and number of proofs:
| tokenize(+text,-list(text),+list(compound)) - one_or_error
| Exceptions:
| The Representation parameter is a variable:
| instantiation_error
| The Representation parameter is neither a variable nor atom, chars, or codes:
| domain_error(text_representation,Representation)
| The Language parameter is a variable:
| instantiation_error
| The Language parameter neither a variable nor an object identifier:
| type_error(object_identifier,Language)
| The Language parameter is an object identifier but not an object conforming to tokenizer_language_protocol:
| domain_error(tokenizer_language_protocol,Language)
| Text is not ground:
| instantiation_error
| The Representation parameter is atom but Text is not an atom:
| type_error(atom,Text)
| The Representation parameter is chars but Text is not a list of characters:
| type_error(chars,Text)
| The Representation parameter is codes but Text is not a list of character codes:
| type_error(codes,Text)
| Options is a variable:
| instantiation_error
| Options is neither a variable nor a list:
| type_error(list,Options)
| An element Option of the list Options is a variable:
| instantiation_error
| An element Option of the list Options is neither a variable nor a compound term:
| type_error(compound,Option)
| An element Option of the list Options is a compound term but not a valid option:
| domain_error(option,Option)
.. index:: tokenize_sentences/2 .. _tokenizer_protocol/0::tokenize_sentences/2:
tokenize_sentences/2 ^^^^^^^^^^^^^^^^^^^^^^^^
Splits text into sentences and tokenizes each sentence using the default options.
| Compilation flags:
| static
| Template:
| tokenize_sentences(Text,TokenizedSentences)
| Mode and number of proofs:
| tokenize_sentences(+text,-list(list(text))) - one_or_error
| Exceptions:
| The Representation parameter is a variable:
| instantiation_error
| The Representation parameter is neither a variable nor atom, chars, or codes:
| domain_error(text_representation,Representation)
| The Language parameter is a variable:
| instantiation_error
| The Language parameter neither a variable nor an object identifier:
| type_error(object_identifier,Language)
| The Language parameter is an object identifier but not an object conforming to tokenizer_language_protocol:
| domain_error(tokenizer_language_protocol,Language)
| Text is not ground:
| instantiation_error
| The Representation parameter is atom but Text is not an atom:
| type_error(atom,Text)
| The Representation parameter is chars but Text is not a list of characters:
| type_error(chars,Text)
| The Representation parameter is codes but Text is not a list of character codes:
| type_error(codes,Text)
.. index:: tokenize_sentences/3 .. _tokenizer_protocol/0::tokenize_sentences/3:
tokenize_sentences/3 ^^^^^^^^^^^^^^^^^^^^^^^^
Splits text into sentences and tokenizes each sentence using the given options.
| Compilation flags:
| static
| Template:
| tokenize_sentences(Text,TokenizedSentences,Options)
| Mode and number of proofs:
| tokenize_sentences(+text,-list(list(text)),+list(compound)) - one_or_error
| Exceptions:
| The Representation parameter is a variable:
| instantiation_error
| The Representation parameter is neither a variable nor atom, chars, or codes:
| domain_error(text_representation,Representation)
| The Language parameter is a variable:
| instantiation_error
| The Language parameter neither a variable nor an object identifier:
| type_error(object_identifier,Language)
| The Language parameter is an object identifier but not an object conforming to tokenizer_language_protocol:
| domain_error(tokenizer_language_protocol,Language)
| Text is not ground:
| instantiation_error
| The Representation parameter is atom but Text is not an atom:
| type_error(atom,Text)
| The Representation parameter is chars but Text is not a list of characters:
| type_error(chars,Text)
| The Representation parameter is codes but Text is not a list of character codes:
| type_error(codes,Text)
| Options is a variable:
| instantiation_error
| Options is neither a variable nor a list:
| type_error(list,Options)
| An element Option of the list Options is a variable:
| instantiation_error
| An element Option of the list Options is neither a variable nor a compound term:
| type_error(compound,Option)
| An element Option of the list Options is a compound term but not a valid option:
| domain_error(option,Option)
Protected predicates
(none)
Private predicates
(none)
Operators
(none)
.. seealso::
:ref:`sentence_splitter_protocol <sentence_splitter_protocol/0>`, :ref:`tokenizer_language_protocol <tokenizer_language_protocol/0>`, :ref:`tokenizer(Representation,Language) <tokenizer/2>`