.. index:: single: tokenizer_rules .. _tokenizer_rules/0:

.. rst-class:: right

category

tokenizer_rules

Reusable portable rules for tokenization and sentence splitting. Importing objects supply language-specific abbreviation and word punctuation hooks. URL and email address recognition delegates to the url library.

| Availability: | logtalk_load(tokenization(loader))

| Author: Paulo Moura | Version: 1:0:0 | Date: 2026-09-05

| Compilation flags: | static

| Implements: | public :ref:`tokenizer_language_protocol <tokenizer_language_protocol/0>` | Extends: | public :ref:`options <options/0>` | Uses: | :ref:`list <list/0>` | :ref:`number_grammars(Format) <number_grammars/1>` | :ref:`string(Representation) <string/1>` | :ref:`url(Representation) <url/1>`

| Remarks: | (none)

| Inherited public predicates: | Â :ref:`options_protocol/0::check_option/1` Â :ref:`options_protocol/0::check_options/1` Â :ref:`options_protocol/0::default_option/1` Â :ref:`options_protocol/0::default_options/1` Â :ref:`options_protocol/0::option/2` Â :ref:`options_protocol/0::option/3` Â :ref:`tokenizer_language_protocol/0::split_sentence_codes/3` Â :ref:`tokenizer_language_protocol/0::tokenize_codes/3` Â :ref:`options_protocol/0::valid_option/1` Â :ref:`options_protocol/0::valid_options/1` Â

.. contents:: :local: :backlinks: top

Public predicates

(no local declarations; see entity ancestors if any)

Protected predicates

.. index:: abbreviation/1 .. _tokenizer_rules/0::abbreviation/1:

abbreviation/1 ^^^^^^^^^^^^^^^^^^

Enumerates canonical lowercase abbreviations, including their periods.

| Compilation flags: | static

| Template: | abbreviation(Abbreviation) | Mode and number of proofs: | abbreviation(?atom) - zero_or_more

---

.. index:: non_terminal_abbreviation/1 .. _tokenizer_rules/0::non_terminal_abbreviation/1:

non_terminal_abbreviation/1 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Enumerates abbreviations that do not terminate a sentence when followed by another token.

| Compilation flags: | static

| Template: | non_terminal_abbreviation(Abbreviation) | Mode and number of proofs: | non_terminal_abbreviation(?atom) - zero_or_more

---

.. index:: internal_apostrophe/1 .. _tokenizer_rules/0::internal_apostrophe/1:

internal_apostrophe/1 ^^^^^^^^^^^^^^^^^^^^^^^^^

Enumerates apostrophe character codes accepted inside words.

| Compilation flags: | static

| Template: | internal_apostrophe(Code) | Mode and number of proofs: | internal_apostrophe(?character_code) - zero_or_more

---

.. index:: internal_hyphen/1 .. _tokenizer_rules/0::internal_hyphen/1:

internal_hyphen/1 ^^^^^^^^^^^^^^^^^^^^^

Enumerates hyphen character codes accepted inside words.

| Compilation flags: | static

| Template: | internal_hyphen(Code) | Mode and number of proofs: | internal_hyphen(?character_code) - zero_or_more

---

Private predicates

(no local declarations; see entity ancestors if any)

Operators

(none)

.. seealso::

:ref:`tokenizer_language_protocol <tokenizer_language_protocol/0>`, :ref:`english_tokenizer <english_tokenizer/0>`, :ref:`url(Representation) <url/1>`