Text Functions¶
Text analysis and natural-language utility functions.
Summary¶
| Function | Signature | Description |
|---|---|---|
bigrams |
string -> array |
Generate word bigrams (2-grams) |
char_count |
string -> number |
Count characters in text |
char_frequencies |
string -> object |
Count character frequencies |
collapse_whitespace |
string -> string |
Normalize whitespace (multiple spaces to single, trim) |
is_stopword |
string, string? -> boolean |
Check if word is a stopword |
ngrams |
string, number, string? -> array |
Generate n-grams from text (word or character) |
normalize_unicode |
string, string? -> string |
Unicode normalization (NFC, NFD, NFKC, NFKD) |
paragraph_count |
string -> number |
Count paragraphs in text |
reading_time |
string -> string |
Estimate reading time |
reading_time_seconds |
string -> number |
Estimate reading time in seconds |
remove_accents |
string -> string |
Strip diacritics/accents from text |
remove_stopwords |
array, string? -> array |
Remove stopwords from token array |
sentence_count |
string -> number |
Count sentences in text |
stem |
string, string? -> string |
Stem a word using Snowball stemmer (Porter algorithm) |
stems |
array, string? -> array |
Stem an array of tokens |
stopwords |
string? -> array |
Get stopwords list for a language (default: English) |
tokenize |
string, object? -> array |
Configurable tokenization with options for case and punctuation handling |
tokens |
string -> array |
Simple word tokenization with normalization (lowercase, strip punctuation) |
trigrams |
string -> array |
Generate word trigrams (3-grams) |
word_count |
string -> number |
Count words in text |
word_frequencies |
string -> object |
Count word frequencies |
Functions¶
bigrams¶
Generate word bigrams (2-grams)
Signature: string -> array
Examples:
# Basic bigrams
bigrams('a b c') -> \[\['a', 'b'\], \['b', 'c'\]\]
# Sentence bigrams
bigrams('the quick brown fox') -> \[\['the', 'quick'\], \['quick', 'brown'\], \['brown', 'fox'\]\]
# Single word
bigrams('single') -> \[\]
char_count¶
Count characters in text
Signature: string -> number
Examples:
# Simple word
char_count('hello') -> 5
# With space
char_count('hello world') -> 11
# Empty string
char_count('') -> 0
char_frequencies¶
Count character frequencies
Signature: string -> object
Examples:
# Count repeated chars
char_frequencies('aab') -> {a: 2, b: 1}
# Word frequencies
char_frequencies('hello') -> {e: 1, h: 1, l: 2, o: 1}
# Empty string
char_frequencies('') -> {}
collapse_whitespace¶
Normalize whitespace (multiple spaces to single, trim)
Signature: string -> string
Examples:
# Collapse spaces
collapse_whitespace(' hello world ') -> 'hello world'
# Collapse tabs and newlines
collapse_whitespace('hello\t\nworld') -> 'hello world'
is_stopword¶
Check if word is a stopword
Signature: string, string? -> boolean
Examples:
# Common stopword
is_stopword('the') -> true
# Not a stopword
is_stopword('elephant') -> false
# Spanish stopword
is_stopword('el', 'es') -> true
ngrams¶
Generate n-grams from text (word or character)
Signature: string, number, string? -> array
Examples:
# Character trigrams
ngrams('hello', `3`, 'char') -> \['hel', 'ell', 'llo'\]
# Word bigrams
ngrams('a b c d', `2`, 'word') -> \[\['a', 'b'\], \['b', 'c'\], \['c', 'd'\]\]
# Text shorter than n
ngrams('ab', `3`, 'char') -> \[\]
normalize_unicode¶
Unicode normalization (NFC, NFD, NFKC, NFKD)
Signature: string, string? -> string
Examples:
# NFC normalization (default)
normalize_unicode('café') -> 'café'
# Compatibility decomposition
normalize_unicode('fi', 'NFKC') -> 'fi'
paragraph_count¶
Count paragraphs in text
Signature: string -> number
Examples:
# Two paragraphs
paragraph_count('A\\n\\nB') -> 2
# Single paragraph
paragraph_count('Single paragraph') -> 1
# Three paragraphs
paragraph_count('A\\n\\nB\\n\\nC') -> 3
reading_time¶
Estimate reading time
Signature: string -> string
Examples:
# Short text
reading_time('The quick brown fox') -> \"1 min read\"
# Empty text minimum
reading_time('') -> \"1 min read\"
reading_time_seconds¶
Estimate reading time in seconds
Signature: string -> number
Examples:
# Short sentence
reading_time_seconds('The quick brown fox jumps over the lazy dog') -> 2
# Empty text
reading_time_seconds('') -> 0
remove_accents¶
Strip diacritics/accents from text
Signature: string -> string
Examples:
# Remove accent
remove_accents('café') -> 'cafe'
# Multiple accents
remove_accents('naïve résumé') -> 'naive resume'
remove_stopwords¶
Remove stopwords from token array
Signature: array, string? -> array
Examples:
# Remove English stopwords
remove_stopwords(\['the', 'quick', 'fox'\]) -> \['quick', 'fox'\]
# Pipeline
tokens('The quick brown fox') | remove_stopwords(@) -> \['quick', 'brown', 'fox'\]
sentence_count¶
Count sentences in text
Signature: string -> number
Examples:
# Two sentences
sentence_count('Hello. World!') -> 2
# Single sentence
sentence_count('One sentence') -> 1
# Different terminators
sentence_count('What? Yes! No.') -> 3
stem¶
Stem a word using Snowball stemmer (Porter algorithm)
Signature: string, string? -> string
Examples:
# Basic stemming
stem('running') -> 'run'
# Plural stemming
stem('cats') -> 'cat'
# Adverb stemming
stem('quickly') -> 'quick'
# German stemming
stem('laufen', 'de') -> 'lauf'
stems¶
Stem an array of tokens
Signature: array, string? -> array
Examples:
# Stem multiple words
stems(\['running', 'cats'\]) -> \['run', 'cat'\]
# Pipeline with tokens
tokens('The cats are running') | stems(@) -> \['the', 'cat', 'are', 'run'\]
stopwords¶
Get stopwords list for a language (default: English)
Signature: string? -> array
Examples:
# English has many stopwords
stopwords() | length(@) > `100` -> true
# Spanish stopwords
stopwords('es') | contains(@, 'el') -> true
tokenize¶
Configurable tokenization with options for case and punctuation handling
Signature: string, object? -> array
Examples:
# Default (lowercase, strip punctuation)
tokenize('Hello, World!') -> \['hello', 'world'\]
# Preserve case
tokenize('Hello, World!', `{"case": "preserve"}`) -> \['Hello', 'World'\]
# Uppercase
tokenize('Hello, World!', `{"case": "upper"}`) -> \['HELLO', 'WORLD'\]
# Keep punctuation
tokenize('Hello, World!', `{"punctuation": "keep"}`) -> \['hello,', 'world!'\]
tokens¶
Simple word tokenization with normalization (lowercase, strip punctuation)
Signature: string -> array
Examples:
# Basic tokenization
tokens('Hello, World!') -> \['hello', 'world'\]
# Strip punctuation
tokens('The quick, brown fox!') -> \['the', 'quick', 'brown', 'fox'\]
# Punctuation-only returns empty
tokens('... --- !!!') -> \[\]
trigrams¶
Generate word trigrams (3-grams)
Signature: string -> array
Examples:
# Basic trigrams
trigrams('a b c d') -> \[\['a', 'b', 'c'\], \['b', 'c', 'd'\]\]
# Sentence trigrams
trigrams('the quick brown fox jumps') -> \[\['the', 'quick', 'brown'\], \['quick', 'brown', 'fox'\], \['brown', 'fox', 'jumps'\]\]
# Too few words
trigrams('a b') -> \[\]
word_count¶
Count words in text
Signature: string -> number
Examples:
# Two words
word_count('hello world') -> 2
# Single word
word_count('one') -> 1
# Empty string
word_count('') -> 0
word_frequencies¶
Count word frequencies
Signature: string -> object
Examples: