apply_filters( 'relevanssi_premium_tokenizer', string $token, string $context )
This filter hook allows Relevanssi Premium to add extra tokenization to enable stemmers.
Parameters
$token
(string) An indexing token, a single word in post content or a search query.
$context
(string) The context for tokenization, either “indexing” or “search_query”.
More information
In tokenization, Relevanssi splits the post content or the search query into individual words, tokens. The tokens are then cleaned from unnecessary punctuation, converted into lowercase and then compiled into an array.
This filter hook gives Relevanssi Premium a way to latch stemming (and other token manipulation) in the process. Relevanssi passes each token that is not a part of a phrase through this filter hook.
The default behaviour in Relevanssi Premium is to add the function relevanssi_enable_stemmer() to this hook, and then that function triggers the relevanssi_stemmer hook that the simple English stemmer and the Snowball Stemmer use.