| Plucene::Analysis::Standard::StandardTokenizer(3pm) | User Contributed Perl Documentation | Plucene::Analysis::Standard::StandardTokenizer(3pm) |
Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer
# isa Plucene::Analysis::CharTokenizer
This is the standard tokenizer.
This should be a good tokenizer for most European-language documents.
The regular expression for tokenising.
Remove 's and .
| 2022-12-04 | perl v5.36.0 |