NAME

Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer

SYNOPSIS

        # isa Plucene::Analysis::CharTokenizer

DESCRIPTION

This is the standard tokenizer.

This should be a good tokenizer for most European-language documents.

METHODS