org.modeshape.common.text

Interface TokenStream.Tokenizer

All Known Implementing Classes:

BasicSqlQueryParser.SqlTokenizer, CndTokenizer, DdlTokenStream.DdlTokenizer, FullTextSearchParser.TermTokenizer, TokenStream.BasicTokenizer, XPathParser.XPathTokenizer

Enclosing class:

TokenStream

public static interface TokenStream.Tokenizer

Interface for a Tokenizer component responsible for processing the characters in a TokenStream.CharacterStream and constructing the appropriate TokenStream.Token objects.

Method Summary

Modifier and Type	Method and Description
void	tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens) Process the supplied characters and construct the appropriate TokenStream.Token objects.

Method Detail

tokenize

void tokenize(TokenStream.CharacterStream input,
              TokenStream.Tokens tokens)
       throws ParsingException

Process the supplied characters and construct the appropriate TokenStream.Token objects.

Parameters:

input - the character input stream; never null

tokens - the factory for TokenStream.Token objects, which records the order in which the tokens are created

Throws:

ParsingException - if there is an error while processing the character stream (e.g., a quote is not closed, etc.)