org.jboss.dna.common.text
Interface TokenStream.Tokenizer

All Known Implementing Classes:

TokenStream.BasicTokenizer

Enclosing class:

TokenStream

public static interface TokenStream.Tokenizer

Interface for a Tokenizer component responsible for processing the characters in a TokenStream.CharacterStream and constructing the appropriate Token objects.

Method Summary

void	tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens) Process the supplied characters and construct the appropriate Token objects.

Method Detail

tokenize

void tokenize(TokenStream.CharacterStream input,
              TokenStream.Tokens tokens)
              throws TokenStream.ParsingException

Process the supplied characters and construct the appropriate Token objects.

Parameters:

input - the character input stream; never null

tokens - the factory for Token objects, which records the order in which the tokens are created

Throws:

TokenStream.ParsingException - if there is an error while processing the character stream (e.g., a quote is not closed, etc.)