org.modeshape.common.text
Interface TokenStream.Tokenizer

All Known Implementing Classes:
CndTokenizer, DdlTokenStream.DdlTokenizer, FullTextSearchParser.TermTokenizer, SqlQueryParser.SqlTokenizer, TokenStream.BasicTokenizer, XPathParser.XPathTokenizer
Enclosing class:
TokenStream

public static interface TokenStream.Tokenizer

Interface for a Tokenizer component responsible for processing the characters in a TokenStream.CharacterStream and constructing the appropriate TokenStream.Token objects.


Method Summary
 void tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens)
          Process the supplied characters and construct the appropriate TokenStream.Token objects.
 

Method Detail

tokenize

void tokenize(TokenStream.CharacterStream input,
              TokenStream.Tokens tokens)
              throws ParsingException
Process the supplied characters and construct the appropriate TokenStream.Token objects.

Parameters:
input - the character input stream; never null
tokens - the factory for TokenStream.Token objects, which records the order in which the tokens are created
Throws:
ParsingException - if there is an error while processing the character stream (e.g., a quote is not closed, etc.)


Copyright © 2008-2010 JBoss, a division of Red Hat. All Rights Reserved.