org.modeshape.graph.query.parse
Class FullTextSearchParser.TermTokenizer

java.lang.Object
  extended by org.modeshape.graph.query.parse.FullTextSearchParser.TermTokenizer
All Implemented Interfaces:
TokenStream.Tokenizer
Enclosing class:
FullTextSearchParser

public static class FullTextSearchParser.TermTokenizer
extends Object
implements TokenStream.Tokenizer

A basic TokenStream.Tokenizer implementation that ignores whitespace but includes tokens for individual symbols, the period ('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments.

Note this Tokenizer may not be appropriate in many situations, but is provided merely as a convenience for those situations that happen to be able to use it.


Field Summary
static int DOUBLE_QUOTED_STRING
          The token type for tokens that consist of all the characters within double-quotes.
static int PLUS_MINUS
          The token type for tokens that consist of an individual '+' or '-' characters.
static int SINGLE_QUOTED_STRING
          The token type for tokens that consist of all the characters within single-quotes.
static int WORD
          The token type for tokens that represent an unquoted string containing a character sequence made up of non-whitespace and non-symbol characters.
 
Constructor Summary
protected FullTextSearchParser.TermTokenizer()
           
 
Method Summary
 void tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens)
          Process the supplied characters and construct the appropriate TokenStream.Token objects.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

WORD

public static final int WORD
The token type for tokens that represent an unquoted string containing a character sequence made up of non-whitespace and non-symbol characters.

See Also:
Constant Field Values

PLUS_MINUS

public static final int PLUS_MINUS
The token type for tokens that consist of an individual '+' or '-' characters. The set of characters includes: -+

See Also:
Constant Field Values

SINGLE_QUOTED_STRING

public static final int SINGLE_QUOTED_STRING
The token type for tokens that consist of all the characters within single-quotes. Single quote characters are included if they are preceded (escaped) by a '\' character.

See Also:
Constant Field Values

DOUBLE_QUOTED_STRING

public static final int DOUBLE_QUOTED_STRING
The token type for tokens that consist of all the characters within double-quotes. Double quote characters are included if they are preceded (escaped) by a '\' character.

See Also:
Constant Field Values
Constructor Detail

FullTextSearchParser.TermTokenizer

protected FullTextSearchParser.TermTokenizer()
Method Detail

tokenize

public void tokenize(TokenStream.CharacterStream input,
                     TokenStream.Tokens tokens)
              throws ParsingException
Process the supplied characters and construct the appropriate TokenStream.Token objects.

Specified by:
tokenize in interface TokenStream.Tokenizer
Parameters:
input - the character input stream; never null
tokens - the factory for TokenStream.Token objects, which records the order in which the tokens are created
Throws:
ParsingException - if there is an error while processing the character stream (e.g., a quote is not closed, etc.)
See Also:
org.modeshape.common.text.TokenStream.Tokenizer#tokenize(CharacterStream, Tokens)


Copyright © 2008-2010 JBoss, a division of Red Hat. All Rights Reserved.