public class CndTokenizer extends Object implements TokenStream.Tokenizer
TokenStream.Tokenizer
implementation that adheres to the CND format by ignoring whitespace while including tokens for individual
symbols, the period ('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments.
This tokenizer optionally includes comments and vendor extensions.Modifier and Type | Field and Description |
---|---|
static int |
COMMENT
The token type for tokens that consist of all the characters between "/*" and "*/" or between "//" and the next line
terminator (e.g., '\n', '\r' or "\r\n").
|
static int |
DECIMAL
The token type for tokens that consist of an individual '.' character.
|
static int |
DOUBLE_QUOTED_STRING
The token type for tokens that consist of all the characters within double-quotes.
|
static int |
SINGLE_QUOTED_STRING
The token type for tokens that consist of all the characters within single-quotes.
|
static int |
SYMBOL
The token type for tokens that consist of an individual "symbol" character.
|
static int |
VENDOR_EXTENSION
The token type for the token containing a vendor extension block.
|
static int |
WORD
The token type for tokens that represent an unquoted string containing a character sequence made up of non-whitespace and
non-symbol characters.
|
Constructor and Description |
---|
CndTokenizer(boolean useComments,
boolean useVendorExtensions) |
Modifier and Type | Method and Description |
---|---|
void |
tokenize(TokenStream.CharacterStream input,
TokenStream.Tokens tokens)
Process the supplied characters and construct the appropriate
TokenStream.Token objects. |
public static final int WORD
public static final int SYMBOL
[]<>=-+(),
public static final int DECIMAL
public static final int SINGLE_QUOTED_STRING
public static final int DOUBLE_QUOTED_STRING
public static final int COMMENT
public static final int VENDOR_EXTENSION
public CndTokenizer(boolean useComments, boolean useVendorExtensions)
public void tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens) throws ParsingException
TokenStream.Tokenizer
TokenStream.Token
objects.tokenize
in interface TokenStream.Tokenizer
input
- the character input stream; never nulltokens
- the factory for TokenStream.Token
objects, which records the order in which the tokens are createdParsingException
- if there is an error while processing the character stream (e.g., a quote is not closed, etc.)Copyright © 2008–2016 JBoss, a division of Red Hat. All rights reserved.