public class DdlTokenStream extends TokenStream
Because of the complexity of DDL, it was necessary to extend TokenStream
in order to override the basic tokenizer to
tokenize the in-line comments prefixed with "--". In addition, because there is not a default ddl command (or statement)
terminator, an override method was added to TokenStream
to allow re-tokenizing the initial tokens to re-type the
tokens, remove tokens, or any other operation to simplify parsing.
In this case, both reserved words (or key words) and statement start phrases can be registered prior to the TokenStream
's start() method. Any resulting tokens that match the registered string values will be re-typed to identify them as key words
(DdlTokenizer.KEYWORD) or statement start phrases (DdlTokenizer.STATEMENT_KEY).
Modifier and Type | Class and Description |
---|---|
static class |
DdlTokenStream.DdlTokenizer |
TokenStream.BasicTokenizer, TokenStream.CaseInsensitiveToken, TokenStream.CaseInsensitiveTokenFactory, TokenStream.CaseSensitiveToken, TokenStream.CaseSensitiveTokenFactory, TokenStream.CharacterArrayStream, TokenStream.CharacterStream, TokenStream.Token, TokenStream.TokenFactory, TokenStream.Tokenizer, TokenStream.Tokens
Modifier and Type | Field and Description |
---|---|
protected Set<String> |
registeredKeyWords |
protected List<String[]> |
registeredStatementStartPhrases |
ANY_TYPE, ANY_VALUE, inputString
Constructor and Description |
---|
DdlTokenStream(String content,
TokenStream.Tokenizer tokenizer,
boolean caseSensitive) |
Modifier and Type | Method and Description |
---|---|
int |
computeNextStatementStartKeywordCount()
Method to determine if next tokens match a registered statement start phrase.
|
static DdlTokenStream.DdlTokenizer |
ddlTokenizer(boolean includeComments)
Obtain a ddl
DdlTokenStream.DdlTokenizer implementation that ignores whitespace but includes tokens for individual symbols, the
period ('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments. |
String |
getMarkedContent()
Returns the string content for characters bounded by the previous marked position and the position of the currentToken
(inclusive).
|
protected List<TokenStream.Token> |
initializeTokens(List<TokenStream.Token> tokens)
Method to allow subclasses to preprocess the set of tokens and return the correct tokens to use.
|
protected boolean |
isKeyWord(String word) |
boolean |
isNextKeyWord()
Method to determine if the next token is of type
DdlTokenStream.DdlTokenizer KEYWORD. |
void |
mark()
Marks the current position (line & column number) of the currentToken
|
void |
registerKeyWord(String keyWord)
Register a single key word.
|
void |
registerKeyWords(List<String> keyWords)
Register an
List of key words. |
void |
registerKeyWords(String[] keyWords)
Register an array of key words.
|
void |
registerStatementStartPhrase(String[] phrase)
Register a phrase representing the start of a DDL statement
Examples would be: {"CREATE", "TABLE"} {"CREATE", "OR", "REPLACE", "VIEW"}
see
DdlConstants for the default SQL 92 representations. |
void |
registerStatementStartPhrase(String[][] phrases) |
basicTokenizer, canConsume, canConsume, canConsume, canConsume, canConsume, canConsume, canConsumeAnyOf, canConsumeAnyOf, canConsumeAnyOf, canConsumeAnyOf, canConsumeAnyOf, consume, consume, consume, consume, consume, consume, consume, consumeBoolean, consumeInteger, consumeLong, getContentBetween, hasNext, matches, matches, matches, matches, matches, matches, matches, matches, matchesAnyOf, matchesAnyOf, matchesAnyOf, matchesAnyOf, matchesAnyOf, nextPosition, previousPosition, rewind, start, throwNoMoreContent, toString
public DdlTokenStream(String content, TokenStream.Tokenizer tokenizer, boolean caseSensitive)
content
- tokenizer
- caseSensitive
- protected List<TokenStream.Token> initializeTokens(List<TokenStream.Token> tokens)
initializeTokens
in class TokenStream
TokenStream.initializeTokens(java.util.List)
public void registerStatementStartPhrase(String[] phrase)
Examples would be: {"CREATE", "TABLE"} {"CREATE", "OR", "REPLACE", "VIEW"}
seeDdlConstants
for the default SQL 92 representations.phrase
- public void registerStatementStartPhrase(String[][] phrases)
public void registerKeyWord(String keyWord)
keyWord
- public void registerKeyWords(List<String> keyWords)
List
of key words.keyWords
- public void registerKeyWords(String[] keyWords)
keyWords
- protected boolean isKeyWord(String word)
word
- public boolean isNextKeyWord()
DdlTokenStream.DdlTokenizer
KEYWORD.public int computeNextStatementStartKeywordCount()
public void mark()
public String getMarkedContent()
public static DdlTokenStream.DdlTokenizer ddlTokenizer(boolean includeComments)
DdlTokenStream.DdlTokenizer
implementation that ignores whitespace but includes tokens for individual symbols, the
period ('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments.
Note that the resulting Tokenizer may not be appropriate in many situations, but is provided merely as a convenience for those situations that happen to be able to use it.
includeComments
- true if the comments should be retained and be included in the token stream, or false if comments
should be stripped and not included in the token streamCopyright © 2008–2016 JBoss, a division of Red Hat. All rights reserved.