JBoss.orgCommunity Documentation
The Teiid Connector Architecture (TCA) provides Teiid with a robust mechanism for integrating with external systems. The TCA defines a common client interface between Teiid and an external system that includes metadata as to what SQL constructs are supported for pushdown and the ability to import metadata from the external system.
A Translator is the heart of the TCA and acts as the bridge logic between Teiid and an external system, which is most commonly accessed through a JCA resource adapter. Refer to the Teiid Developers Guide for details on developing custom Translators and JCA resource adapters for use with Teiid.
The TCA is not the same as the JCA, the JavaEE Connector Architecture, although the TCA is designed for use with JCA resource adapters.
The import capabilities of Teiid Translators is currently only used in dynamic VDBs and not by the Teiid Designer.
A Translator is typically paired with a particular JCA resource adapter. In instances where pooling, environment dependent configuration management, advanced security handling, etc. are not needed, then a JCA resource adapter is not needed. The configuration of JCA ConnectionFactories for needed resource adapters is not part of this guide, please see the Teiid Administrator Guide and the kit examples for configuring resource adapters for use in JBossAS.
Translators can have a number of configurable properties. These are broken down into execution properties, which determine aspects of how data is retrieved, and import settings, which determine what metadata is read for import.
The execution properties for a translator typically have reasonable defaults. For specific translator types, e.g. the Derby translator, base execution properties are already tuned to match the source. In most cases the user will not need to adjust their values.
Table 9.1. Base Execution Properties - shared by all translators
Name | Description | Default |
---|---|---|
Immutable | Set to true to indicate that the source never changes. | false |
RequiresCriteria | Set to true to indicate that source SELECT/UPDATE/DELETE queries require a where clause. | false |
SupportsOrderBy | Set to true to indicate that the ORDER BY clause is supported. | false |
SupportsOuterJoins | Set to true to indicate that OUTER JOINs are supported. | false |
SupportsFullOuterJoins | If outer joins are supported, true indicates that FULL OUTER JOINs are supported. | false |
SupportsInnerJoins | Set to true to indicate that INNER JOINs are supported. | false |
SupportedJoinCriteria | If joins are supported, defines what criteria may be used as the join criteria. May be one of (ANY, THETA, EQUI, or KEY). | ANY |
Only a subset of the metadata as to what SQL constructs the source supports can be set through execution properties. If more control is needed, please consult the Teiid Developers Guide.
There are no base importer settings.
The file translator, known by the type name file, exposes stored procedures to leverage file system resources exposed by the file resource adapter. It will commonly be used with the TEXTTABLE or XMLTABLE table functions to use CSV or XML formated data.
Table 9.2. Execution Properties
Name | Description | Default |
---|---|---|
Encoding | The encoding that should be used for CLOBs returned by the getTextFiles procedure | The system default encoding |
There are file importer settings, but it does provide metadata for dynamic vdbs.
Retrieve all files as BLOBs with the given extension at the given path.
call getFiles('path/*.ext')
If the extension pattern is not specified and the path is a directory, then all files in the directory will be returned. If the path or filename doesn't exist, then no results will be returned.
Retrieve all files as CLOBs with the given extension at the given path.
call getTextFiles('path/*.ext')
Save the CLOB, BLOB, or XML file to given path
call saveFile('path', value)
See the database metadata for full descriptions of the getFiles, getTextFiles, and saveFile procedures.
The JDBC translator bridges between SQL semantic and data type difference between Teiid and a target RDBMS. Teiid has a range of specific translators that target the most popular open source and proprietary databases.
Type names:
jdbc-ansi - declares support for most SQL constructs supported by Teiid, except for row limit/offset and EXCEPT/INTERCECT. Translates source SQL into ANSI compliant syntax. This translator should be used when another more specific type is not available.
jdbc-simple - same as jdbc-ansi, except disables support for function, UNION, and aggregate pushdown.
db2 - for use with DB2 8 or later.
derby - for use with Derby 10.1 or later.
h2 - for use with H2 version 1.1 or later.
hsql - for use with HSQLDB 1.7 or later.
informix - for use with any version.
metamatrix - for use with MetaMatrix 5.5.0 or later.
mysql/mysql5 - for use with MySQL version 4.x and 5 or later respectively.
The MySQL Translators expect the database or session to be using ANSI mode. If the database is not using ANSI mode, an initialization query should be used on the pool to set ANSI mode:
set SESSION sql_mode = 'ANSI'
oracle - for use with Oracle 9i or later.
Sequences may be used with the Oracle translator. A sequence may be
modeled as a table with a name in source of DUAL and columns with the
name in source set to <sequencesequence name>.[nextval|currentval].
You can use a sequence as the default value for insert columns by
setting the column to autoincrement and the name in source to
<element name>:SEQUENCE=<sequence name>.<sequence value>
.
postgresql - for use with 8.0 or later clients and 7.1 or later server.
sybase - for use with Sybase version 12.5 or later.
teiid - for use with Teiid 6.0 or later.
teradata - for use with Teradata V2R5.1 or later.
Table 9.3. Execution Properties - shared by all JDBC Translators
Name | Description | Default |
---|---|---|
DatabaseTimeZone | The time zone of the database. Used when fetchings date, time, or timestamp values. | The system default time zone |
DatabaseVersion | The specific database version. Used to further tune pushdown support. | The base supported version |
TrimStrings | true to trim trailing whitespace from fixed length character strings. Note that Teiid only has a string, or varchar, type that treats trailing whitespace as meaningful. | false |
UseBindVariables | true to indicate that PreparedStatements should be used and that literal values in the source query should be replace with bind variables. If false only LOB values will trigger the use of PreparedStatements. | true |
UseCommentsInSourceQuery | This will embed a /*comment*/ leading comment with session/request id in source SQL query for informational purposes | false |
MaxPreparedInsertBatchSize | The max size of a prepared insert batch. | 2048 |
Table 9.4. Importer Properties - shared by all JDBC Translators
Name | Description | Default |
---|---|---|
catalog | See DatabaseMetaData.getTables[1] | null |
schemaPattern | See DatabaseMetaData.getTables[1] | null |
tableNamePattern | See DatabaseMetaData.getTables[1] | null |
procedurePatternName | See DatabaseMetaData.getProcedures[1] | null |
tableTypes | Comma separated list - without spaces - of imported table types. See DatabaseMetaData.getTables[1] | null |
useFullSchemaName | When false, directs the importer to drop the source catalog/schema from the Teiid object name, so that the Teiid fully qualified name will be in the form of <model name>.<table name> - Note: that this may lead to objects with duplicate names when importing from multiple schemas, which results in an exception | true |
importKeys | true to import primary and foriegn keys | true |
importIndexes | true to import index/unique key/cardinality information | true |
importApproximateIndexes | true to import approximate index information. See DatabaseMetaData.getIndexInfo[1] | true |
importProcedures | true to import procedures and procedure columns - Note that it is not always possible to import procedure result set columns due to database limitations. It is also not currently possible to import overloaded procedures. | true |
widenUnsignedTypes | true to convert unsigned types to the next widest type. For example SQL Server reports tinyint as an unsigned type. With this option enabled, tinyint would be imported as a short instead of a byte. | true |
quoteNameInSource | false will override the default and direct Teiid to create source queries using unquoted identifiers. | true |
[1] Full JavaDoc for DatabaseMetaData |
The default import settings will crawl all available metadata. This import process is time consuming and full metadata import is not needed in most situations. Most commonly you'll want to limit import by schemaPattern and tableTypes.
Example importer settings to only import tables and views from my-schema.
...
<property name="importer.tableTypes" value="TABLE,VIEW"/>
<property name="importer.schemaPattern" value="my-schema"/>
...
The LDAP translator, known by the type name ldap, exposes an LDAP directory tree relationally with pushdown support for filtering via criteria. This is typically coupled with the LDAP resource adapter.
Table 9.5. Execution Properties
Name | Description | Default |
---|---|---|
SearchDerfaultBaseDN | Default Base DN for LDAP Searches | null |
SearchDefaultScope | Default Scope for LDAP Searches. Can be one of SUBTREE_SCOPE, OBJECT_SCOPE, ONELEVEL_SCOPE. | ONELEVEL_SCOPE |
RestrictToObjectClass | Restrict Searches to objectClass named in the Name field for a table | false |
The Loopback translator, known by the type name loopback, provides a quick testing solution. It supports all SQL constructs and returns default results, with configurable behavior.
Table 9.6. Execution Properties
Name | Description | Default |
---|---|---|
ThrowError | true to always throw an error | false |
RowCount | Rows returned for non-update queries. | 1 |
WaitTime | Wait randomly up to this number of milliseconds with each sourc query. | 0 |
PollIntervalInMilli | if positive results will be "asynchronously" returned - that is a DataNotAvailableException will be thrown initially and the engine will wait the poll interval before polling for the results. | -1 |
There are no import settings for the Loopback translator; it also does not provide metadata - it should be used as a testing stub.
The Salesforce translator, known by the type name salesforce supports the SELECT, DELETE, INSERT and UPDATE operations against a Salesforce.com account. It is designed for use with the Teiid Salesforce resource adapter.
The Salesforce translator can import metadata, but does not currently have import settings.
Salesforce does not provide the same set of functionality as a relational database. For example, Salesforce does not support arbitrary joins between tables. However, working in combination with the Teiid Query Planner, the Salesforce connector supports nearly all of the SQL syntax supported by the Teiid.
The Salesforce Connector executes SQL commands by “pushing down” the command to Salesforce whenever possible, based on the supported capabilities. Teiid will automatically provide additional database functionality when the Salesforce Connector does not explicitly provide support for a given SQL construct. In these cases, the SQL construct cannot be “pushed down” to the data source, so it will be evaluated in Teiid, in order to ensure that the operation is performed.
In cases where certain SQL capabilities cannot be pushed down to Salesforce, Teiid will push down the capabilities that are supported, and fetch a set of data from Salesforce. Then, Teiid will evaluate the additional capabilities, creating a subset of the original data set. Finally, Teiid will pass the result to the client.
SELECT sum(Reports) FROM Supervisor where Division = 'customer support';
Neither Salesforce nor the Salesforce Connector support the sum() scalar function, but they do support CompareCriteriaEquals, so the query that is passed to Salesforce by the connector will be transformed to this query.
SELECT Reports FROM Supervisor where Division = 'customer support';
The sum() scalar function will be applied by the Teiid Query Engine to the result set returned by the connector.
In some cases multiple calls to the Salesforce application will be made to support the SQL passed to the connector.
DELETE From Case WHERE Status = 'Closed';
The API in Salesforce to delete objects only supports deleting by ID. In order to accomplish this the Salesforce connector will first execute a query to get the IDs of the correct objects, and then delete those objects. So the above DELETE command will result in the following two commands.
SELECT ID From Case WHERE Status = 'Closed'; DELETE From Case where ID IN (<result of query>);
*The Salesforce API DELETE call is not expressed in SQL, but the above is an SQL equivalent expression.
It's useful to be aware of unsupported capabilities, in order to avoid fetching large data sets from Salesforce and making you queries as performant as possible. See all Supported Capabilities.
A multi-select picklist is a field type in Salesforce that can contain multiple values in a single field. Query criteria operators for fields of this type in SOQL are limited to EQ, NE, includes and excludes. The full Salesforce documentation for selecting from mullti-select picklists can be found at the following link. Querying Mulit-select Picklists
Teiid SQL does not support the includes or excludes operators, but the Salesforce connector provides user defined function definitions for these operators that provided equivalent functionality for fields of type multi-select. The definition for the functions is:
boolean includes(Column column, String param) boolean excludes(Column column, String param)
For example, take a single multi-select picklist column called Status that contains all of these values.
current
working
critical
For that column, all of the below are valid queries:
SELECT * FROM Issue WHERE true = includes (Status, 'current, working' ); SELECT * FROM Issue WHERE true = excludes (Status, 'current, working' ); SELECT * FROM Issue WHERE true = includes (Status, 'current;working, critical' );
EQ and NE criteria will pass to Salesforce as supplied. For example, these queries will not be modified by the connector.
SELECT * FROM Issue WHERE Status = 'current'; SELECT * FROM Issue WHERE Status = 'current;critical'; SELECT * FROM Issue WHERE Status != 'current;working';
The Salesforce connector supports the calling the queryAll operation from the Salesforce API. The queryAll operation is equivalent to the query operation with the exception that it returns data about all current and deleted objects in the system.
The connector determines if it will call the query or queryAll operation via reference to the isDeleted property present on each Salesforce object, and modeled as a column on each table generated by the importer. By default this value is set to False when the model is generated and thus the connector calls query. Users are free to change the value in the model to True, changing the default behavior of the connector to be queryAll.
The behavior is different if isDeleted is used as a parameter in the query. If the isDeleted column is used as a parameter in the query, and the value is 'true' the connector will call queryAll.
select * from Contact where isDeleted = true;</programlisting> <para> If the isDeleted column is used as a parameter in the query, and the value is 'false' the connector perform the default behavior will call query. </para> <programlisting language="SQL"><![CDATA[select * from Contact where isDeleted = false;
If the option is selected when importing metadata from Salesforce, a GetUpdated procedure is generated in the model with the following structure:
GetUpdated (ObjectName IN string, StartDate IN datetime, EndDate IN datetime, LatestDateCovered OUT datetime) returns ID string
See the description of the GetUpdated operation in the Salesforce documentation for usage details.
If the option is selected when importing metadata from Salesforce, a GetDeleted procedure is generated in the model with the following structure:
GetDeleted (ObjectName IN string, StartDate IN datetime, EndDate IN datetime, EarliestDateAvailable OUT datetime, LatestDateCovered OUT datetime) returns ID string, DeletedDate datetime
See the description of the GetDeleted operation in the Salesforce documentation for usage details.
Salesforce does not support joins like a relational database, but it does have support for queries that include parent-to-child or child-to-parent relationships between objects. These are termed Relationship Queries. The SalesForce connector supports Relationship Queries through Outer Join syntax.
SELECT Account.name, Contact.Name from Contact LEFT OUTER JOIN Account on Contact.Accountid = Account.id
This query shows the correct syntax to query a SalesForce model with to produce a relationship query from child to parent. It resolves to the following query to SalesForce.
SELECT Contact.Account.Name, Contact.Name FROM Contact
select Contact.Name, Account.Name from Account Left outer Join Contact on Contact.Accountid = Account.id
This query shows the correct syntax to query a SalesForce model with to produce a relationship query from parent to child. It resolves to the following query to SalesForce.
SELECT Account.Name, (SELECT Contact.Name FROM Account.Contacts) FROM Account
See the description of the Relationship Queries operation in the SalesForce documentation for limitations.
The following are the the connector capabilities supported by the Salesforce Connector. These SQL constructs will be pushed down to Salesforce.
SELECT command
INSERT Command
UPDATE Command
DELETE Command
CompareCriteriaEquals
InCriteria
LikeCriteria - Supported for String fields only.
RowLimit
AggregatesCountStar
NotCriteria
OrCriteria
CompareCriteriaOrdered
OuterJoins with join criteria KEY
The Web Services translator, known by the type name ws, exposes stored procedures for calling web services backed by a Teiid WS resource adapter. It will commonly be used with the TEXTTABLE or XMLTABLE table functions to use CSV or XML formated data.
Table 9.8. Execution Properties
Name | Description | Default |
---|---|---|
DefaultBinding | The binding that should be used if one is not specified. Can be one of HTTP, SOAP11, or SOAP12 | SOAP12 |
DefaultServiceMode | The default service mode. For SOAP, MESSAGE mode indicates that the request will contain the entire SOAP envelope and not just the contents of the SOAP body. Can be one of MESSAGE or PAYLOAD | PAYLOAD |
XMLParamName | Used with the HTTP binding (typically with the GET method) to indicate that the request document should be part of the query string. | null - unused |
There are ws importer settings, but it does provide metadata for dynamic VDBs.
The WS translator exposes low level procedures for accessing web services. See also the ws-weather example in the kit.
Invoke allows for multiple binding, or protocol modes, including HTTP, SOAP11, and SOAP12.
Procedure invoke(binding in STRING, action in STRING, request in XML, endpoint in STRING) returns XML
The binding may be one of null (to use the default) HTTP, SOAP11, or SOAP12. Action with a SOAP binding indicates the SOAPAction value. Action with a HTTP binding indicates the HTTP method (GET, POST, etc.), which defaults to POST.
A null value for the binding or endpoint will use the default value. The default endpoint is specified in the WS resource adapter configuration. The endpoint URL may be absolute or relative. If it's relative then it will be combined with the default endpoint.
Since multiple parameters are not required to have values, it is often more clear to call the invoke procedure with named parameter syntax.
call invoke(binding=>'HTTP', action=>'GET')
The request XML should be a valid XML document or root element.
invokeHttp
can return the byte contents of an HTTP(S) call.
Procedure invokeHttp(action in STRING, request in OBJECT, endpoint in STRING, contentType out STRING) returns BLOB
Action indicates the HTTP method (GET, POST, etc.), which defaults to POST.
A null value for endpoint will use the default value. The default endpoint is specified in the WS resource adapter configuration. The endpoint URL may be absolute or relative. If it's relative then it will be combined with the default endpoint.
Since multiple parameters are not required to have values, it is often more clear to call the invoke procedure with named parameter syntax.
call invokeHttp(action=>'GET')
The request can be one of SQLXML, STRING, BLOB, or CLOB. The request will be sent as the POST payload in byte form. For STRING/CLOB values this will default to the UTF-8 encoding. To control the byte encoding, see the to_bytes function.
Teiid integration is available via a "Dynamic VDB" without the need for Teiid Designer tooling. While this mode of operation does not yet allow for the creation of view layers, the underlying sources can still be queried as if they are a single source. See the kit's "teiid-example/dynamicvdb-*" for working examples.
To build a dynamic VDB, you'll need to create a
file. The XML file captures
information about the VDB, the sources it integrate, and preferences for importing metadata.
SOME-NAME
-vdb.xml
VDB name pattern must adhere to "-vdb.xml" for the Teiid VDB deployer to recognize this file as a dynamic VDB.
my-vdb.xml: (The vdb-deployer.xml schema for this file is available in the schema folder under the docs with the Teiid distribution.)
<vdb name="${vdb-name}" version="${vdb-version}">
<property name="UseConnectorMetadata" value="..." />
<!-- define a model fragment for each data source -->
<model name="${model-name}">
<property name="..." value="..." />
...
<source name="${source-name}" translator-name="${translator-name}"
connection-jndi-name="${deployed-jndi-name}">
</model>
<!-- create translator instances that override default properties -->
<translator name="${translator-name}" type="${translator-type}" />
<property name="..." value="..." />
...
</translator>
</vdb>
Attributes
name
The name of the VDB. The VDB name referenced through the driver or datasource during the connection time.
version
The version of the VDB (should be an positive integer). This determines the deployed directory location (see Name), and provides an explicit versioning mechanism to the VDB name.
Property Elements
UseConnectorMetadata
Setting to use connector
supplied metadata. Can be "true" or "cached". "true" will obtain
metadata once for every launch of Teiid. "cached" will save a file
containing the metadata into the
directory
PROFILE
/data/teiid
Attributes
name
The name of the model is used as a top level schema name for all of the metadata imported from the connector. The name should be unique among all Models in the VDB and should not contain the '.' character.
version
The version of the VDB (should be an positive integer). This determines the deployed directory location (see Name), and provides an explicit versioning mechanism to the VDB name.
Source Element
name
The name of the source to use for this model. This can be any name you like, but will typically be the same as the model name. Having a name different than the model name is only useful in multi-source scenarios.
translator-name
The name or type of the Teiid Translator to use. Possible values include the built-in types (ws, file, ldap, oracle, sqlserver, db2, derby, etc.) and translators defined in the translators section.
connection-jndi-name
The JNDI name of this source's connection factory. There should be a corresponding "-ds.xml" file that defines the connection factory in the JBoss AS. Check out the deploying VDB dependencies section for info. You also need to deploy these connection factories before you can deploy the VDB.
Property Elements
importer.<propertyname>
Property to be used by the connector importer for the model for purposes importing metadata. See possible property name/values in the Translator specific section. Note that using these properties you can narrow or widen the data elements available for integration.
Attributes
name
The name of the the Translator. Referenced by the source element.
type
The base type of the Translator. Can be one of the built-in types (ws, file, ldap, oracle, sqlserver, db2, derby, etc.).
Property Elements
Set a value that overrides a translator default property. See possible property name/values in the Translator specific section.
When you have multiple instances of data that are using identical schema (horizontal sharding), Teiid can help you aggregate data across all the instances, using "multi-source" models. In this scenario, instead of creating/importing a model for every data source, user needs to define one source model that represents the schema and configure multiple data "sources" underneath it. During runtime, when a query issued against this model, the query engine analyzes the information and gathers the required data from all the sources configured and aggregates the results and provides in a single result set.
To mark a model as multi-source, the user needs to supply property called supports-multi-source-bindings, in the "vdb.xml" file. Also, the user needs to define multiple sources. Here is code example showing dynamic vdb with single model with multiple sources defined.
<vdb name="vdbname" version="1">
<model visible="true" type="PHYSICAL" name="Customers" path="/Test/Customers.xmi">
<property name="supports-multi-source-bindings" value="true"/>
<source name="chicago"
translator-name="oracle" connection-jndi-name="chicago-customers"/>
<source name="newyork"
translator-name="oracle" connection-jndi-name="newyork-customers"/>
<source name="la"
translator-name="oracle" connection-jndi-name="la-customers"/>
</model>
</vdb>
In the above example, the VDB defined has single model called Customers
,
that has multiple sources (chicago
, newyork
,
and la
) that define different instances of data. Every
time a model is marked as "multi-source", the
runtime engine adds a additional column called "SOURCE_NAME" to every table in
that model. This column maps to the source's name from the XML. In
the above XML code that would be chicago
, la
,
newyork
. This allows queries like the following:
select * from table where SOURCE_NAME = 'newyork' update table column=value where SOURCE_NAME='chicago' delete from table where column = x and SOURCE_NAME='la' insert into table (column, SOURCE_NAME) VALUES ('value', 'newyork')
Note that when user do not supply the "SOURCE_NAME" in the criteria, the command applies to all the sources. If SOURCE_NAME supplied, the query is executed only aginst the source specified. Another useful feature along with this feature is "partial results" to skip unavailable sources if they are down.
Currently the tooling support for managing the multi-source feature is limited, so if you need to use this feature build the VDB as usual in the Teiid Designer and then edit the "vdb.xml" file in the VDB archive using a Text editor to add the additional sources as defined above. You must deploy a separate data source for each source defined in the xml file.
If you would like to use "SOURCE_NAME" in your transformations to control which sources are accessed or updated, you would manually need to add this extra column on your view table in the Designer. This column will not be automatically added on the source table, when you import the medata from source.