JBoss Community Archive (Read Only)

RHQ 4.9

Hosting RHQ Server in AS7

Hosting RHQ Server in JBossAS 7 App Server

https://bugzilla.redhat.com/show_bug.cgi?id=851268 tracks this AS7 port work

JIRAs encountered

These are JIRAs that have been come across during our port. These either must be worked around or fixed for our port to be considered 100% done.

List of things to do or be aware of

RHQ 4 and below host the RHQ Server .ear (and related deployments) in a JBossAS 4.2.3 app server instance. We want to upgrade our container to JBossAS 7. This page is just a set of brainstorming notes to list the issues that will need to be discussed in order to successfully port the RHQ Server to use JBossAS 7. Some tips to migrate can be found in the AS7.1 Developer's Guide, section "How do I migrate my application from JBoss AS 5 or AS 6 to JBoss AS 7?".

We may want to conform to the new Layered Distribution model.

Already Done

  • The installer uses very specific JBossAS 4 features to install and uninstall the RHQ Server ear and the installer war (i.e. it uses the ".rej" extension to denote when something should not be deployed). I believe we can utilize the new JBossAS 7 feature of using ".dodeploy" type file markers to install and uninstall the RHQ Server ear and the installer.

  • We could take this opportunity to re-implement the installer. Today, the installer does nothing but file manipulation to install things and configure the server. We can now utilize the JBoss management Java API to call into the server to deploy and configure things.

  • The JBossAS 7 server comes with Hibernate 4. Our domain and server/jar code may have to change to accomodate this. We also need to look at our code that makes specific Hibernate API calls (like the HibernateDetachUtility) because it may be broken once it is run on Hibernate 4

  • We have to look at how our comm layer will operate within JBossAS 7. We currently use an old version of JBoss/Remoting (and this includes not only the client code, but also the server-side servlet that accepts incoming messages from agents). We need to determine what kind of work needs to be done in order to convert the RHQ Server's comm module to use whatever version of remoting can run in JBossAS 7.

    • this stuff "just worked" as before. our old remoting jars are still packaged in server ear and in agent and all appears working

  • StartupServlet is used to perform alot of startup code. Can that stay? [answer: we reimplemented it as a EJB startup singleton bean- see StartupBean]

  • need to pre-deploy the RT helper filter so we can monitor the server itself

  • We do alot of customization of the container in the rhq-container-build.xml - we'll need to refactor that alot to get it to work for AS7.

  • JBoss Modules looks like it could be very useful - it could perhaps even replace our server-side and agent-side plugin container classloading mechanism. (this is for future efforts)

  • need some explicit dependencies specified using the jboss dependency descriptor xml file

  • I think we need to fix all of our JNDI lookups since JNDI naming is more strict in AS7.

  • The RHQ Server has code within it that talks to the underlying JBossAS server via JMX - it does this to get things like the JBossAS home directory, deploy directory, information about the version of the server and so on. We will need to port this over to talk to the AS7 server to get similar data. [AS7 still has some MBeans. We still use them where we can or we use AS7 mgmt API]

  • WebDAV goes away now that REST is in

  • We have many (much more than I thought) MBeans that use the old @Management annotation. These are gone now. We have to refactor. 

  • Removing Web Service support.

    • After removal has been completed and we are running in AS7, we can look to remove the safe-invoker module completely. This was used for WS support but contains HibernateDetachUtility, which is used for all of our client support. That one class should probably migrate somewhere else, like server jar.

  • Java7 builds only. Maven3.0.4 only. Java6 or Java7 runtime.

  • Improve testing (all subitems below are done)

    • Can we now use JBossAS 7 in our integration tests (today we use a very old and obsolete "Embedded EJB3 Container" to run our tests. How do we run our server/jar tests, for example, in a way that runs inside of JBossAS 7 and not the embedded EJB3 container.

    • As of 10/5/2012, all modules pass tests except:

      • RHQ Domain Model

      • RHQ Enterprise Server JAR

      • RHQ Server Integration Tests

      • RHQ Enterprise Agent (this is due to port conflicts, when I run the tests individually, they all pass)

    • As of 11/21/2012, all modules pass

  • See if we should remove container-lib module, I don't think we need it anymore

  • We have customized secured JNDI lookup to prevent serverside scripts from accessing database. This is AS4 bound and will have to change for AS7.

    • Enable the integration tests

  • Reconfigure JMS system to use HornetQ since it is the new messaging provider in AS7

    • See https://docs.jboss.org/author/display/AS7/Messaging+configuration

    • We cannot configure the JMS system to use the backend DB - hornetQ will not support this

      • https://issues.jboss.org/browse/HORNETQ-887

      • This thread describes our situation exactly: https://community.jboss.org/thread/153581

      • My thread asking how hard it would be to implement this: https://community.jboss.org/thread/205741

      • Some notes from Justin during our 9/24/2012 IRC chat:

        (11:49:36 AM) ccrouch: jbertram: do you have an ETA on when the replication work will hit JBAS, and then when it will hit EAP ?
        (11:49:38 AM) jshaughn: mazz, right
        (11:50:03 AM) ccrouch: jbertram: I'm referring to https://issues.jboss.org/browse/HORNETQ-887?focusedCommentId=12703606&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12703606
        (11:50:33 AM) jbertram: ccrouch: we're still in ALPHA in the community...once were out then it will have to go to AS7 (which of course will depend on their release cycle) and then to EAP...I can't say when all that will line up
        (11:50:47 AM) ccrouch: (10:46:07 AM) jshaughn: whether or not we can live with messages not being processed on a single server until that server comes back up
        (11:50:47 AM) ccrouch: assuming it does come back up and the HDD isnt dead
        (11:50:48 AM) jbertram: ccrouch: but I think we'll have a GA release before the end of theyear

      • Some notes with Clebert from #hornetq
        ?

        (12:41:27 PM) jshaughn: (12:29:58 PM) clebert: mazz: we need to comunicate between the servers : So, if I had multiple AS7 servers with a durable JMS Queue configured to use the same journal directory, would those AS7 servers need to communicate in any other way to process messages off of the queue effectively?
        (12:42:04 PM) clebert: jshaughn: you can't use the same journal directory between the servers.. each instance will have its own journal
        (12:42:25 PM) jshaughn: If it were a shared disk?
        (12:42:47 PM) clebert: jshaughn: each running instance will have its own folder
        (12:42:59 PM) clebert: jshaughn: we don't share journals among running instances
        (12:43:13 PM) clebert: the only scenario we share journal folder is between an active and a backup server
        (12:43:15 PM) jshaughn: ah, ok, so each server/broker would still be processing its own messages
        (12:43:21 PM) clebert: but you would only have one server alive per folder
        (12:43:53 PM) clebert: jshaughn: if you actually manage to place multipl servers on the same folder.. you will create a mess.. no server will be able to restart
        (12:43:54 PM) jshaughn: so if one server went down it's persisted messages would not get processed until that server came back up
        (12:44:08 PM) clebert: jshaughn: or it's backup
        (12:44:12 PM) clebert: jshaughn: correct
        (12:44:20 PM) jshaughn: ok, thanks
        (12:44:21 PM) clebert: jshaughn: it's a journal system.. not a db system
        (12:44:26 PM) clebert: jshaughn: the journal is write only

        images/author/images/icons/emoticons/smile0.gif

        (12:44:33 PM) clebert: until you restart the server
        (12:44:38 PM) clebert: that's the only time we read the journal
        (12:44:56 PM) jshaughn: right, we are just trying to get the right mind-set after years of using the jbossmq/jdbc setup we had in place
        (12:44:57 PM) clebert: a journal is a write-only system.. only used for recover
        (12:45:15 PM) clebert: jshaughn: I see
        (12:45:29 PM) clebert: jshaughn: perhaps that was a better usage for you guys as you didn't need to scale it
        (12:45:37 PM) clebert: jshaughn: it was a singleton queue per cluster
        (12:45:46 PM) jshaughn: right
        (12:45:50 PM) clebert: doesn't work very well on real message scenarios
        (12:45:55 PM) jshaughn: it worked well for our needs in that way
        (12:46:49 PM) jshaughn: due to relatively low message volumes and a way for our servers to share messages without actually having any comm between servers, other than the db
        (12:50:15 PM) mazz: yeah, that's a good point that needs to be mentioned
        (12:50:22 PM) mazz: this was a low volume scenario
        (12:50:41 PM) mazz: we didn't need to shoot hundreds/thousands of messages per minute or anything like that

  • Can we get the installer to "inject" the ear, datasources, etc, into an existing, external AS7 installation?

  • After the installer runs, we do not want the deployment scanner running.

  • Need someone with Windows to test the wrapper stuff (to install and run as a service) and running in foreground in console.

Left To Do

I put images/author/images/icons/emoticons/error0.gif next to items still left to do so its easier to see. Remove that icon when a line item is crossed out.

RHQ 4.6

  • (4.6) MAZZ Web server/Tomcat SSL secure configuration (ports, certs). Need to go through the "securing communications" documentation and make sure that stuff still works. We need to be able to use certs so the agent can be authorized.

    • make sure out of box browser https access works

  • (4.6) MIKE AND QE QE to sign off on automated test results    Automated tests are running and generally passing.  Evidence of test automation results provided internally to team members. 

  • (4.6) (THOMAS) Engineering Test HA (shut down nodes, see failover, test GUI)

  • (4.6) (JIRI) Engineering Test upgrades from older versions, e.g. RHQ4.5, with data in the database

    • (jay) I also did some upgrade testing, specifically with LDAP coonfigured. Working.

  • (4.6 (JAY) release notes) Document dealing with removal JMS table removal.

  • (4.6) JAY Investigate if we need to do anything globally regarding application exceptions and transaction rollbacks – --https://bugzilla.redhat.com/show_bug.cgi?id=886850-

    • (4.6) JAY verify exceptions getting handled appropriately by reviewing code coverage

  • (4.6) MAZZ We need to look very carefully at the transaction manager and make sure that we are properly configured so we don't run into problems like we did early on in RHQ, that being, the tx manager would grow the filesystem unbounded when transactions failed. We had to set up our database as an XA datasource in order to get things working and we had to patch the tx manager in JBossAS 4.2.3 to get things to work. Update: I tried two things. One, I artificially forced exceptions to be thrown from our alert consumer JMS bean and I didn't see any bad behavior. I also shutdown the database while our server was running and after about a minute or two, restarted it. Again no bad behavior. things look ok

  • (4.6) JAY release notes - Document the change in behavior with the HornetQ JMS message storage

  • (4.6) HEIKO Rest WAR can go away - REST is folded into the EAR - get Heiko to sign off on it. This needs to stay if at least for security wiring. But also to define the context-root

  • (4.6) MAZZ look into running rhq-installer.sh --reconfig inside rhq-server.sh. see if the values did not change, if they did not change, then don't restart. This is to work around things like https://issues.jboss.org/browse/AS7-5321 so we can customize the email service via our .properties file (other settings are like this)

  • (4.6) SIMEON Test LDAP integration - get Simeon to sign off on it

  • (4.6) MAZZ see if we can we edit alert email template file now that it is located in the modules/ directory. Update: we can. since we deploy the ear exploded inside modules/ you can edit the alert templates at <install-dir>/modules/org/rhq/rhq-enterprise-server-startup-subsystem/main/deployments/rhq.ear/rhq-enterprise-server-ejb3.jar/alert-email-template.txt

  • (4.6) MAZZ Support direct access to RHQ UI through http://localhost:7080 (with no context root). Update: portal-war has a index.html at the context root that redirects to /coregui automatically.

  • (4.6) LUKAS arquillian+jacoco code coverage patch - lukas to sign off on this signed off

    images/author/images/icons/emoticons/check0.gif

  • (4.6) JAY check on hibernate patches

  • (4.6) MIKE T/JOHN S/HEIKO https://issues.jboss.org/browse/AS7-3199 (this will NOT be fixed - NOT WORKED AROUND - I had to disable our reports stuff in coregui to workaround this)

    • JAY It was some time since mazz ran into this issue, and a lot changed after that in our setup.  Although the AS bug is not fixed, I was able to the "export" feature working again after fixing some JNDI naming issues in coregui's web.xml.  So, it seems we're not dependent on that fix and the feature seems to be working.

  • (4.6) HEIKO we need a new RHQ doc space in confluence for the new docs

    • MAZZ Installer docs

    • MAZZ Comm Configuration doc changes

    • MAZZ GettingStarted>Features and UserDocumentation>UDFeatures are duplicating stuff, we need to have one "Features" section (probably under UserDocumentation and have GettingStarted just link to those pages).

  • (4.6) MAZZ When using the Support subsystem, when the agent uploads the data for users to download, the server stores the file(s) in rhq.ear/rhq-downloads (which is now inside /modules). Our DownloadServlet uses this location to find things that an HTTP client wants to download (http://localhost:7080/downloads/something-from-rhq-downloads-directory.dat) We will need to change the downloads servlet to look somewhere else perhaps. Update: we already do this - as part of the AS7 port work, I put the support zip in the AS7 data directory. The download servlet will look in data if it can't find what its looking for under the ear rhq-download directory which is under our /modules dir.

  • (4.6) https://bugzilla.redhat.com/show_bug.cgi?id=902876 what happens if you are running a standlone AS7 already and you try to install?

  • (4.6) https://bugzilla.redhat.com/show_bug.cgi?id=906500 quartz throwing NoClassDefFoundError: oracle/sql/BLOB

JON 3.2

  • (JON 3.2) The installer should disable the AS7 admin web console, including the root welcome page (we already disable the root welcome page)

  • (JON 3.2) Verify the Windows remove option. Ensure that 'disabled' will eventually finish with a full remove. (what about after a reboot, is the service still there?).

    • from jshaughn: This seems to be a normal windows thing wrt service removal. I haven't noticed any issues with removal.

  • (JON 3.2) If not already done so remove all unused profiles from the server (the only configuration we ship with is standalone's standalone-full.xml)

  • (JON 3.2) https://issues.jboss.org/browse/AS7-4975 and related issues and forum thread (memory leak in XA datasources in AS7.1.1.Final NOT WORKED AROUND I think an upgrade to the postgres Version 9.2-1002 driver will solve this ... We already moved to the updated driver in the 4.6 release time-frame.

  • (JON 3.2) Now that we are going to be on at least EAP 6.1.alpha for upstream, see if we can remove temp fixes - BZ 893609 that worked around expression non-support in AS7.

  • (JON 3.2) Running on IBM JRE 6: see https://bugzilla.redhat.com/show_bug.cgi?id=rhq-2065 for issues we had running on IBM JRE 6. We need to make sure the SSL stuff works and we need to make sure we don't have to play games with xalan.jar (I don't think we do, but this note is here to double check this).

  • (JON 3.2) There seems to be way more ports open than we actually need right now. There needs to be a specific, documented reason, why each port is needed to be ...(not sure what ports this is referring to, but all ports have been examined and their binding addresses adjusted - see BZ 966160)

  • (JON 3.2) Make sure the interface that the various services are binding to is configurable using the standard mechanism and all services use only the interface or interfaces as configured. See BZ 966160 for more.

  • (JON 3.2) Turn off unneeded services, prevent content deployment either through the AS cli or any other means, remove example DS, test JMS queue (I don't see any test JMS queues in RHQ Server, either there isn't one in EAP or we already removed it).

  • (JON 3.2) fix the distribution; there is are old artifacts sitting in <install-dir>/jbossas/server/... - "server" and everything underneath should not be here anymore. BZ 966760

  • (JON 3.2) replace the obsolete JBoss-by-Redhat logo with the new RH logotype

  • (JON 3.2) Make sure any users needed to connect to RHQs' AS7 server do not have their passwords stored in cleartext. Use a random password which is different per install and it is possible to change the password in case it becomes known or preferably use the default file permission based access method which doesn't utilize password

  • (JON 3.2) Deploy JON on top of EAP6.1.x.GA versus EAP6.1.0.alpha

    • requires changes to maven pom.xml and rhq-container.build.xml

  • (JON 3.2) Update jboss/bin/product.conf to RHQ

    • And correspondingly to JON as part the JON build

  • (JON 3.2) Enable/Run the Perf integration tests? Do we ever actually run these tests?

  • (JON 3.2) Interesting perf issue   https://bugzilla.redhat.com/show_bug.cgi?id=912468

  • (JON 3.2) New RH doc changes

FUTURE

  • (FUTURE) Split out current single module into multiple modules to help with patching (i.e. smaller patches)

  • (FUTURE) Regarding Remoting - we should look to see if we can move to Remoting 3 (AS7.1.1.Final uses 3.2.3.GA)

  • (FUTURE) research how we can do Domain Controller integration - can we get the RHQ Server to be deployed inside a Domain Controller? what does this buy us?

  • (FUTURE) would be nice to put things like the alert email template file and the downloadable things (e.g. agent binary) in an external dir. This is so users can change them but not get in the way of the module content so we can patch our stuff and not clobber user-customizations

  • (FUTURE)Contemplate using different sequence generator allocationSize values

    • We use allocation size = 1 for all entities right now. See Constants.ALLOCATION_SIZE for more. For high cardinality tables it may be more efficient to increase the
      allocation size to lessen db round trips when persisting entities. For each entity it would require a db upgrade step to update the sequence's increment and an accompanying
      change to the SequenceGenerator declaration.

  • (FUTURE) Be able to support the new AS7 patching mechanism

  • (FUTURE) write some unit tests for PersistenceUtil so we can make sure we catch any changes in Hibernate internals in the future

  • (FUTURE) Remove temp workaround code once WFLY-1177 is fixed - this is BZ https://bugzilla.redhat.com/show_bug.cgi?id=958494

    Embedded Agent and Standalone Agent Packaging

As part of the AS7 porting work, I spent some time looking at the packaging of both the embedded agent (within a SAR deployed in our RHQ Server AS container) and the standalone agent.

As for the embedded agent - when we start shipping with an AS7 container, we will no longer ship with even an experimental version of the embedded agent. AS7 has some additional issues we would have to resolve in order to get the embedded agent to work. I managed to work around some, but not all the way. Rather than spend more time on this issue, I have taken out the embedded agent from the build (this is in the rhq-on-as7 branch, we'll merge to master within the next few months).

As for the standalone agent, I wanted to start using some newer JBoss jars that come from the AS7 line (for example, the JMX jar "jboss-as-jmx-7.1.1.jar" to replace "jboss-jmx-4.2.3.GA.jar"). Alas, we can't do it without upgrading our remoting infrastructure. This is because the remoting infrastructure needs code that exists in the old JMX jar but is no longer in the new JMX jar.

In order to move away from some of these older JBoss jars, we'd need to investigate how we can swap out the Remoting 2.x stuff with the newer Remoting 3.x.

Located Management Client

I figured out how to get a co-located client and I blogged about it here: http://management-platform.blogspot.com/2012/07/co-located-management-client-for.html

Server Installation Details

Here's the server's basic details using jboss-cli.sh: 

/core-service=server-environment/:read-resource(include-runtime=true)
{
    "outcome" => "success",
    "result" => {
        "base-dir" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final/standalone",
        "config-dir" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final/standalone/configuration",
        "config-file" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final/standalone/configuration/standalone.xml",
        "content-dir" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final/standalone/data/content",
        "data-dir" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final/standalone/data",
        "deploy-dir" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final/standalone/data/content",
        "ext-dirs" => [
            "/home/software/java/jdk1.6.0_29/jre/lib/ext",
            "/usr/java/packages/lib/ext"
        ],
        "home-dir" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final",
        "host-name" => "mazztower",
        "initial-running-mode" => "NORMAL",
        "launch-type" => "STANDALONE",
        "log-dir" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final/standalone/log",
        "modules-dir" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final/modules",
        "node-name" => "mazztower",
        "qualified-host-name" => "mazztower",
        "server-name" => "mazztower",
        "temp-dir" => "/home/mazz/bin/jbossas/jboss-as-7.1.1.Final/standalone/tmp"
    }
}

DMR and ModelController Libraries

The AS7 detyped management library can be used to connect to an AS7 and manage it, as opposed to using HTTP/JSON/REST. DMR is just the small library that provides the ModelNode detyped API, the ModelController/ModelControllerClient classes use DMR to access the management infrastructure in AS7.

The CLI has what is called "batch" mode - which is called "composite" in the Management API docs. This lets you atomically invoke N commands and if any fails, the entire set is rolled back. The operation is called "composite" and you execute it on the root resource (address is ""). There is a "steps" list that are all the executions you want invoked. See Composite Operations.

To run a bsh and test DMR API calls: 

$ java -cp ~/.m2/repository/bsh/bsh/1.3.0/bsh-1.3.0.jar:<AS7-install>/modules/org/jboss/dmr/main/jboss-dmr-1.1.1.Final.jar bsh.Interpreter
BeanShell 1.3.0 - by Pat Niemeyer (pat@pat.net)
bsh % import org.jboss.dmr.*;
bsh % ModelNode node = new ModelNode();

Client Requirements

ccrouch: I think we need to reference what new jar's people are going to need when writing a Java client to talk to JON hosted on AS7. It would be great to have an example of an app running inside an EAP6 instance calling out to JON.

How SOA is doing it

Here's how SOA is currently doing this (as of dec 5, 2012 from #jbosssoa) 

(11:05:14 AM) mazz: dpalmer: so, that document you [linked|https://docspace.corp.redhat.com/docs/DOC-121099] did help me understand what you are doing. are you getting any pushback for having your own startup script? (standalone-soa.sh)
(11:05:16 AM) mazz: ?
(11:05:19 AM) dpalmer: mazz: BTW, EAP 6.0.1.ER4 fixes the product.conf issue
(11:05:45 AM) dpalmer: mazz: a little pushback from Max Andersen
(11:06:09 AM) dpalmer: mazz: Max is asking Jason to move the jboss specific properties into a .properites file
(11:06:17 AM) mazz: it would be nice for tooling to have a common startup mechanism so it doesn't surprise me Max doesn't like this
(11:06:55 AM) dpalmer: mazz: I would prefer not to have my own scripts too
(11:07:16 AM) mazz: ok, one other question is the conf file - do you guys ship with your own standalone.xml ? or do you make AS mgmt calls to setup the out-of-box AS config for SOA?
(11:07:26 AM) dpalmer: mazz: the script only exports RUN_CONF and calls standalone.sh
(11:07:59 AM) dpalmer: mazz: we ship our own standalone.xml in soa/standalone/configuration
(11:08:21 AM) mazz: so you'll have to be careful to track any changes in future AS7 versions to remani compatible then, right?
(11:08:38 AM) dpalmer: mazz: setting JBOSS_BASE_DIR means that version is read instead of the default
(11:08:43 AM) dpalmer: mazz: yes
(11:09:13 AM) dpalmer: mazz: for soa, our config is an xslt transform of the EAP version
(11:09:55 AM) mazz: does soa have its own installer? or do you just unzip and go? (and I assume you haven't done any rpm packaging yet?)
(11:10:11 AM) dpalmer: mazz: currently unzip and go
(11:10:29 AM) dpalmer: mazz: installers and rpms are on PM's wish list for 6

Miscellaneous Good Things

(jsanda) I want to ask about though our authentication code. I know that there have been discussions of a container-based solution, but as a first step I would like to see our authentication code, which is based on Struts 1.x, moved out of portal-war and into coregui. Now seems like the perfect time to make the change. It is somewhat related to this discussion in so far as portal-war prevents us from using CDI. There is a problem with the versions of the JSF libraries we package in portal-war that cause CDI to fail upon initialization and subsequently prevents the RHQ ear from successfully initializing.

(jshaughn) This is a bit tangential but I think is something we need to ask ourselves about in the JON release timeframe. We have had other inquiries into our auth approach:

(tfonteyne) I wanted to check what the plans are for supporting standard JavaEE security to login instead of the GWT + proprietary stuff. Reason for asking is two recent requests:

Support for Windows logins, with SPNEGO SSO

Support for PKI certificates and similarly client certificates.

If JON would delegate the authentication to JavaEE, then any standard JAAS module could be used making the above and other methods just work.

(jsanda) I raised two points, one being the mechanism we use for authentication and the other being where the authentication code lives. Since our existing authentication code works, for the short term, I advocate moving that logic out of portal-war and into coregui to get use closer to being able to completely remove portal-war. An alternative authentication strategy is/could be a separate discussion.

JBoss.org Content Archive (Read Only), exported from JBoss Community Documentation Editor at 2020-03-13 08:08:43 UTC, last content change 2013-09-18 19:41:46 UTC.