Skip to end of metadata
Go to start of metadata

In this article, I'd like to show you how to set up WildFly 8 in domain mode and enable clustering so we get HA and session replication working among the nodes. It's a step-by-step guide so you can follow the instructions in this article and build the sandbox yourself .

Preparation & Scenario

Preparation

We need to prepare two hosts (or virtual hosts) to carry out the experiment. We will use the two hosts as follows:

  • Install Fedora 16 on them (other Linux versions are probably fine as well but I'll use Fedora 16 in this article)
  • Make sure that they are in same local network (subnet)
  • Make sure that they can access each other via different TCP/UDP ports (best to turn off the firewall and disable SELinux during the experiment or these will cause network problems)

Scenario

Here are some details on what we are going to do:

  • Let's call one host 'master' and the other 'slave'.
  • Both master and slave will run WildFly 8: master will run as domain controller, and slave will under the domain management of master.
  • Apache httpd will be run on master, and in httpd we will enable the mod_cluster module. The WildFly 8 on master and slave will form a cluster and be discovered by httpd.

  • We will deploy a demo project into domain, and verify that the project is deployed into both master and slave by domain controller. Thus we will see that domain management provide us a single point to manage the deployments across multiple hosts in a single domain.
  • We will access the cluster URL and verify that httpd has distributed the request to one of the WildFly host. So we could see the cluster is working properly.
  • We will try to make a request of the cluster, and if the request is forwarded to master, we then kill the WildFly process on master. After that we will go on making requests of the cluster and we should see the request forwarded to slave without the session being lost. Our goal is to verify that HA is working and sessions are replicated.
  • After the previous step has finished, we reconnect the master by restarting it. We should see the master is registered back into the cluster, and we should also see slave recognising master as domain controller again and connect to it.

Please don't worry if you cannot digest so many details currently. Let's move on and you will get the points step by step.

Download WildFly 8

First we should download WildFly 8 from the website:

The version I downloaded is JBoss AS 7.1.0.CR1b, please don't use the version prior to this one, or you will meet this bug when running in clustering mode:

After download finished, I got the zip file:

Note: The name of your archive will differ slightly due to version naming conventions.
Then I unzipped the package to master and try to make a test run:

If all is well, we should see WildFly successfully start in domain mode:

Now exit master and let's repeat the same steps on slave. Once we get WildFly 8 running on both master and slave, we can move on to the next step.

Domain Configuration

Interface config on master

In this section we'll set up both master and slave to run in domain mode, and configure master to be the domain controller.

First open the host.xml in master for editing:

The default settings for interface in this file are:

We need to change the address to the management interface so slave can connect to master. The public interface allows the application to be accessed by non-local HTTP, and the unsecured interface allows remote RMI access. My master's IP address is 10.211.55.7, so I change the config to:

Interface config on slave

Now we will set up the interfaces on slave. First we need to remove the domain.xml from slave, because slave will not act as domain controller and it will operate under the management of master. I just rename the domain.xml so it won't be processed by WildFly:

From JBoss AS 7.1.Final you don't need to rename domain.xml anymore.

Then let's edit host.xml. Similar to the steps on master, open host.xml first:

The configuration we'll use on slave is a little bit different, because we need to let slave connect to master. First we need to set the hostname. We change the name property from:

to:

Then we need to modify the domain-controller section so slave can connect to master's management port:

As we know, 10.211.55.7 is the IP address of master.

Finally, we also need to configure interfaces section and expose the management ports to the public address:

10.211.55.2 is the IP address of the slave. Refer to the domain controller configuration above for an explanation of the management, public, and unsecured interfaces.

It is easier to turn off all firewalls for testing, but in production, you need to enable the firewall and allow access to the following ports: TBD

Security Configuration

If you start WildFly on both master and slave now, you will see that slave cannot be started, with following error:

This is because we haven't properly set up the authentication between master and slave. Now let's work on it:

Master

In the bin directory there is a script called add-user.sh. We'll use it to add new users to the properties file used for domain management authentication:

As shown above, we have created a user named 'admin' whose password is '123123'. Then we add another user called 'slave':

We will use this user for slave host to connect to master.

Notice that the username must be equal to the name given in slave's host element. This means that you need a user for each additional host.

In newer versions of WildFly, the add-user.sh script will let you choose the type of the user. Here we need to choose the 'Management User' type for both the 'admin' and 'slave' accounts:

Slave

On slave we need to configure host.xml for authentication. We should change the security-realms section as following:

We've added server-identities into security-realm, which is used for authenticating the host when slave tries to connect to master. Because slave's host name is set to 'slave', we should use the 'slave' user's password on master. In the secret value property we have 'MTIzMTIz=', which is the Base64 code for '123123'. You can generate this value by using a base64 calculator such as the one at http://www.webutils.pl/index.php?idx=base64.

Then in the domain controller section we also need to add the security-realm property:

So the slave host could use the authentication information we provided in 'ManagementRealm'.

Dry Run

Now everything is set for the two hosts to run in domain mode. Let's start them by running domain.sh on both hosts. If everything goes fine, we can see from the log on master:

That means all the configurations are correct and two hosts are running in domain mode now as expected. Hurrah!

Deployment

Now we can deploy a demo project into the domain. I have created a simple project located at:

We can use the git command to fetch a copy of the demo:

In this demo project we have a very simple web application. In web.xml we've enabled session replication by adding following entry:

And it contains a jsp page called put.jsp which will put the current time into a session attribute called 'current.time':

Then we can obtain this value using get.jsp:

It's an extremely simple project but it could help us to test the cluster later: we will access put.jsp from the cluster and check that the request is distributed to master, then we disconnect master and access get.jsp. We should see the request is forwarded to slave but the 'current.time' value is held by session replication. We'll cover this in more detail later.

Let's go back to this demo project. Now we need to create a war from it. In the project directory, run the following command to get the war:

It will generate cluster-demo.war. Then we need to deploy the war into domain. First we should access the http management console on master (because master is acting as domain controller):

It will prompt you for an account name and password.  We can use the 'admin' account we added a short while ago. After logging in we see the 'Server Instances' window. By default there are three servers listed, which are:

  • server-one
  • server-two
  • server-three

We can see that server-one and server-two have the status 'running' and that they belong to main-server-group; server-three has the status 'idle', and belongs to other-server-group.

All these servers and server groups are set in domain.xml on master as7. What we are interested in is the 'other-server-group' in domain.xml:

We can see that this server-group is using the 'ha' profile, which then uses the 'ha-sockets' socket binding group. It enables all the modules we need to establish the cluster later (including infinispan, jgroup and mod_cluster modules). So we will deploy our demo project into a server that belongs to 'other-server-group', so 'server-three' is our choice.

In newer versions of WildFly, the profile 'ha' changes to 'full-ha':

Let's go back to the domain controller's management console:

Now server-three is not running, so let's click on 'server-three' and then click the 'start' button at the bottom right of the server list. Wait a moment and server-three should start.

Now we should also enable 'server-three' on slave: from the top of the menu list on the left-hand side of the page, we can see that we are currently managing master. Click on the list, select 'slave', then choose 'server-three'.  We are now on the host management page for slave.

Then repeat these steps on master to start 'server-three' on slave.

server-three on master and slave are two different hosts; their names can be different.

After server-three on both master and slave are started, we will add our cluster-demo.war for deployment. Click on the 'Manage Deployments' link at the bottom of left menu list.


We should ensure that server-three is started on both master and slave.

After the enter 'Manage Deployments' page, click 'Add Content' at the top right-hand corner. Then choose the cluster-demo.war, and follow the instructions to add it into our content repository.

Now we can see cluster-demo.war is added. Next we click the 'Add to Groups' button and add the war to 'other-server-group' and then click 'save'.

After a few seconds, the management console will say that the project is deployed into 'other-server-group'.:

Please note that we have two hosts participating in this server group, so the project should be deployed in both master and slave now - that's the power of domain management.

Now let's verify this, trying to access cluster-demo from both master and slave:

Now that we have finished the project deployment and seen the usage of the domain controller, we will then use these two hosts to establish a cluster

Why is the port number 8330 instead of 8080? This has to do with the port-offset property of socket-bindings.  Check host.xml on both master and slave:

The port-offset is set to 250, so 8080 + 250 = 8330

Now we quit the WildFly process on both master and slave. We have some work left to do on the host.xml configurations. Open host.xml on master, and make some modifications to the servers section:

to:

We've set auto-start to true so we don't need to enable it in the management console each time WildFly restarts. Now open slave's host.xml, and modify the server-three section:

Besides setting auto-start to true, we've also renamed the 'server-three' to 'server-three-slave'. We need to do this because mod_cluster will fail to register the hosts with same name in a single server group, due to a name conflict.

After finishing the above configuration, let's restart the two as7 hosts and examine the cluster configuration.

Cluster Configuration

We will use mod_cluster + apache httpd on master as our cluster controller here. WildFly 8 has been configured to support mod_cluster out of box, so it's the easiest way.

The WildFly 8 domain controller and httpd don't have to be on same host. But in this article they are all installed on master for convenience.

First we need to ensure that httpd is installed:

And then we need to download newer version of mod_cluster from its website:

The version I downloaded is:

Jean-Frederic has suggested using mod_cluster 1.2.x, since1.1.x is affected by CVE-2011-4608.

With mod_cluster-1.2.0 you need to add EnableMCPMReceive in the VirtualHost.

Then we extract it into:

Then we edit httpd.conf:

We should add the modules:

Note that we should comment out the proxy_balancer_module:

This is because it will otherwise conflict with the cluster module. And then we need to make httpd listen on the public address so we can perform the testing. Because we installed httpd on master host, we know its IP address:

Then we do the necessary configuration at the bottom of httpd.conf:

For more details on mod_cluster configuration please see this document:

Testing

All being well, we can start the httpd service:

Now we access the cluster:

We should see from the WildFly log that the request is distributed to one of the hosts (master or slave). In this instance, the request is sent to master:

Now I disconnect master by using the management interface. Select 'runtime' and the server 'master' in the upper corners.

Select 'server-three' and click the stop button.  The active icon should change.

Killing the server by using system commands will cause the Host-Controller to restart the instance immediately!

After a few seconds, access the cluster:

Now the request should be served by slave, which should see borne out in slave's log:

And from get.jsp we should see that the time returned is the same gathered by 'put.jsp'. Thus it's proven that the session has correctly replicated to slave.

Now we restart master and should see that the host is registered back to the cluster.

It doesn't matter if you found the request is sent to slave the first time. In that case, just disconnect slave and perform the same test: the request should be sent to master instead. The point is that we should see the request redirected from one host to another and the session is maintained.

Special Thanks

Wolf-Dieter Fink has contributed the updated add-user.sh usages and configs in host.xml from 7.1.0.Final.
Jean-Frederic Clere provided the mod_cluster 1.2.0 usage.
Misty Stanley-Jones has given a lot of suggestions and helps to make this document readable.

Labels:
jboss jboss Delete
cluster cluster Delete
domain domain Delete
clustering clustering Delete
mod_cluster mod_cluster Delete
high-availability high-availability Delete
l l Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Feb 20, 2012

    A couple of things.

    1. Am I correct that the first half of this demo shows session replication, while the other half shows HTTP front-end clustering? If not, what is the second part showing?

    2. I am really confused how you knew the proper way to encode the password for the host.xml file. I tried to read carefully but didn't see this mentioned as a requirement.

    3. Is it possible to do any of this configuration using the Management CLI instead of editing the XML directly? We are trying to discourage people from editing the XML.  

    1. Feb 20, 2012

      1. The demo project itself is a very simple web app that has session replication config enabled in web.xml:

      At first half of the article we've setup two hosts and make them form a domain. master is acting as domain controller+host and slave is a host that under the control of master. We then use the domain controller to deploy the web project with one click on both master and slave (by HTTP management console). So we are showing the functions provided by domain controller at this point: Provides a central point to management the project deployments across all the hosts in domain.

      Then we use mod_cluster module in http to connect with two AS7 hosts to form a cluster. And then we create a scenario by disconnecting a host that is accepting requests to test the failover are working correctly, and sessions are replicated to the other host. Actually this is achieved by Infinispan, JGroup and mod_cluster modules together, which are defined in domain.xml. They are just configured out of box.

      I should explain more details on the clustering in article. I'll add them soon.

      2. It's described in this doc:

      https://docs.jboss.org/author/display/AS7/Admin+Guide#AdminGuide-Secret

      I just use bash command to generate the base64 encoded message. I'll add more details on this one.

      3. Most of the configurations in this article could not be done by using CLI. Because these are the configurations that should be completed before server start(Like IP bindings, etc).

      1. Feb 20, 2012

        Thanks!

  2. Feb 21, 2012

    I was wondering when i start the  slace host controller i get the follow message  

    [Server:malcolm] 20:34:34,179 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) Operation ("add") failed - address: ([("interface" => "unsecure")]) - failure description: "JBAS014763: any-address is required"

    why am i getting this error I am jboss 7.1 final on wondows.  is this saying that my host can not connect to the unsecure interface.  I notice that I also do not see my servers on the domain control panel.

    1. Feb 23, 2012

      Sorry I couldn't reproduce your problem on my machines. Maybe posting host.xml on both your master and slave to https://gist.github.com/ and provide a link here could be more helpful for me to investigate your problem.

  3. Feb 23, 2012

    Hi Weinan;

    Thanks for the cluster guide. It was really helpfull in getting everything set up. However I used a Windows XP pro OS, and the MS Loopback adapter with 3 custom IP's. I then used these as my "servers". All goes well and the clustering from the JBoss 7.1 AS servers works well. However when I try to setup the httpd server I run into problems.

    I first tried a already installed Apache HTTP server with the mod_x.so mentioned just copied from the mod_cluster zip file, and the config you mentioned. No dice. Then I tried both the 1.20 mod cluster httpd and the 1.1.3 without any success. I can get the default "IT WORKS" page, but it seems not to  forward to the JBoss AS 7.1 when trying to access cluster_demo.

    Here are the questions I have. If you are not running the httpd on the same server as the master jboss, what is the ip supposed to be for the following:

    Listen <?>:6666

    <VirtualHost <?>:6666>

    Allow from <?>

    1. Feb 23, 2012

      Hi Gerhard,

      I'm a little bit confused about your problem. You said the mod_cluster doesn't work in http. But then you said you can get the default page when accessing cluster url. Could you please tell me more details on your problem? 

      For the listening address configurations of httpd, you should set the Listen to the IP address of your host which httpd is running on, and please ensure the IP address are in same local network with your AS7 hosts.

      VirtualHost address should be same with the Listen address, which is also the IP address that your httpd is running on.

      Allow from should be set to the subnet address that all your machines are in(Your httpd and AS7 hosts are all in same subnet right? Please ensure that's true).

      1. Mar 01, 2012

        The Apache HTTP zip comes with a page in the htdocs directory with one index.html file that just reads "IT WORKS!" So when I go to http://localhost/ thats what I get. However when I try to go to http://localhost/cluster-demo/put.jsp  I just get a 404. So what I am seeing is that its trying to get cluster-demo/put.jsp from the htdocs directory. So the mod_cluster on its own does nothing. Do we need to add a entry to forward all requests of a certain type to the server?

        1. Mar 03, 2012

          Sounds like your 'other-server-group' in AS7 is not started. Please login into AS7 management console and ensure 'other-server-group' is running.

  4. Feb 25, 2012

    Hi Weinan

    After struggling with my own httpd setup I finally removed and installed mod_cluster-1.2.0.Final-linux2-x64-ssl.tar.gz

    That solved a lot of issues, I now see:

    127.0.0.1 - - [26/Feb/2012:10:27:38 +0700] "STATUS / HTTP/1.1" 200 85
    127.0.0.1 - - [26/Feb/2012:10:27:48 +0700] "STATUS / HTTP/1.1" 200 85
    127.0.0.1 - - [26/Feb/2012:10:27:58 +0700] "STATUS / HTTP/1.1" 200 85
    127.0.0.1 - - [26/Feb/2012:10:28:08 +0700] "STATUS / HTTP/1.1" 200 85

    in the httpd access_log

    I see the welcome on port 8000 (httpd has Listen 8000)

    httpd.conf has the following directives:

    <IfModule ssl_module>
    SSLRandomSeed startup builtin
    SSLRandomSeed connect builtin
    </IfModule>
    # MOD_CLUSTER_ADDS
    # Adjust to you hostname and subnet.
    <IfModule manager_module>
      Listen 127.0.0.1:6666
      ManagerBalancerName mycluster
      <VirtualHost 127.0.0.1:6666>
        <Location />
         Order deny,allow
         Deny from all
         Allow from all
        </Location>

        KeepAliveTimeout 300
        MaxKeepAliveRequests 0
        #ServerAdvertise on http://@IP@:6666
        AdvertiseFrequency 5
        #AdvertiseSecurityKey secret
        #AdvertiseGroup  (ADVIP):23364
        EnableMCPMReceive

        <Location /mod_cluster_manager>
           SetHandler mod_cluster-manager
           Order deny,allow
           Deny from all
           Allow from 127.0.0
        </Location>

      </VirtualHost>
    </IfModule>

    My last question is, how do I get to the web service from my front end

    The front end is a c sharp application that connect with:

    <client>
          <endpoint address="http://192.168.2.3:8082/beans/XMLTranServiceBean/XMLTranServiceBean"
              binding="basicHttpBinding" bindingConfiguration="XMLTranServiceBeanSoapBinding"
              contract="CapitalWebService.XMLTranServiceBean" name="XMLTranServiceBeanPort" />
    </client>

    The above example works because it goes straight to JBoss http port (as defined in domain.xml).

    The question is: how do I play this into the cluster via httpd ?

    To put it simply, I am looking for the mod_cluster equivalent of the mod_jk config file workers.properties

    It had this directive inside: /STARXMLTranServiceBean/STAR=loadbalancer (STAR = * which seems to be meaningful with this editor)

    That fished out my front end transactions in httpd before, any URL with the above strings inside would go to JBoss 4 via Tomcat 8009

    How does that work with mod_cluster ?

    Thanks and best regards

    Johan Vermeij

    Jakarta

    1. Feb 26, 2012

      Hi Johan,

      After two AS7 hosts are setup they can be accessed directly by their own IPs. And session replication is working with the help of JGroup and Infinispan configured out-of-box in AS7 domain.xml. But the cluster is useless if we could not have a 'single entry point' for user to access(We don't want the users to know each AS7 node's IP address).

      So we need mod_cluster+httpd to provide an address that user could use for accessing the cluster: httpd will provide a listening IP address that user can access, and mod_cluster is used to distribute the requests to one of the node in cluster(depending on each node's load, such as CPU load, memory usage, request numbers, etc.)

      I guess you have gotten the idea now :-) So the endpoint address used in your web service should not be one of the AS7 host's ip address. It should be http://<your_httpd_listening_address>:8000 (Because you said your httpd is listening on port 8000).

      In my example I've put httpd on the same machine with one of the AS7 host, so the IP address sames like the same with master AS7 ip address. This may create some confusions :-) It could be more clear if I use three machines to show the demo then you could see the httpd IP address is standalone.

      Actually we don't have to use mod_cluster to do the request distribution, we could use LVS, or even hardware devices. But with mod_cluster we could get some nice features out of box such as nodes auto-discovery, recovered host auto registering back, etc. Because both mod_cluster and AS7 are developed by JBoss dev team and they are designed to be integrated with each other nicely :-)

      So let's go back to your httpd config, you should set it to listen to a public ip address instead of 127.0.0.1, and you could access your cluster by http://<your_httpd_listening_address>:<your_httpd_listening_port>

      Hope these could help.

      1. Mar 01, 2012

        Hi Weinan

        I am sorry to say but I still cannot get anything into the cluster (but I can send client transactions to both master and slave on port 8082 as defined in domain.xml on the master. The slave does not have a domain.xml as per instruction. It reads domain.xml from the server using port 9999).

        I have installed httpd on a third machine now.

        The warning below is something I never got before:

        [Thu Mar 01 19:52:04 2012] [notice] Child 3464: Child process is running
        [Thu Mar 01 19:52:04 2012] [notice] Child 3464: Acquired the start mutex.
        [Thu Mar 01 19:52:04 2012] [notice] Child 3464: Starting 64 worker threads.
        [Thu Mar 01 19:52:04 2012] [notice] Child 3464: Starting thread to listen on port 6666.
        [Thu Mar 01 19:52:04 2012] [notice] Child 3464: Starting thread to listen on port 8000.
        [Thu Mar 01 19:52:09 2012] [warn] manager_handler STATUS error: MEM: Can't read node

        That looks a lot like the good old node under mod_jk load balancing.

        Does this mean anything to you ?

        I listen on port 8000, this is the first time that httpd tries to access the cluster, before I always went straight to DocumentRoot.

        My end point is a web service under JBoss 7 AS

        Thanks and best regards

        Johan

        1. Mar 01, 2012

          The above warning only came out once.

          Now I am back to good old "no end point found".

          This is from the Apache httpd log

          192.168.2.2 - - [01/Mar/2012:20:07:04 +0700] "POST /beans/XMLTranServiceBean/XMLTranServiceBean HTTP/1.1" 404 1088
          192.168.2.4 - - [01/Mar/2012:20:07:10 +0700] "STATUS / HTTP/1.1" 200 85
          192.168.2.3 - - [01/Mar/2012:20:07:12 +0700] "STATUS / HTTP/1.1" 200 85
          192.168.2.4 - - [01/Mar/2012:20:07:20 +0700] "STATUS / HTTP/1.1" 200 85
          192.168.2.3 - - [01/Mar/2012:20:07:22 +0700] "STATUS / HTTP/1.1" 200 85
          192.168.2.4 - - [01/Mar/2012:20:07:30 +0700] "STATUS / HTTP/1.1" 200 85
          192.168.2.3 - - [01/Mar/2012:20:07:32 +0700] "STATUS / HTTP/1.1" 200 85

          We are now working with 2 people on this, both are very experienced IT professionals.

          We have a feeling that we are missing a small detail somewhere.

          How does httpd know which port to use when it sends messages to JBoss 7 AS ?

          Or is that automatically discovered ?

  5. Mar 01, 2012

    I made some changes to the article, to simplify exactly what needs to be configured. Tomorrow I will make some more changes to keep things secure (exactly what needs to be changed on the firewall and selinux configs).

  6. Mar 02, 2012

    Hi Weinan,

    I am having similar issue faced by Gerhard Visagie.

    Able to view "It Works" page when http://172.18.140.91:666 is specified. However when we say http://172.18.140.91/cluster-demo/put.jsp getting page not found error.

    Following error is logged in AS7 :

    "18:05:31,228 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [null: null:

    Unknown macro: {4}

    ] sending command INFO to proxy mach.cmcltd.com/172.18.140.91:6666, configuration will be reset"

    Following error is logged in Apache:

    [Fri Mar 02 18:07:11 2012] [error] [client 172.18.140.91] Invalid method in request INFO / HTTP/1.1

    We are using JBOSS AS 7.1.0 final, Apache 2.2.22 and mod_cluster 1.2.0

    And the configuration in httpd.conf :

    <VirtualHost 172.18.140.91:6666>
     
      <Directory />
        Order deny,allow
        Deny from all
        Allow from all
      </Directory>
     
      KeepAliveTimeout 60
      MaxKeepAliveRequests 0
     
      ManagerBalancerName other-server-group
      AdvertiseFrequency 5
      EnableMCPMReceive
    </VirtualHost>

    1. Mar 02, 2012

      The question is: how does httpd know what goes to DocumentRoot and what goes to the JBoss cluster.

      This (I could be wrong) needs some filter mechanism, something has to be fished out of the URL.

      As I asked before:

      -----------------------------------------

      To put it simply, I am looking for the mod_cluster equivalent of the mod_jk config file workers.properties

      It had this directive inside: /STARXMLTranServiceBean/STAR=loadbalancer (STAR = * which seems to be meaningful with this editor)

      That fished out my front end transactions in httpd before, any URL with the above strings inside would go to JBoss 4 via Tomcat 8009

      How does that work with mod_cluster ?

      ------------------------------------------

      In my view nothing can be intelligent enough to decide what goes where.

      httpd may be serving many web sites, subversion interfave, php interface etc

      1. Mar 03, 2012

        I don't hear much.

        Either everybody is up and running with mod_cluster load balancing or ...

        Anyway, I just try things:

        I changed domain.xml (the XMLTranServiceBean is in the URL that I send to the webservice):

                    <subsystem xmlns="urn:jboss:domain:webservices:1.1">
                        <modify-wsdl-address>true</modify-wsdl-address>
                        <wsdl-host>$

        Unknown macro: {jboss.bind.address}

        </wsdl-host>
                        <endpoint-config name="Standard-Endpoint-Config"/>
                        <endpoint-config name="Recording-Endpoint-Config">
                            <pre-handler-chain name="recording-handlers" protocol-bindings="/XMLTranServiceBean/">
                                <handler name="RecordingHandler" />
                            </pre-handler-chain>
                        </endpoint-config>
                    </subsystem>

        This gave me the following error in the Apache httpd error log:

        [Sat Mar 03 12:13:30 2012] [error] [client 192.168.2.2] (OS 10054)An existing connection was forcibly closed by the remote host.  : proxy: error reading status line from remote server 192.168.2.4:8332
        [Sat Mar 03 12:13:30 2012] [error] [client 192.168.2.2] proxy: Error reading from remote server returned by /beans/XMLTranServiceBean/XMLTranServiceBean

        And I see the following in the server-three-slave log:

        20:28:04,243 WARNING [org.jgroups.protocols.UDP] (OOB-20,null) null: no physical address for 2864c345-49ad-670b-dc4a-083fd7156c5a, dropping message
        20:28:18,037 INFO  [org.apache.coyote.http11.Http11AprProtocol] (MSC service thread 1-4) Pausing Coyote HTTP/1.1 on http--192.168.2.4-8332
        20:28:18,049 INFO  [org.apache.coyote.http11.Http11AprProtocol] (MSC service thread 1-4) Stopping Coyote HTTP/1.1 on http--192.168.2.4-8332
        20:28:18,335 INFO  [org.jboss.as.osgi] (MSC service thread 1-4) JBAS011942: Stopping OSGi Framework
        20:28:21,222 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-3) JBAS010410: Unbound JCA ConnectionFactory [java:/JmsXA]
        20:28:20,479 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-2) All active sessions drained from context [/wssimple] in 0 seconds
        20:28:25,082 INFO  [org.jboss.as.webservices] (MSC service thread 1-3) JBAS015540: Stopping service jboss.ws.port-component-link
        20:28:29,103 ERROR [org.apache.catalina.core.ContainerBase] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Exception invoking periodic operation: : java.lang.IllegalStateException
                at org.jboss.modcluster.container.catalina.CatalinaEngine.getProxyConnector(CatalinaEngine.java:153)
                at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:464)
                at org.jboss.modcluster.container.catalina.CatalinaEventHandlerAdapter.lifecycleEvent(CatalinaEventHandlerAdapter.java:239)
                at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.0.10.Final.jar:]
                at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337) [jbossweb-7.0.10.Final.jar:]
                at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) [jbossweb-7.0.10.Final.jar:]
                at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) [jbossweb-7.0.10.Final.jar:]
                at java.lang.Thread.run(Thread.java:636) [rt.jar:1.6.0_20]

        So there is a pass through mechanism, same as in mod_jk.
        No software can be so intelligent that it can decide what goes where, that is for sure.
        And Apache httpd can do a lot of things at once.

        I only get this error on the slave and I only get it once, I have to restart the slave server to reproduce the error.

        Who knows what the correct parameters are ?

        Thanks

        Johan

        1. Mar 03, 2012

          I'm not so familiar with the WebServices configuration in domain.xml (I generally use JAX-RS which has configured out-of-box by AS7), that may need some extra configurations for it to work in cluster environment.

    2. Mar 03, 2012

      Hi Shyam,

      Similar with what I've replied to Gerhard, sounds like your 'other-server-group' in AS7 is not started. Please login into AS7 management console and ensure 'other-server-group' is running.

      In addition, the 6666 port is the management port of mod_cluster. If you've seen 'it works', then there's something wrong. You can try to add following config into virtual-host:

      After restarting the HTTP server, try to access 6666 port:

      You should see the management console of mod_cluster:

      1. Jun 19, 2012

        Hi [Weinan Li||\||]

        Now i have tried clustering for standalone application with AS7 and Apache 2.2. First i

        will explain what i did.

        First i run the standalone server on ip:192.168.111.19:8231

        And following is the changes what i did in httpd.conf file

        Now following is the code which i changed in httpd-vhosts.conf file

        Now when i start the server i cant get the advertise message and i cant get my application

        from following i.p:192.168.111.19/Demo/test.jsp. When i checked through :list-proxies command

        in jboss-cli.bat file then it shows no proxy. Then i add the proxy through the following command

        Proxy was added successfully. Then i can get the application from

        192.168.111.19/Demo/test.jsp. Now when i monitor the mod-cluster-manager in my browser

        then i got the following screen.

        So following are the problems:

        I cant get specific node name. So how can i get?

        Now in jboss-cli.bat file if i tried for :read-proxies-configuration command i get

        following output:

        So here i cant get JVMRoute specifically. So what should i do?

        And i cant get specific clustering.

        And can you say me how clustering and load balancing is done in standalone mode.

        Thanks in advance

  7. Mar 05, 2012

    We have put almost 3 weeks into getting the new load balancing working with JBoss AS 7 and mod_cluster. We = two very experienced professional IT people with a shared experience span of around 60 years.

    I am very grateful to the community members who have put lots of effort into the new JBoss clustering.

    These are my observations:

    1) Setting up clustering under Unix / Linux is a breeze. I set it up under 30 minutes on Mac OS X yesterday (that made me very happy, amazing software). I also had two instances running under Centos 6.

    2) Setting up the same, using the same domain.xml and host.xml files from Unix/Linux, under Windows 7 is very cumbersome. Despite all our efforts we could not make it work

    3) mod_jk load balancing works using a httpd instance and 2 (or more) JBoss standalone (not domain) instances

    4) We cannot get anything into the cluster using mod_cluster via httpd. We can send transactions to all cluster instances but we cannot send anything to the actual cluster. And for intelligent load balancing we have to be able to send transactions to the cluster, not to cluster members. Otherwise you get the same inflexible load balancing as under mod_jk where you have to guess how busy a server is going to be.

  8. Mar 06, 2012

    I can get to the point where I can see the mod_cluster-manager, but I can't access my deployed app via Apache, only directly via JBoss Web. I think there is some subsystem configuration or something that is missing, that tells Apache where to look for the deployment in the AS server. I'll look into it and post back here when I find it.

    1. Mar 06, 2012

      What a difference a day can make.

      Today we managed to bring up a cluster under Windows 7 and we managed to get transactions into the cluster using port 8000.

      I have Apache httpd running on a Windows XP box and 2 JBoss instances on a Linux server and an iMac.

      Please see below for the httpd VirtualHost config (192.168.2.2 is the XP box, no need to make reference to any cluster members by IP address, that is automatic. Please note the URL used by ProxyPass (as per Jean Frederic's suggestion).

      Having different logs per VirtualHost is also nice, it makes things more readable.

      -------------------

      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
      LoadModule slotmem_module modules/mod_slotmem.so
      LoadModule advertise_module modules/mod_advertise.so
      LoadModule manager_module modules/mod_manager.so

      # MOD_CLUSTER_ADDS
      # Adjust to you hostname and subnet.
      <IfModule manager_module>
        Listen 192.168.2.2:6666
        ManagerBalancerName other-server-group
        <VirtualHost 192.168.2.2:6666>
          <Location />
           Order deny,allow
           Deny from all
           Allow from all
          </Location>

          ErrorLog "logs/chatter_log"
          LogLevel debug
          KeepAliveTimeout 300
          MaxKeepAliveRequests 0
          #ServerAdvertise on http://127.0.0.1:6666
          AdvertiseFrequency 30
          #AdvertiseSecurityKey secret
          #AdvertiseGroup 224.0.1.105:23364
          EnableMCPMReceive

      #    <Location /mod_cluster_manager>
      #       SetHandler mod_cluster-manager
      #       Order deny,allow
      #       Deny from all
      #       Allow from all
      #    </Location>

        </VirtualHost>
      </IfModule>
      NameVirtualHost 192.168.2.2:8000

      <VirtualHost 192.168.2.2:8000>
          ServerAdmin johanvermeij@gmail.com
          ServerName 192.168.2.2
          ErrorLog "logs/cluster_log"
          LogLevel debug
      #    ServerAlias 192.168.1.10

          ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On
          ProxyPassReverse / balancer://other-server-group
          ProxyPreserveHost On

          <Location />
              Order deny,allow
              Allow from All
          </Location>

          <Location /mod_cluster_manager>
              SetHandler mod_cluster-manager
              Order deny,allow
              Deny from all
              Allow from all
          </Location>
      </VirtualHost>

  9. Mar 07, 2012

    Session replication doesn't seem to work for me. I followed the steps in this guide, but when I kill server three in master, the process also destroys the session that has been made. So when I run the get.jsp in slave, it shows that time is null (it didn't get any session).

    Is there something else that I should configure?

    Thanks!

    1. Mar 09, 2012

      Can you write down what you do, step by step.

      We do the following:

      1. set auto-start for server-three to false in host.xml on master
      2. run domain.sh in master
      3. start server-three using admin console on port 9990
      4. deploy the application through the console (deploy to ha only)
      5. set auto-start for server-three-slave to false in host.xml on slave
      6. remove or rename domain.sh in slave
      7. run domain.sh in salve
      8. start server-three-slave using admin console (this will trigger deployment to the slave)
      9. see if you can access the http port on master and slave (as per http port defined in domain.xml)
      10. see if you can access the cluster using httpd (port 8000 as per above example, up to you which port to use)

      What do you do next ?

      If you let me know I will simulate it and send you the results

      1. Apr 08, 2013

        hi,johan!

        remove or rename domain.sh in slave 
        run domain.sh in salve  ??    the domain.sh already remove or rename!  should be domain.xml !!  
        But i have a quastion, my operating system is XP, if remove or rename domain.xml, The call domain.bat enter cmd command, 
        The console will display an error message! 
        Because domian.bat the reference domain.conf.bat and edit domain.conf.bat,
        Will find set "JAVA_OPTS =% JAVA_OPTS%-Djboss.domain.default.config = domain.xml-Djboss.host.default.config = host.xml" This line of command.
        I removed-Djboss.domain.default.config =  domain.xml this command, perform call domain.bat, console or display an error message.
        I do not know how to go next step, I would like to ask that you do not encounter this problem? Hope to get your answer, thank you!

  10. Mar 14, 2012

    hi! how do you configure JBoss AS 7.1.1 for clustering? I did the same steps here, but I cannot even get cluster-demo to deploy to all nodes. Also, my httpd cannot detect the nodes.

    1. Mar 15, 2012

      If you see a lot of infinispan errors (your deployment "hangs") do the following:

      1) Download the latest infinispan libraries and put them in \jboss-as-7.1.0.Final\modules\org\infinispan

      2) Locate in domain.xml:

      <subsystem xmlns="urn:jboss:domain:infinispan:1.1" default-cache-container="hibernate">
                      <cache-container name="hibernate" default-cache="local-query">
                          <local-cache name="entity">
                              <transaction mode="NON_XA"/>
                              <eviction strategy="LRU" max-entries="10000"/>
                              <expiration max-idle="100000"/>
                          </local-cache>
                          <local-cache name="local-query">
                              <transaction mode="NONE"/>
                              <eviction strategy="LRU" max-entries="10000"/>
                              <expiration max-idle="100000"/>
                          </local-cache>
                          <local-cache name="timestamps">
                              <transaction mode="NONE"/>
                              <eviction strategy="NONE"/>
                          </local-cache>
                      </cache-container>
                  </subsystem>

      Change <eviction strategy="NONE"/> under "timestamps" to <eviction strategy="NONE" max-entries="0" />

      That was the finishing touch for us, the cluster was established after we changed that.

      Remember that only HA (High Availability) is clustered so only deploy to HA

      As always, the problems are not big but tricky.

      The max-entries one is crucial.

      1. Mar 15, 2012

        thanks, i will try that! is it also required to add the AJP connector and socket-binding for ha-sockets in the domain.xml?

        1. Mar 15, 2012

          We didn't use the AJP connector, HTTP only

          I attach the master and slave config files (only domain.xml (master, shared with slave) and host.xml (master and slave) matter)

          configuration_master_and_slave.rar

          1. Mar 15, 2012

            hi. i get this error message when i don't use the AJP connector:

            Proxy Error

            The proxy server received an invalid response from an upstream server.
            The proxy server could not handle the request GET /cluster-demo.

            Reason: Error reading from remote server

            1. Mar 16, 2012

              i'm still encountering a lot of errors when deploying large web applications in the cluster, i've been experiencing the following errors

              over and over again:

              1.    JBAS010806: Caught exception closing input stream: java.io.IOException: Stream closed

              2.    ISPN000097: Error while processing 1PC PrepareCommand: org.infinispan.CacheException: java.lang.RuntimeException:

                     Failure to marshal argument(s)

              3.    org.infinispan.CacheException: Could not commit.

              4.    java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException:

                     timeout sending message to master:server-three/web

              5.    ISPN000172: Failed to prepare view CacheView

  11. Mar 27, 2012

    This is an excellent how to guide for a starter like me on Jboss..

    However I have a question from above howto..In Apache Config, May this has been answered.. But I haven;t understood.. Sorry guys am a starter on Jboss

    I set a similar 2 node cluster but a small differenc ei I had setup apache on my two nodes.. so that I can do a  Lload Blancing using Pound.

    No my question is I followed instructions as it is from the doc. and went to do the same

    I 'am listening on on 10001 in my apache config

    http:<hostname>:10001/mod_cluster-manager  works fine :)

    http:<hostname>:10001/cluster-demo works fine

    however if I go 

    http:<hostname>/cluster-demo doesn't work as it shows in the demo and screen shots

    is there something am doing wrong ?

    thi sis what am using my httpd.conf and 

    Listen 10.36.200.10:10001
    MemManagerFile /apps/cache/httpd
    <VirtualHost 10.36.200.10:10001>
    <Directory />
    Order deny,allow
    Deny from none
    #Allow from 10.36.
    </Directory>
    ServerAdvertise On
    EnableMCPMReceive
    KeepAliveTimeout 60
    MaxKeepAliveRequests 0
    ManagerBalancerName mycluster
    AdvertiseFrequency 5
    </VirtualHost>
    <Location /mod_cluster-manager>
    Listen 10.36.200.16:10001

    MemManagerFile /apps/cache/httpd

    <VirtualHost  10.36.200.16 :10001>

    <Directory />

    Order deny,allow

    Deny from none

    </Directory>

    EnableMCPMReceive

    KeepAliveTimeout 60

    MaxKeepAliveRequests 0

    ManagerBalancerName mycluster

    AdvertiseFrequency 5

    </VirtualHost>

    <Location /mod_cluster-manager>

    am I doing anything wrong ?

    S

  12. Mar 27, 2012

    These instructions work for jboss-as-7.1.0.Final "Thunder" but not for jboss-as-7.1.1.Final "Brontes".  On 7.1.1, starting server-three fails with "jboss.binding.jacorb (missing) dependents" as in https://community.jboss.org/thread/197320

    1. Mar 29, 2012

      Try using full-ha-sockets in other-server-group.

  13. Mar 28, 2012

    Everything works fine, except the last step - http://<ip_address>cluster-demo/get.jsp :-( .  I even read the above, and I saw few people having similar problems. I have checked that other-group server instance is up.

    I was also able to access mod_cluster-manager.

    Could you explain how does it know that it has to connect to JBoss ? probably I am missing something very basic here !

    http://<your_ip_address>:<port>/mod_cluster-manager
    http://<ip_address>cluster-demo/get.jsp

  14. Apr 08, 2012

    Finally I got this working.

    mod_cluster version - 1.2.0.FINAL

    JBoss AS7.1.1 FINAL.

    There were couple of changes I had to do in domain.xml.

    1.) Had to add in web subsystem

    Update :A known issue : https://issues.jboss.org/browse/JBPAPP-8451

    2.) proxy-list="<ip address>:<portno.> " under <subsystem xmlns="urn:jboss:domain:modcluster:1.0"> in <mod-cluster-config advertise-socket="modcluster" proxy-list="IP:Port">

    So I guess the change 2 is required because due to some reason JBoss does not seem to work out of the box. Looks like an issue there with 7.1.1.

    Without (1) there were errors in JBoss AS7 server instances like - manager_handler CONFIG error: MEM: Old node still exist

    1. Mar 29, 2012

      thanks! finally made it work!

  15. Apr 08, 2012

    when i start server-three on master is working fine, but when i start server-three on 'slave' getting the following error

    [Server:server-three] 07:13:38,903 ERROR [org.hornetq.core.protocol.core.impl.Ho
    rnetQPacketHandler] (Old I/O server worker (parentId: 594465015, [id: 0x236ed0f7
    , hostname/192.168.1.2:5695])) Failed to create session : HornetQException[er
    rorCode=105 message=Unable to validate user: HORNETQ.CLUSTER.ADMIN.USER

    Please find the attached file which i have configured

    1. Apr 08, 2012

      Try the below link...it has the solution and why this problem occurs.

      https://community.jboss.org/thread/195452

      1. Apr 08, 2012

        Thanks for your valuable response.. now i am getting

        [Server:server-three] 20:53:05,768 ERROR [org.jboss.modcluster.mcmp.impl.Default
        MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [ME
        M: MEM: Can't read node:

        Unknown macro: {4}

        ] sending command STATUS to proxy 192.16
        8.1.2:6666, configuration will be reset

        1. Apr 08, 2012

          Could you paste some more of the stack trace? are u running all of your servers in master and slave on the same machine ?

          1. Apr 08, 2012

            Both are running in different machines but on same network. master at 192.168.1.2 and slave at 192.168.1.3. default gateway is

            192.168.1.1.,   http server running at 192.168.1.2 (on master machine )

            STACK TRACE:

            [Server:server-three] 21:11:18,247 ERROR [org.jboss.modcluster.mcmp.impl.Default
            MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [ME
            M: MEM: Old node still exist:

            Unknown macro: {4}

            ] sending command CONFIG to proxy DivyaArvind/1
            92.168.1.2:6666, configuration will be reset
            [Server:server-three] 21:11:38,268 ERROR [org.jboss.modcluster.mcmp.impl.Default
            MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [ME
            M: MEM: Can't read node:

            ] sending command STATUS to proxy DivyaArvind/192.16
            8.1.2:6666, configuration will be reset
            [Server:server-three] 21:11:48,278 ERROR [org.jboss.modcluster.mcmp.impl.Default
            MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [ME
            M: MEM: Old node still exist:

            Unknown macro: {4}

            ] sending command CONFIG to proxy DivyaArvind/1
            92.168.1.2:6666, configuration will be reset
            [Server:server-three] 21:12:08,298 ERROR [org.jboss.modcluster.mcmp.impl.Default
            MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [ME
            M: MEM: Can't read node:

            ] sending command STATUS to proxy DivyaArvind/192.16
            8.1.2:6666, configuration will be reset
            [Server:server-three] 21:12:18,308 ERROR [org.jboss.modcluster.mcmp.impl.Default
            MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [ME
            M: MEM: Old node still exist:

            Unknown macro: {4}

            ] sending command CONFIG to proxy DivyaArvind/1
            92.168.1.2:6666, configuration will be reset
            [Server:server-three] 21:12:38,329 ERROR [org.jboss.modcluster.mcmp.impl.Default
            MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [ME
            M: MEM: Can't read node:

            ] sending command STATUS to proxy DivyaArvind/192.16
            8.1.2:6666, configuration will be reset
            [Server:server-three] 21:12:48,340 ERROR [org.jboss.modcluster.mcmp.impl.Default
            MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [ME
            M: MEM: Old node still exist:

            Unknown macro: {4}

            ] sending command CONFIG to proxy DivyaArvind/1
            92.168.1.2:6666, configuration will be reset

            *httpd.conf file:*

            Listen 192.168.1.2:6666

            <VirtualHost 192.168.1.2:6666>
            <Location />
            Order deny,allow

            Allow from all
            </Location>

             # This directive allows you to view mod_cluster status at URL http://192.168.1.2:6666/mod_cluster-manager
              <Location /mod_cluster-manager>
               SetHandler mod_cluster-manager
               Order deny,allow
               Allow from 192.168.1.
              </Location>

            KeepAliveTimeout 60
            MaxKeepAliveRequests 0
            ManagerBalancerName other-server-group
            AdvertiseFrequency 5
            EnableMCPMReceive
            </VirtualHost>

            1. Apr 08, 2012

              I feel the port of server three is overlapping with a different server, can u try by giving a different offset in host.xml ?

              1. Apr 09, 2012

                i changed the port but getting the same error. please find attached file which contains config details

                1. Apr 08, 2012

                  From your config files I see you have the following :-

                  1.)  Master 

                  server- one

                  server-two -> offset -150

                  server-three -> offset  ->250

                  2.) Slave

                  server- one

                  server-two -> offset -150

                  server-three -> offset  ->250

                  So are you starting 6 servers in total (3 in each machine ? ) ? To debug start each of the servers one after another (disable auto start =true in domain.xml host.xml of master and slave) and find out on start of which of the server causes this problem to be showing up in console continously? 

                  Another simple test you can try if it is indeed caused by port - change  offset for server two and server three in Master to 151 and 251 and in slave as 101 and 201 (or some odd numbers) (This is just a guess )

                  1. Apr 08, 2012

                    I followed what you said. first i started server three of master (it went fine, no errors) but when i started server three of

                    slave, getting error continuously

                    [Server:server-three] 22:42:03,808 ERROR [org.jboss.modcluster.mcmp.impl.Default
                    MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [ME
                    M: MEM: Old node still exist:

                    Unknown macro: {4}

                    ] sending command CONFIG to proxy DivyaArvind/1
                    92.168.1.2:6666, configuration will be reset

                    Even i tried changing port , but no use

                    1. Apr 08, 2012

                      Ok, the ports are not the cause, this is an issue with AS7. To get this working you would need to add an instance id in web subsystem. you cannot hard-code this to master, instead you need to have:-

                      1. Apr 08, 2012

                        thanks now the error has gone.... but nothing display when i test the deployed demo on browser.

                        Please help me, Do i need to change/add anything on http server CONFIG?

                        I am able to access mod_cluster-manager.

                        1. Apr 08, 2012

                          How are you testing it ? http://192.168.1.2:6666/cluster-demo

                          1. Apr 08, 2012

                            finally i got everything ....

                            1) I made a call to put.jsp, it was directed to master machine and i stopped server three of

                                master. made a call to get.jsp and it was directed to slave machine. everything fine.

                            2) Brought back server three of master, now again i made a call to put.jsp and it was

                                failed 'service temporary unavailable'

                            3) the error on console when i brought back 'master'  

                            [Server:server-three] 23:46:22,379 WARN  [org.hornetq.core.cluster.impl.Di
                            scoveryGroupImpl] (hornetq-discovery-group-thread-dg-group1) There are more than
                             one servers on the network broadcasting the same node id. You will see this mes
                            sage exactly once (per node) if a node is restarted, in which case it can be saf
                            ely ignored. But if it is logged continuously it means you really do have more t
                            han one node on the same network active concurrently with the same node id. This
                             could occur if you have a backup node active at the same time as its live node.
                             nodeID=6498e4cb-8061-11e1-a0d6-24b6fd012f2a  

                            4) Error logs in HTTP server:

                                [Sun Apr 08 23:46:59 2012] [error] proxy: CLUSTER: (balancer://other-server-group). All workers are in error state

                             

                            1. Apr 08, 2012

                              Point 3 is not an error if you get it only once (during restart). I dont' have an idea of why you get this issue.

                              Some points you may want to check :-

                              1.) Hope you have <distributable/> in your web.xml.

                              2.) Any errors when you brought up the master , server three (related to cache) , also any errors in slave - master three.

                              Dont have a setup, I'll check this example again and get back to you later. 

                              1. Apr 08, 2012

                                message from management console. i sent you an email please check it. 

                                i was not able to attach a file here.

                                Thanks for your co-operation

  16. Apr 17, 2012

    OK, so I decided to try using linux instead of windows XP. I used Virtualbox with two Ubuntu server installs. I used JBoss AS 7.1 and jdk 7.0.3. After configuration of the cluster I deploy the "cluster-demo.war". The deploy screen then gets stuck in "Please wait: Your request is being processed ...".

    In the log I get this printout for master:

    [Server:server-three] 10:02:53,654 INFO  [org.jboss.as.server] (host-controller-connection-threads - 2) JBAS015870: Deploy of deployment "cluster-demo.war" was rolled back with failure message JBAS014750: Operation handler failed to complete
    [Server:server-three] 10:02:53,664 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010282: Stopped //default-host//cluster-demo cache from web container
    [Server:server-three] 10:02:53,778 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-14-thread-1) ISPN000029: Passivating all entries to disk
    [Server:server-three] 10:02:53,779 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-14-thread-1) ISPN000030: Passivated 0 entries in 0 milliseconds
    [Server:server-three] 10:02:53,783 INFO  [org.jboss.as.clustering.infinispan] (pool-14-thread-1) JBAS010282: Stopped repl cache from web container
    [Server:server-three] 10:02:53,794 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment cluster-demo.war in 148ms
    [Server:server-three] 10:02:53,998 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-12-thread-1) ISPN000082: Stopping the RpcDispatcher
    [Server:server-three] 10:02:54,119 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-7,null) ISPN000094: Received new cluster view: [master:server-three/web|2] [master:server-three/web]
    [Host Controller] 10:02:54,129 ERROR [org.jboss.as.controller.management-operation] (HttpManagementService-threads - 1) JBAS014612: Operation ("composite") failed - address: ([]): java.lang.IllegalArgumentException
    [Host Controller]       at org.jboss.dmr.ModelValue.getChild(ModelValue.java:108) [jboss-dmr-1.1.1.Final.jar:1.1.1.Final]
    [Host Controller]       at org.jboss.dmr.ModelNode.get(ModelNode.java:798) [jboss-dmr-1.1.1.Final.jar:1.1.1.Final]
    [Host Controller]       at org.jboss.as.domain.controller.operations.coordination.DomainFinalResultHandler.collectHostFailures(DomainFinalResultHandler.java:158) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.domain.controller.operations.coordination.DomainFinalResultHandler.execute(DomainFinalResultHandler.java:80) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:385) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:272) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:200) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.executeTwoPhaseOperation(OperationCoordinatorStepHandler.java:225) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.execute(OperationCoordinatorStepHandler.java:118) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.domain.controller.operations.coordination.PrepareStepHandler.execute(PrepareStepHandler.java:85) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:385) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:272) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:200) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:121) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelControllerImpl.java:309) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelControllerImpl.java:299) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.domain.http.server.DomainApiHandler.processRequest(DomainApiHandler.java:294) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.domain.http.server.DomainApiHandler.doHandle(DomainApiHandler.java:201) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.domain.http.server.DomainApiHandler.handle(DomainApiHandler.java:208) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.as.domain.http.server.security.SubjectAssociationHandler.handle(SubjectAssociationHandler.java:51) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)
    [Host Controller]       at org.jboss.sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:69)
    [Host Controller]       at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
    [Host Controller]       at org.jboss.sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:710)
    [Host Controller]       at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)
    [Host Controller]       at org.jboss.as.domain.http.server.RealmReadinessFilter.doFilter(RealmReadinessFilter.java:54) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]
    [Host Controller]       at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
    [Host Controller]       at org.jboss.sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:682)
    [Host Controller]       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_03]
    [Host Controller]       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_03]
    [Host Controller]       at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_03]
    [Host Controller]       at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
    [Host Controller]

    On the slave I see this:

    [Server:server-three-slave] 10:02:53,943 INFO  [org.jboss.as.server] (host-controller-connection-threads - 1) JBAS015870: Deploy of deployment "cluster-demo.war" was rolled back with failure message JBAS014750: Operation handler failed to complete
    [Server:server-three-slave] 10:02:53,957 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-1) ISPN000029: Passivating all entries to disk
    [Server:server-three-slave] 10:02:53,958 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-1) ISPN000030: Passivated 0 entries in 0 milliseconds
    [Server:server-three-slave] 10:02:53,963 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010282: Stopped //default-host//cluster-demo cache from web container
    [Server:server-three-slave] 10:02:54,031 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-14-thread-1) ISPN000029: Passivating all entries to disk
    [Server:server-three-slave] 10:02:54,033 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-14-thread-1) ISPN000030: Passivated 1 entries in 1 milliseconds
    [Server:server-three-slave] 10:02:54,045 INFO  [org.jboss.as.clustering.infinispan] (pool-14-thread-1) JBAS010282: Stopped repl cache from web container
    [Server:server-three-slave] 10:02:54,056 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment cluster-demo.war in 108ms
    [Server:server-three-slave] 10:02:54,295 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-12-thread-1) ISPN000082: Stopping the RpcDispatcher
    [Server:server-three-slave] 10:02:54,300 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-6,null) Problems unmarshalling remote command from byte buffer: org.infinispan.CacheException: Cache manager is either starting up and type (id=74) cannot be resolved (thread not interrupted)
    [Server:server-three-slave]     at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:274) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]
    [Server:server-three-slave]     at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
    [Server:server-three-slave]     at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
    [Server:server-three-slave]     at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.11.GA.jar:1.3.11.GA]
    [Server:server-three-slave]     at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:148) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]
    [Server:server-three-slave]     at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:110) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]
    [Server:server-three-slave]     at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:84) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]
    [Server:server-three-slave]     at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]
    [Server:server-three-slave]     at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:196) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]
    [Server:server-three-slave]     at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.1.Final.jar:7.1.1.Final]
    [Server:server-three-slave]     at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.1.Final.jar:7.1.1.Final]
    [Server:server-three-slave]     at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:793) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:595) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final.jar:3.0.6.Final]
    [Server:server-three-slave]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_03]
    [Server:server-three-slave]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_03]
    [Server:server-three-slave]     at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_03]
    [Server:server-three-slave]

    Any ideas On what im doing wrong?

    1. Apr 17, 2012

      I noticed in the top right hand corner at "Messages" there was an "Unknown Error" When I clocked it, this is what I saw:

      "

      Unknown error
      Unexpected HTTP response: 500

      Request
      {
      "operation" => "composite",
      "address" => [],
      "steps" => [

      Unknown macro: { "address" => [ ("server-group" => "other-server-group"), ("deployment" => "cluster-demo.war") ], "operation" => "add" }

      ,

      Unknown macro: { "address" => [ ("server-group" => "other-server-group"), ("deployment" => "cluster-demo.war") ], "operation" => "deploy" }

      ]
      }

      Response

      Internal Server Error
      {
      "outcome" => "failed",
      "result" => {
      "step-1" =>

      Unknown macro: { "outcome" => "failed", "rolled-back" => true }

      ,
      "step-2" =>

      },
      "failure-description" => "JBAS014654: Composite operation was rolled back",
      "rolled-back" => true
      }
      "

  17. Apr 19, 2012

    Hi

    I try create HA cluster, as described in thi aricle and as described in this comment

    https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto#comment-22872548

    But HA not work: I put date on master (i see from log) stop server on master, on slave "The time is null "

    Session not replicated.

    In logs I see"JBAS010206: Number of cluster members: 1"
    What I do wrong?
    config files and logs I post there: https://gist.github.com/2419905

  18. May 09, 2012

    Hi 

    I downloaded ver 7.1.1 and followed the instructions to create a master-slave set of servers. I am successfully able to remotely start a slave server, but I am unable to deploy the cluster-demo.war.

    The sequence I follow is as detailed in the guide;

    • I start the master server-three 
    • I start the slave server-three
    • I upload the cluster-demo.war
    • I attempt to deploy to the cluster group

    There are no overt errors on the slave (and master) console when I start the default servers, but I do see the following on starting server-three:

    [Server:server-three] JBAS014775:    New missing/unsatisfied dependencies:
    [Server:server-three]       service jboss.binding.jacorb (missing) dependents: [service jboss.jacorb.orb-service] 
    [Server:server-three]       service jboss.binding.jacorb-ssl (missing) dependents: [service jboss.jacorb.orb-service] 
    [Server:server-three]       service jboss.binding.messaging (missing) dependents: [service jboss.messaging.default] 
    [Server:server-three]       service jboss.binding.messaging-throughput (missing) dependents: [service jboss.messaging.default] 
    [Server:server-three] 
    [Server:server-three] 17:25:20,392 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss AS 7.1.1.Final "Brontes" started (with errors) in 5684ms - Started 146 of 280 services (12 services failed or missing dependencies, 121 services are passive or on-demand)
    [Server:server-three] 17:25:20,387 INFO  [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report

    [Server:server-three] JBAS014775:    New missing/unsatisfied dependencies:

    [Server:server-three]       service jboss.binding.jacorb (missing) dependents: [service jboss.jacorb.orb-service] 

    [Server:server-three]       service jboss.binding.jacorb-ssl (missing) dependents: [service jboss.jacorb.orb-service] 

    [Server:server-three]       service jboss.binding.messaging (missing) dependents: [service jboss.messaging.default] 

    [Server:server-three]       service jboss.binding.messaging-throughput (missing) dependents: [service jboss.messaging.default] 

    [Server:server-three] 

    [Server:server-three] 17:25:20,392 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss AS 7.1.1.Final "Brontes" started (with errors) in 5684ms - Started 146 of 280 services (12 services failed or missing dependencies, 121 services are passive or on-demand)

    The remote deployment hangs and terminates with a 500 error in the master log. The slave console shows the following messages:

    [Server:server-three] 16:33:08,859 INFO  [stdout] (pool-13-thread-1) 
    [Server:server-three] 16:33:08,860 INFO  [stdout] (pool-13-thread-1) -------------------------------------------------------------------
    [Server:server-three] 16:33:08,861 INFO  [stdout] (pool-13-thread-1) GMS: address=slaveport:server-three/web, cluster=web, physical address=10.16.221.102:55450
    [Server:server-three] 16:33:08,861 INFO  [stdout] (pool-13-thread-1) -------------------------------------------------------------------
    [Server:server-three] 16:33:11,133 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
    [Server:server-three] 16:33:11,135 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
    [Server:server-three] 16:33:11,190 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-1) ISPN000078: Starting JGroups Channel
    [Server:server-three] 16:33:11,195 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-1) ISPN000094: Received new cluster view: [slaveport:server-three/web|0] [slaveport:server-three/web]
    [Server:server-three] 16:33:11,196 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-1) ISPN000079: Cache local address is slaveport:server-three/web, physical addresses are [10.16.221.102:55450]
    [Server:server-three] 16:33:11,201 INFO  [org.infinispan.factories.GlobalComponentRegistry] (pool-14-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL
    [Server:server-three] 16:33:11,201 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (pool-14-thread-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
    [Server:server-three] 16:33:11,302 INFO  [org.infinispan.jmx.CacheJmxRegistration] (pool-14-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server.
    [Server:server-three] 16:33:11,314 INFO  [org.jboss.as.clustering.infinispan] (pool-14-thread-1) JBAS010281: Started repl cache from web container
    [Server:server-three] 16:33:11,323 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (MSC service thread 1-4) JBAS010206: Number of cluster members: 1
    [Server:server-three] 16:33:11,368 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-3) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
    [Server:server-three] 16:33:11,388 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (MSC service thread 1-3) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
    [Server:server-three] 16:33:11,410 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform mbean server.
    [Server:server-three] 16:33:11,412 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started //default-host//cluster-demo cache from web container
    [Server:server-three] 16:33:11,427 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-3) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
    [Server:server-three] 16:33:11,429 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-3) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
    [Server:server-three] 16:33:11,541 INFO  [org.jboss.web] (MSC service thread 1-3) JBAS018210: Registering web context: /cluster-demo
    [Server:server-three] 16:33:11,748 INFO  [org.jboss.as.server] (host-controller-connection-threads - 1) JBAS015870: Deploy of deployment "cluster-demo.war" was rolled back with failure message JBAS014750: Operation handler failed to complete
    [Server:server-three] 16:33:11,768 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-2) ISPN000029: Passivating all entries to disk
    [Server:server-three] 16:33:11,769 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-2) ISPN000030: Passivated 0 entries in 1 milliseconds
    [Server:server-three] 16:33:11,772 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010282: Stopped //default-host//cluster-demo cache from web container
    [Server:server-three] 16:33:11,802 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-14-thread-1) ISPN000029: Passivating all entries to disk
    [Server:server-three] 16:33:11,803 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-14-thread-1) ISPN000030: Passivated 0 entries in 0 milliseconds
    [Server:server-three] 16:33:11,806 INFO  [org.jboss.as.clustering.infinispan] (pool-14-thread-1) JBAS010282: Stopped repl cache from web container
    [Server:server-three] 16:33:11,831 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment cluster-demo.war in 81ms
    [Server:server-three] 16:33:11,935 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-12-thread-1) ISPN000082: Stopping the RpcDispatcher
    I'm running this on os X (Lion) and java version "1.6.0_31"

    Java(TM) SE Runtime Environment (build 1.6.0_31-b04-415-11M3635)

    Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01-415, mixed mode)

    Any help will be greatly appreciated.

    David.

    1. May 10, 2012

      Try using full-ha-sockets in other-server-group, ? Have a look at this thread if you encounter a different error after trying this ...

      https://community.jboss.org/message/727190#727190

      1. May 10, 2012

        Thank you that fixed the problem.

        The issue I run into now relates to configuring httpd - I am attempting to include the cluster modules into the bundled apache 2.2 installation on a Mac (Lion).

        I get the following error:

        sudo apachectl -k restart
        httpd: Syntax error on line 116 of /private/etc/apache2/httpd.conf: Cannot load /usr/libexec/apache2/mod_slotmem.so into server: dlopen(/usr/libexec/apache2/mod_slotmem.so, 10): no suitable image found.  Did find:\n\t/usr/libexec/apache2/mod_slotmem.so: mach-o, but wrong architecture
        sudo apachectl -k restart

        httpd: Syntax error on line 116 of /private/etc/apache2/httpd.conf: Cannot load /usr/libexec/apache2/mod_slotmem.so into server: dlopen(/usr/libexec/apache2/mod_slotmem.so, 10): no suitable image found.  Did find:\n\t/usr/libexec/apache2/mod_slotmem.so: mach-o, but wrong architecture

        Anybody have any success with this, or should I just go ahead and try the httpd bundle from jboss directly?

        Thanks.

  19. May 10, 2012

    I have progressed to the point where everything but clustering seems to be working.

    I have a master and a slave.

    I installed the httpd binary from jboss and configured it according to the instructions given in the main document. 

    I am able to see the mod_cluster_manager page (but none of the links show anything).

    I can invoke the put.jsp method in cluster-demo when I go directly to port 8330, but I get an page not found error when I try navigate using port number 10001.

    The relevant portion of my httpd.conf file is below. The error log show

    File does not exist: /opt/jboss/httpd/htdocs/htdocs/cluster-demo

    As usual, any help will be greatly appreciated! 

    # MOD_CLUSTER_ADDS

    # Adjust to you hostname and subnet.

    <IfModule manager_module>

     

    # This Listen port is for the mod_cluster-manager, where you can see the status of mod_cluster.

    # Port 10001 is not a reserved port, so this prevents problems with SELinux.

    Listen 172.16.129.148:10001 <VirtualHost 172.16.129.148:10001>  

    <Directory />

       Order deny,allow

       Deny from all

       Allow from all

     </Directory>  # This directive allows you to view mod_cluster status at URL http://10.211.55.4:10001/mod_cluster-manager

     <Location /mod_cluster-manager>
      SetHandler mod_cluster-manager

      Order deny,allow

      Deny from all

        Allow from all
     </Location>

     KeepAliveTimeout 60

     MaxKeepAliveRequests 0

     ManagerBalancerName other-server-group

     AdvertiseFrequency 5

     EnableMCPMReceive

    </VirtualHost>

    </IfModule>

    1. May 10, 2012

      Hi,

      if mod_cluster-manager only shows a heading like "mod_cluster/1.2.0.Final" and some links (showing nothing like you say) your nodes have not been registered on the apache-side of mod_cluster. This would also explain that you get a 404 when you try to access the cluster-demo through mod_cluster (port 10001) but it works if you go directly on a JBoss instance.

      I had this problem by myself and it stinks like JBPAPP-7516 because it works fine with the latest EAP6 beta and the JBoss AS7.1.2 (you can get a nightly build here).

      Also watch your JBoss for any error-messages on the terminal. Sometimes the error-messages are hard to see between all the other messages.

      Maybe you should give the latest working nightly-build a try, But be warned, just copying your actual configuration files to the new JBoss installation won't work, because the JBoss makes changes to these files during deployment.

      Greetings from Berlin!

      1. May 10, 2012

        Immanuel,

        Thanks for the prompt response. I agree with your conclusion that the cluster is not being registered on the Apache side as this explains all I see. There are no errors in my JBoss console and I will try the nightly build next. I don't understand your point about configuration files though - why will JBoss overwrite my config files (domain.xml and host.xml) or did I misunderstand you?

        I think we will probably end up running a number of JBoss instances behind a hardware balancer so httpd may be moot, but I still need to prove this works in our dev environment.

        Danke for your help

        David. 

        1. May 10, 2012

          Hi,

          the point with the configuration files (I think it was just the host.xml) is that if you deploy something into the JBoss it will write that (with some hash-values and so on) into the host.xml so the deployment is not lost after a restart of the JBoss. Maybe this behaviour does only occur when you deploy over the management-console. But after trying different JBoss 7 versions (including 7.1.0 which refuses to deploy SFSBs annotated with @Clustered) I ran into this problem a couple of times. Just copying the configuration files won't work: The JBoss you copied your configuration to doesn't find the deployments of the old JBoss you copied from and maybe some more issues.

          I don't think it's the most efficient way but to solve the issue, but I alway ended up setting up a fresh configuration (so I'm quite fit in setting up the cluster described in this howto ). After all this issue isn't really a big deal. I just wanted give a little hint if you run into a problem that looks like this.

          Greetings from Berlin!

      2. May 11, 2012

        Hi 

        Thanks for your help, you saved me a lot of head-banging!

        I finally got everything to work with one caveat and one small issue.

        First the caveat: it seems all nodes in a cluster need to run the same version of JBoss 7.

        Second: I was unable to suppress this message:

        [Server:server-three] 13:36:29,732 WARN  [org.hornetq.core.cluster.impl.DiscoveryGroupImpl] (hornetq-discovery-group-thread-dg-group1) There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=5b2e186c-9af8-11e1-9675-e3f2a1f44414[Server:server-three] 13:36:29,732 WARN  [org.hornetq.core.cluster.impl.DiscoveryGroupImpl] (hornetq-discovery-group-thread-dg-group1) There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=5b2e186c-9af8-11e1-9675-e3f2a1f44414

        Again, any suggestion about killing this would be great. I'm running on os X (Lion) so the comments about NFS don't really help me much.

        Thanks David.

        1. May 24, 2012

          Hi David,

          ok, maybe I'm a bit late but: Have you created the slave by copying the master? Then try to delete the contents of domain/data and/or domain/servers/yourserver/data . I by myself didn't experience this problem.

  20. Jun 12, 2012

    Hi everybody,

    I have a question: can I use the capabilities of AS7 clustering & HA configurations in order to use other protocols

    than http? We need to use telnet (this is the application requirement)

    Thank you

  21. Jun 18, 2012

    I followed these directions and comments by others very carefully and I got everything working *except* for session replication. I tried adding the <cluster-user> and <cluster-password> as was suggested. I tried both 7.1.1-Final and yesterdays latest 7.1.x build (really 7.2.0 Alpha-1). I made the minimal changes required and everything else is stock settings.

    Once the other-server-group (server three) begins running I start getting errors on both master/slave as others have reported.

    As was suggested in a number of forums postings I tried adding cluster-password. It wasn't clear if you were supposed to add <cluster-name> and <cluster-password> to both master and slave so I added it to both but to no avail (still doesn't work).

    There is NO jboss user named this but according to the directions there doesn't seem to need to be one. It's interesting that the exception says 'HORNETQ.CLUSTER.ADMIN.USER' even though I specified to use 'admin'

    This is how I start both master and slave:

    master:

    $ sudo -u jboss bin/domain.sh  -b 192.168.150.100 -bmanagement 192.168.150.100

    slave:

    $ sudo -u jboss bin/domain.sh  -b 192.168.150.200 -bmanagement 192.168.150.200 -Djboss.domain.master.address=192.168.150.100 -Djboss.domain.master.port=9999

    I disabled selinx and the firewall on both nodes. Everything else works great: deployments to nodes, mod_cluster but not replication. :(

    Any ideas?

    Thanks

    1. Jun 15, 2012

      It should be a valid username and the right password. 

      1. Jun 15, 2012

        I should have been more clear but I already tried using a valid user from the ManagementRealm previously (added via add-user.sh) and that didn't help. Like I said based on the error message it doesn't appear like it was honoring the <cluster-user> tag since the exception referred to 'HORNETQ.CLUSTER.ADMIN.USER'

        I also tried adding 'HORNETQ.CLUSTER.ADMIN.USER'  as a valid user to the ManagementRealm and ran into the same problem.

        1. Jun 16, 2012

          Could you confirm if you added the username and password in both master and slave ? 

          1. Jun 18, 2012

            Yes I added the same username/password to both master and slave.

      2. Jul 30, 2012

        Is there anyway that we can encrypt this password?

  22. Jun 18, 2012

    How to configure the clustering for standalone server. I tried with standalone-ha.xml configuration file with

    two different ip address, but if i shutdown one node it will not redirect request to other node automatically.

  23. Jun 16, 2012

    I have only one laptop, please let me know how I can setup Jboss AS7 clustering on single machine, I want to test some of my project stuff on clustered environment.

    Thanks in Advance!

    Raj

    1. Jun 16, 2012

      Do u want to apply it in standalone server or domain server?

      1. Jun 16, 2012

        I do not know the difference between standalone and domain, please explain.

        1. Jun 16, 2012

          See in standalone server there is only one server running,either in one machine with two

          different i.p or on two different machine with two different machine. And in domain server there

          will be a group of server. So,if u deploy an application in one server it will be automatically

          available on another servers. in domain server on one i.p address it will be running group of

          servers with different port number.

          1. Jun 16, 2012

            Got it.so as I have only one machine then I can only run in standalone mode, right?

            Can you share me the steps to setup on one machine then please?

            1. Jun 18, 2012

              No you can run domain configuration in one machine for that you have to just

              change the port offset for different server. So, if you run one server on

              127.0.0.1:8080,then you have to run other server on 127.0.0.1:8230 port or any

              other. So you can run domain server on your laptop too. So you have to decide

              as per your need. And even you can also run two standalone server on one

              machine with either different i.p or different port. And i will share you the steps

              very soon. Thanks

  24. Jul 20, 2012

    Hello, Weinan.Li, I need help when I follow the steps in article: https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto
    server-one and server-two can start up successfully on domain master,
    but it failed to start server-three at : http://128.128.86.222:9990/console/App.html#server-instances

    server-one and server-two log content is as bellow:
    ----------------
    [Server:server-two] 10:59:46,003 INFO  [org.jboss.as] (Controller Boot Thread) JBoss AS 7.1.0.CR1b "Flux Capacitor" started in 30718ms - Started 148 of 215 services (65 services are passive or on-demand)
    [Server:server-one] 10:59:46,021 INFO  [org.jboss.as] (Controller Boot Thread) JBoss AS 7.1.0.CR1b "Flux Capacitor" started in 31217ms - Started 148 of 215 services (65 services are passive or on-demand)
    ----------------
    when I click "start" button at administration console page, the console log is like this:
    ----------------
    [Host Controller] 11:04:37,394 INFO  [org.jboss.as.host.controller] (HttpManagementService-threads - 6) JBAS010922: Starting server server-three
    11:04:40,969 INFO  [org.jboss.as.process.Server:server-three.status] (ProcessController-threads - 6) JBAS012017: Starting process 'Server:server-three'
    [Server:server-three] 11:04:46,925 信息    [org.jboss.modules] (main) JBoss Modules version 1.1.0.CR6
    [Server:server-three] 11:04:50,392 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.1.GA
    [Server:server-three] 11:04:50,567 INFO  [org.jboss.as] (MSC service thread 1-1) JBoss AS 7.1.0.CR1b "Flux Capacitor" starting
    [Server:server-three] 11:04:52,017 INFO  [org.xnio] (MSC service thread 1-2) XNIO Version 3.0.0.CR7
    [Server:server-three] 11:04:52,110 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.0.0.CR7
    [Server:server-three] 11:04:52,232 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 3.2.0.CR8
    [Server:server-three] 11:05:00,410 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC00001: Failed to start service jboss.host.controller.channel: org.jboss.msc.service.StartException in service jboss.host.controller.channel: java.net.ConnectException: JBAS012144: Could not connect to remote://128.128.86.222:9999. The connection timed out
    [Server:server-three]     at org.jboss.as.server.mgmt.domain.HostControllerConnectionService.start(HostControllerConnectionService.java:101) [jboss-as-server-7.1.0.CR1b.jar:7.1.0.CR1b]
    [Server:server-three]     at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1824) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]
    [Server:server-three]     at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1759) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]
    [Server:server-three]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_12]
    [Server:server-three]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_12]
    [Server:server-three]     at java.lang.Thread.run(Thread.java:619) [:1.6.0_12]
    [Server:server-three] Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://128.128.86.222:9999. The connection timed out
    [Server:server-three]     at org.jboss.as.protocol.ProtocolChannelClient.connectSync(ProtocolChannelClient.java:166) [jboss-as-protocol-7.1.0.CR1b.jar:7.1.0.CR1b]
    [Server:server-three]     at org.jboss.as.protocol.ProtocolChannelClient.connectSync(ProtocolChannelClient.java:129) [jboss-as-protocol-7.1.0.CR1b.jar:7.1.0.CR1b]
    [Server:server-three]     at org.jboss.as.server.mgmt.domain.HostControllerConnectionService.start(HostControllerConnectionService.java:98) [jboss-as-server-7.1.0.CR1b.jar:7.1.0.CR1b]
    [Server:server-three]     ... 5 more
    [Server:server-three]
    [Server:server-three] 11:05:08,973 INFO  [org.jboss.as.logging] (MSC service thread 1-1) JBAS011502: Removing bootstrap log handlers
    [Server:server-three] 11:05:09,046 INFO  [org.jboss.as.clustering] (ServerService Thread Pool -- 55) JBAS010360: Activating JGroups subsystem.
    [Server:server-three] 11:05:09,099 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 48) JBAS011910: Activating OSGi Subsystem
    [Server:server-three] 11:05:09,270 INFO  [org.jboss.as.clustering] (ServerService Thread Pool -- 60) JBAS010300: Activating Infinispan subsystem.
    [Server:server-three] 11:05:09,537 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 49) JBAS011800: Activating Naming Subsystem
    [Server:server-three] 11:05:12,976 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 63) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
    [Server:server-three] 11:05:12,978 INFO  [org.jboss.as.security] (MSC service thread 1-1) Picketbox version=4.0.6.Beta2
    [Server:server-three] 11:05:13,618 INFO  [org.jboss.as.connector] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.6.Final)
    [Server:server-three] 11:05:19,011 INFO  [org.jboss.as.naming] (MSC service thread 1-2) JBAS011802: Starting Naming Service
    [Server:server-three] 11:05:19,651 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-1) JBoss Web Services - Stack CXF Server 4.0.0.GA
    [Server:server-three] 11:05:21,271 INFO  [org.jboss.as.modcluster] (MSC service thread 1-1) JBAS011704: Mod_cluster uses default load balancer provider
    [Server:server-three] 11:05:21,349 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) Listening on /128.128.86.222:4697
    [Server:server-three] 11:05:21,645 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-1) Initializing mod_cluster 1.2.0.Beta2
    [Server:server-three] 11:05:22,205 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (MSC service thread 1-1) Listening to proxy advertisements on 224.0.1.105:23364
    [Server:server-three] 11:05:22,726 INFO  [org.apache.coyote.http11.Http11AprProtocol] (MSC service thread 1-1) Starting Coyote HTTP/1.1 on http-128.128.86.222-128.128.86.222-8330
    [Server:server-three] 11:05:22,919 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-1) JBAS015400: Bound mail session [java:jboss/mail/Default]
    [Server:server-three] 11:05:26,128 INFO  [org.jboss.as.jacorb] (MSC service thread 1-1) CORBA ORB Service Started
    [Server:server-three] 11:05:26,385 信息    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/usr/jboss-as-7.1.0.CR1b/domain/servers/server-three/data/messagingjournal,bindingsDirectory=/usr/jboss-as-7.1.0.CR1b/domain/servers/server-three/data/messagingbindings,largeMessagesDirectory=/usr/jboss-as-7.1.0.CR1b/domain/servers/server-three/data/messaginglargemessages,pagingDirectory=/usr/jboss-as-7.1.0.CR1b/domain/servers/server-three/data/messagingpaging)
    [Server:server-three] 11:05:26,546 警告    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
    [Server:server-three] 11:05:27,646 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-1) Started Netty Acceptor version 3.2.3.Final-r$

    Unknown macro: {buildNumber}

    128.128.86.222:5695 for CORE protocol
    [Server:server-three] 11:05:27,649 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-1) Started Netty Acceptor version 3.2.3.Final-r$

    128.128.86.222:5705 for CORE protocol
    [Server:server-three] 11:05:27,651 信息    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) Server is now live
    [Server:server-three] 11:05:27,651 信息    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) HornetQ Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [ff1f011e-d17b-11e1-b314-000c29eaa811] started
    [Server:server-three] 11:05:27,982 INFO  [org.jboss.as.jacorb] (MSC service thread 1-1) CORBA Naming Service Started
    [Server:server-three] 11:05:28,024 WARN  [com.arjuna.ats.arjuna] (Transaction Expired Entry Monitor) ARJUNA012210: Unable to use InetAddress.getLocalHost() to resolve address.
    [Server:server-three] 11:05:28,438 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
    [Server:server-three] 11:05:28,441 信息    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) trying to deploy queue jms.topic.testTopic
    [Server:server-three] 11:05:29,034 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/topic/test
    [Server:server-three] 11:05:29,038 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory
    [Server:server-three] 11:05:29,039 信息    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) trying to deploy queue jms.queue.testQueue
    [Server:server-three] 11:05:29,407 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/queue/test
    [Server:server-three] 11:05:30,218 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010406: Registered connection factory java:/JmsXA
    [Server:server-three] 11:05:30,275 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-2) HornetQ resource adaptor started
    [Server:server-three] 11:05:30,276 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-2) IJ020002: Deployed: file://RaActivatorhornetq-ra
    [Server:server-three] 11:05:30,284 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
    [Server:server-three] 11:05:30,315 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
    [Server:server-three] 11:05:30,379 ERROR [org.jboss.as] (Controller Boot Thread) JBoss AS 7.1.0.CR1b "Flux Capacitor" started (with errors) in 46986ms - Started 146 of 254 services (2 services failed or missing dependencies, 101 services are passive or on-demand)
    ----------------
    It seems that when I start process 'Server:server-three', it tried to connect to remote://128.128.86.222:9999, and then it caused "connection timed out".
    Should I modify the "domain-controller" property in host.xml on domain master host?

    P.s. domain master IP: 128.128.86.222; OS: CentOs 5.4; Memery Size: 256M; Run mode: linux text.
    and my master host config file "host.xml":
    ----------------
    <host name="master" xmlns="urn:jboss:domain:1.1">

        <management>
            <security-realms>
                <security-realm name="ManagementRealm">
                    <authentication>
                        <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
                    </authentication>
                </security-realm>
            </security-realms>
            <management-interfaces>
                <native-interface security-realm="ManagementRealm">
                    <socket interface="management" port="$

    Unknown macro: {jboss.management.native.port}

    "/>
                </native-interface>
                <http-interface security-realm="ManagementRealm">
                    <socket interface="management" port="$

    Unknown macro: {jboss.management.http.port}

    "/>
                </http-interface>
            </management-interfaces>
        </management>

        <domain-controller>
           <local />
           <!- Alternative remote domain controller configuration with a host and port ->
           <!-- <remote host="$

    Unknown macro: {jboss.domain.master.address}

    " port="$

    Unknown macro: {jboss.domain.master.port}

    "/> -->
        </domain-controller>

        <interfaces>
            <interface name="management">
                <inet-address value="128.128.86.222"/>
            </interface>
        <interface name="public">
               <inet-address value="128.128.86.222"/>
            </interface>
            <interface name="unsecured">
               <inet-address value="128.128.86.222"/>
            </interface>
        </interfaces>

            <jvms>
               <jvm name="default">
              <heap size="64m" max-size="256m"/>
           </jvm>
            </jvms>

        <servers>
            <server name="server-one" group="main-server-group">
                <!-- Remote JPDA debugging for a specific server
                <jvm name="default">
                  <jvm-options>
                    <option value="-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"/>
                  </jvm-options>
               </jvm>
               -->
            </server>
            <server name="server-two" group="main-server-group" auto-start="true">
        <!-- server-two avoids port conflicts by incrementing the ports in
                     the default socket-group declared in the server-group -->
                <socket-bindings port-offset="150"/>
            </server>
            <server name="server-three" group="other-server-group" auto-start="false">
                <!-- server-three avoids port conflicts by incrementing the ports in
                     the default socket-group declared in the server-group -->
                <socket-bindings port-offset="250"/>
            </server>
        </servers>
    </host>

  25. Aug 09, 2012

    Hi, 

            I was able to setup till the last step but the request is not getting forwarded from Apache Http server to AS7 servers. Both my Apache Http Server and AS7 master are running on a Windows7 machine and the slave AS7 is running on Window 7 laptop. Both the main-server-group and other-server-group (full-ha) are up and running correctly. I am able to access the deployed app on port 8330. Also the Http server is up and running on port 80 so when I put my master ip address I get "It Works!" message. Also, able to access the mod_cluster_manager on port 8000 correctly. Not sure what I am missing currently as the logs are not pointing me to any errors that might be happening.

    The current httpd.conf is as below: 

    # MOD_CLUSTER_ADDS
    # Adjust to you hostname and subnet.
    <IfModule manager_module>
      Listen 172.20.201.24:10001
      ManagerBalancerName other-server-group
      <VirtualHost 172.20.201.24:10001>
        <Location />
         Order deny,allow
         Allow from all
        </Location>
        ErrorLog "logs/chatter_log"
        LogLevel debug
        KeepAliveTimeout 300
        MaxKeepAliveRequests 0
        #ServerAdvertise on http://127.0.0.1:10001
        AdvertiseFrequency 5
        #AdvertiseSecurityKey secret
        #AdvertiseGroup 224.0.1.105:23364
        EnableMCPMReceive
        #<Location /mod_cluster_manager>
        #   SetHandler mod_cluster-manager
        #   Order deny,allow
        #   Allow from all
        #</Location>
      </VirtualHost>
    </IfModule>
    NameVirtualHost 172.20.201.24:8000
    Listen 172.20.201.24:8000
    <VirtualHost 172.20.201.24:8000>
        ServerAdmin sudeep.batra@synechron.com
        ServerName 172.20.201.24
        ErrorLog "logs/cluster_log"
        LogLevel debug
        EnableMCPMReceive
        ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On
        ProxyPassReverse / balancer://other-server-group
        ProxyPreserveHost On
        <Location />
            Order deny,allow
            Allow from All
        </Location>
        <Location /mod_cluster_manager>
            SetHandler mod_cluster-manager
            Order deny,allow
            Deny from all
            Allow from all
        </Location>
    </VirtualHost>
    # MOD_CLUSTER_ADDS

    # Adjust to you hostname and subnet.

    <IfModule manager_module>

      Listen 172.20.201.24:10001

      ManagerBalancerName other-server-group

      <VirtualHost 172.20.201.24:10001>

        <Location />

         Order deny,allow

         Allow from all

        </Location>

        ErrorLog "logs/chatter_log"

        LogLevel debug

        KeepAliveTimeout 300

        MaxKeepAliveRequests 0

        #ServerAdvertise on http://127.0.0.1:10001

        AdvertiseFrequency 5

        #AdvertiseSecurityKey secret

        #AdvertiseGroup 224.0.1.105:23364

        EnableMCPMReceive

        #<Location /mod_cluster_manager>

        #   SetHandler mod_cluster-manager

        #   Order deny,allow

        #   Allow from all

        #</Location>

      </VirtualHost>

    </IfModule>

    NameVirtualHost 172.20.201.24:8000

    Listen 172.20.201.24:8000

    <VirtualHost 172.20.201.24:8000>

        ServerAdmin sudeep.batra@synechron.com

        ServerName 172.20.201.24

        ErrorLog "logs/cluster_log"

        LogLevel debug

        EnableMCPMReceive

        ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

        ProxyPassReverse / balancer://other-server-group

        ProxyPreserveHost On

        <Location />

            Order deny,allow

            Allow from All

        </Location>

        <Location /mod_cluster_manager>

            SetHandler mod_cluster-manager

            Order deny,allow

            Deny from all

            Allow from all

        </Location>

    </VirtualHost>

  26. Feb 08, 2014

    With Wildfly CR1 you'll run into this bug: https://issues.jboss.org/browse/WFLY-2288

    The multicast advertisements from the httpd will be silently ignored by the Wildfly nodes. This might be fixed in Wildfly final.

    Change your domain.xml on all nodes:

    and add this to your Apache config in the virtual host:

    1. Feb 14, 2014

      Right, resoved in 8.0.0.Final. Nevertheless, it's a good practice to use security key.

  27. Jun 09, 2014

    Thanks for you nice documentation.

    I’m currently running my JBoss cluster using S3Ping configuration in AWS under Elastic Load Balencer, and this configuration is active-active and my application is not working as expected with this configuration and its more suitable with active-passive configuration, So I followed this documentation --[https://docs.jboss.org/author/display/WFLY8/WildFly+8+Cluster+Howto|../../../../../../../../../../display/WFLY8/WildFly+8+Cluster+Howto] , It’s worked for me except mod-cluster configuration.

    As I mentioned above, running all the instances in AWS cloud, launch 3 instances
    1. Domain controller and Master
    2. Slave
    3 Mod-cluster

    Master-Slave configuration is working perfectly and mod-cluster configuration is not working for me.

    1. In mod-cluster configuration, how will detect my master/slave instances, because there is no such configs in VirtualHost file.
    2. Is this configuration will work in AWS infrastructure

    Please advise me to configure this. thanks in advance

    OS - Ubuntu, Version - wildfly8 

    1. Jun 25, 2014

      AWS does not support UDP multicast, so you need to configure proxy-list="loadbalancerhost:port,..." in the modcluster subsystem explicitly.

  28. Aug 05, 2014

    Thanks Husar, proxy-list config is worked for me!!

    but here in this config has a single point of failure, if the domain controller is down all the hosts will be down. Is there anyway to configure Domain controller in HA(two domain controller shares the session data) mode?

    Thanks in advance!

  29. Feb 05, 2015

    I am able to run my application successfully in cluster which I created following this document.

    However when I try the web socket part of my application I get the error "UT000077: The underlying transport does not support HTTP upgrade". Can you please suggest what are the options to resolve it?

    Thanks.

  30. Feb 19, 2015

    Thanks.