JBoss Community Archive (Read Only)

JBoss Cloud Access

Launching clustered JBoss Enterprise Application Platform instances

In this example clustered JBoss Enterprise Application Platform instances are deployed with the Day Trader sample application. A JBoss Enterprise Web Server instance is deployed as a proxy and an instance of MySQL DBMS.

Procedure

  1. Create a Relational Database Service database instance;

  2. Create a Virtual Private Cloud (VPC);

  3. Launch a JBoss Enterprise Web Server instance to server as a mod_cluster proxy and a NAT instance for the VPC;

  4. Configure VPC routing tables for the JBoss Enterprise Application Platform (EAP) nodes;

  5. Launch JBoss Enterprise Application Platform nodes with the Day Trader sample application deployed;

  6. Verify the JBoss Enterprise Application Platform cluster.

Create a Relational Database Service database instance

Relational Database Service (RDS) is a convenient service provided in Amazon EC2 cloud to easily manage an RDBMS with failover, back-up and restore capabilities. Currently several versions of Oracle and MySQL database servers are available. This example uses the MySQL database server but JBoss Enterprise Application Platform also supports Oracle Database.

  1. Go to the RDS tab in the AWS console;

  2. Subscribe to the service if needed;

  3. Go to the RDS tab in the AWS console;

  4. Subscribe to the service if needed;

  5. Click on "Launch DB instance";

  6. Click on MySQL

    1. Select version (for example 5.5.12);

    2. Select small instance;

    3. Multi-AZ Deployment and Auto upgrade: off;

    4. Storage: 5GB;

    5. Choose the database administrator's username and password;

    6. On the next screen choose a database name to be created with the instance (e.g. dtrader);

    7. On the next screen you can disable back-up and maintenance;

    8. Confirm settings and database should come up after a few minutes.

The above settings are suitable and sufficient for trying out this example scenario but for a production environment it's highly recommended to consider the failover and back-up features.  It's a good practice to create separate user/password pairs for the applications accessing the database. Tune other configuration options according to your applications requirements.

Create a Virtual Private Cloud (VPC)

  1. In the AWS console go to the VPC tab (subscribe to the service if necessary);

  2. Click on "Create new VPC";

  3. Choose a VPC with one public and one private subnet

    1. Set the public subnet to be 10.0.0.0/24;

    2. Set the private subnet - 10.0.1.0/24;

  4. Go to Elastic IPs;

  5. Create an elastic IP for use by the EWS mod_cluster proxy/NAT instance;

  6. Security groups - create a security group to allow all traffic in and out;

  7. Network ACLs

    1. An ACL to allow all traffic in and out;

    2. An ACL to allow all traffic out and traffic in on only TCP ports 22, 8009, 8080, 8443 and 16163.

VPC is recommended for a JBoss Enterprise Application Platform cluster setup because it greatly simplifies ensuring secure communication between the cluster nodes, JON Server and the mod_cluster  proxy. Without a VPC all these communication channels need to be encrypted and authenticated. Please refer to JBoss Enterprise Application Platform for detailed instructions on configuring SSL.

Also note that in this example we are using a database external to the VPC. Your security policies may require connection to it to be encrypted for a production setup. Please refer to Amazon's RDS FAQ for details about encrypting the database connections.

Launch a JBoss Enterprise Web Server instance to server as a mod_cluster proxy and a NAT instance for the VPC

  • Create an elastic IP for this instance;

  • Select AMI;

  • Security Group - allow all traffic (use Red Hat Enterprise Linux's built-in firewall capabilities to restrict access if desired);

  • Choose running in the public subnet of the VPC;

  • Choose a static IP (e.g. 10.0.0.4);

  • Put the following in the user-data field: 

    echo 1 > /proc/sys/net/ipv4/ip_forward
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter

iptables -I INPUT 4 -s 10.0.0.0/24 -p tcp --dport 7654 -j ACCEPT
iptables -I INPUT 4 -p tcp --dport 80 -j ACCEPT

iptables -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD -s 10.0.1.0/24 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 ! -s 10.0.0.4 -j MASQUERADE

# balancer module incompatible with mod_cluster
sed -i -e 's/LoadModule proxy_balancer_module/#\0/' /etc/httpd/conf/httpd.conf

cat > /etc/httpd/conf.d/mod_cluster.conf << "EOF"
#LoadModule proxy_module modules/mod_proxy.so
#LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule manager_module modules/mod_manager.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so

Listen 7654

# workaround JBPAPP-4557
MemManagerFile /var/cache/mod_proxy/manager

<VirtualHost *:7654>
   <Location /mod_cluster-manager>
      SetHandler mod_cluster-manager
      Order deny,allow
      Deny from all
      Allow from 127.0.0.1
   </Location>

   <Location />
      Order deny,allow
      Deny from all
      Allow from 10.
      Allow from 127.0.0.1
   </Location>

   KeepAliveTimeout 60
   MaxKeepAliveRequests 0
   ManagerBalancerName mycluster
   ServerAdvertise Off
   EnableMCPMReceive On
</VirtualHost>
EOF

echo "`hostname | sed -e 's/ip-//' -e 'y/-/./'`        `hostname`" >> /etc/hosts

service httpd start

  • Disable the Amazon EC2 cloud source/dest checking for this instance so it can act as a router;

    • Right-click on the running EWS instance and choose "Change Source/Dest check";

    • Confirm disabling the check;

Starting JBoss Enterprise Application Platform instances in a cluster

EC2 clustering profiles

The JBoss Enterprise Application Platform AMIs provided by Red Hat feature two additional server profiles called cluster_ec2 and mod_cluster-ec2. They both feature the ability to form a cluster inside the EC2 environment where multicast is not available. This is done by configuring JGroups to use only TCP unicast for cluster communication and S3_PING as the discovery protocol. Additionally mod_cluster-ec2 is pre-configured to easily register with mod_cluster proxies.

The following sections describe the additional setup required for S3_PING, clustering and mod_cluster.

VPC private subnet default route

Since JBoss Enterprise Application Platform cluster nodes will be run in the private subnet of the VPC, but cluster nodes require Internet access for S3 connectivity, default route needs to be set to go through the NAT instance.

  1. Look at Amazon AWS console and see instance ID of the EWS instance;

  2. Go to the VPC tab -> route tables -> click on the routing table used by the private subnet;

  3. In the field for a new route write 0.0.0.0/0;

  4. Click on "select a target" and choose "Enter Instance ID";

  5. Choose the ID of the running EWS instance.

IAM setup

The S3_PING protocol as suggested by its name is using an S3 bucket to discover other cluster members. The JGroups versions 2.6.x require Amazon AWS account access and secret keys to authenticate against the S3 service. It is a security risk to enter your main account credentials in user-data field, store online or in an AMI. Fortunately a separate account can be created using the Amazon IAM feature which would be only granted access to a single S3 bucket:

  1. Go to the IAM tab in the AWS console;

  2. Click on users and then "Create New Users";

  3. Choose a name (e.g. jbosscluster) and leave "Generate an access key for each User" option checked;

  4. Click "download credentials" and keep these in a safe place;

  5. Close the window and click on the newly created user;

  6. In the summary tab you will see User ARN (e.g. arn:aws:iam::05555555555:user/jbosscluster*)*, needed for the next step.

S3 bucket setup

  1. Open the S3 tab in AWS console;

  2. Click on "Create Bucket";

  3. Choose a name (e.g. clusterbucket123) and click on create. (Note that bucket names are unique across the whole S3 so you can't use exactl this name);

  4. Right click over the new bucket and choose Properties;

  5. In the permissions tab click on "Add bucket policy";

  6. If you click on "new policy" a web based policy creation wizard will open but you can just paste the following one and change user ARN and bucket name: 

    {
    "Version": "2008-10-17",
    "Id": "Policy1312228794320",
    "Statement": [
        {
            "Sid": "Stmt1312228781799",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::055555555555:user/jbosscluster"
                ]
            },
            "Action": [
                "s3:ListBucketVersions",
                "s3:GetObjectVersion",
                "s3:ListBucket",
                "s3:PutBucketVersioning",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:GetObject",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:GetBucketVersioning"
            ],
            "Resource": [
                "arn:aws:s3:::jbosscluster/*",
                "arn:aws:s3:::jbosscluster"
            ]
        }
    ]
}

Launching the JBoss Enterprise Application Platform AMIs

All necessary preparation for running a cluster is now complete. The last task is to put the right configuration in User Data field.

  1. Select AMI;

  2. Choose desired number of instances (this would be cluster size, e.g. 2);

  3. Choose VPC and instance type;

  4. Security Group - allow all traffic from the JBoss Enterprise Application Platform cluster subnet (other restrictions as desired);

  5. Put the following into the User Data field:

EAP cluster user-data
MOD_CLUSTER_PROXY_LIST=10.0.0.4:7654

## clustering setup
JBOSS_JGROUPS_S3_PING_SECRET_ACCESS_KEY=<your secret key>
JBOSS_JGROUPS_S3_PING_ACCESS_KEY=<your access key>
JBOSS_JGROUPS_S3_PING_BUCKET=clusterbucket123

## this will workaround the problem that in a VPC, instance hostname is not resolvable
for (( i=1 ; i<255 ; i++ )); do
     echo -e "10.0.1.$i\tip-10-0-1-$i" ;
done >> /etc/hosts

PORTS_ALLOWED="1024:65535"
JBOSSAS_ADMIN_PASSWORD=<your password for opening admin console>
JBOSS_IP=`hostname` #listen on public/private EC2 IP address

cat> $USER_SCRIPT << "EOF"
## deploy application /TODO: use examples from RPM/
#wget https://s3.amazonaws.com/jenkins-store/cache/JMS-test/jmssender-1.0.war -O $JBOSS_DEPLOY_DIR/jmssender.war
#wget https://s3.amazonaws.com/jenkins-store/cache/JMS-test/mdbtest-1.1.jar -O $JBOSS_DEPLOY_DIR/mdbtest-1.1.jar
cp /usr/share/java/jboss-ec2-eap-samples/jmssender-1.0.war $JBOSS_DEPLOY_DIR/jmssender.war
cp /usr/share/java/jboss-ec2-eap-samples/mdbtest-1.1.jar $JBOSS_DEPLOY_DIR/
wget https://s3.amazonaws.com/jenkins-store/tools/lib/mysql-connector-java-5.1.17-bin.jar -O /var/lib/jbossas/server/$JBOSSCONF/lib/mysql-connector-java-5.1.17-bin.jar

## DefaultDS configuration
cd /tmp
rm -f /var/lib/jbossas/server/$JBOSSCONF/deploy/hsqldb-ds.xml /var/lib/jbossas/server/$JBOSSCONF/deploy/messaging/hsqldb-persistence-service.xml
sed -e 's#\("Clustered">\)false\(</attribute>\)#\1true\2#' -e 's#\("FailoverOnNodeLeave">\)false\(</attribute>\)#\1true\2#' /usr/share/doc/jbossas-5.1.2/examples/jms/mysql-persistence-service.xml > /var/lib/jbossas/server/$JBOSSCONF/deploy/messaging/mysql-persistence-service.xml
sed -i -e 's#<fk-constraint>false</fk-constraint>#<fk-constraint>true</fk-constraint>#' /var/lib/jbossas/server/$JBOSSCONF/conf/standardjbosscmp-jdbc.xml
cat > /var/lib/jbossas/server/$JBOSSCONF/deploy/mysql-ds.xml << "EODS"
<?xml version="1.0" encoding="UTF-8"?>
<datasources>
   <local-tx-datasource>
      <jndi-name>DefaultDS</jndi-name>
      <connection-url>jdbc:mysql://${db.host}:3306/${db.database}</connection-url>
      <driver-class>com.mysql.jdbc.Driver</driver-class>
      <user-name>${db.user}</user-name>
      <password>${db.passwd}</password>
      <metadata>
         <type-mapping>mySQL</type-mapping>
      </metadata>
      <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
   </local-tx-datasource>
</datasources>
EODS

EOF

## database credentials configuration
JAVA_OPTS="$JAVA_OPTS -Ddb.host=instancename.something.rds.amazonaws.com -Ddb.database=mydatabase -Ddb.user=<user> -Ddb.passwd=<pass>"

Running JBoss Enterprise Application Platform cluster in a subnet with network mask smaller than 24 bits or spanning multiple subnets complicates acquiring a unique server peer ID for each cluster member. Please refer to CLUSTER_ID variable in Appendix I for making such setup work reliably.

The auto-scaling Amazon EC2 feature can be used with JBoss Enterprise Application Platform cluster nodes. However make sure you test before deployment that your particular workloads scale to the desired number of nodes and performance according to your needs with the the instance type you are planning to use. Different instance types receive a different share of the EC2 cloud resources. Furthermore instance locality and current network/storage/host machine/RDS utilization can affect performance of a cluster. Test with your expected real-life loads and plan some headroom for unexpected conditions.

Down-scaling a cluster

Amazon EC2 scale-down action terminates the nodes without any chance to gracefully shutdown, hence transactions might be interrupted, other cluster nodes and load balancers will need time to failover. This is likely to hurt your application users' experience. For these reasons it's recommended to scale down your application cluster manually by disabling the server from mod_cluster management interface until processed sessions are finished and shutting down a JBoss Enterprise Application Platform instance gracefully (SSH access to the instance or JON server can be used).

Test that your chosen procedure for scaling-down does not lead to adverse effects on your users' experience. Additional measures might be required for particular workloads, load balancers and setups.

Verifying everything is connected and running

  1. Open http://<elastic IP of EWS> in browser;

  2. Open http://<elastic IP of EWS>/jmssender in browser;

    1. Check all cluster nodes are logging a message per every jmssender request (no matter which node jmssender is opened from);

  3. ssh -L7654:localhost:7654 <elastic IP of EWS> ; open http://localhost:7654/mod_cluster-manager in browser.

If these are to be production instances, add the following text to the contents of the User Data field, to ensure that security updates are applied on boot: 

yum -y update
JBoss.org Content Archive (Read Only), exported from JBoss Community Documentation Editor at 2020-03-11 09:48:22 UTC, last content change 2012-05-22 15:09:31 UTC.