System properties

To set a system property, pass its new value using the standard jvm -Dkey=value options:

$JBOSS_HOME/bin/standalone.sh -Djboss.home.dir=some/location/wildFly \
    -Djboss.server.config.dir=some/location/wildFly/custom-standalone

This command starts up a standalone server instance using a non-standard AS home directory and a custom configuration directory. For specific information about system properties, refer to the definitions below.

Instead of passing the parameters directly, you can put them into a properties file, and pass the properties file to the script, as in the two examples below.

$JBOSS_HOME/bin/domain.sh --properties=/some/location/jboss.properties
$JBOSS_HOME/bin/domain.sh -P=/some/location/jboss.properties

Note however, that properties set this way are not processed as part of JVM launch. They are processed early in the boot process, but this mechanism should not be used for setting properties that control JVM behavior (e.g. java.net.perferIPv4Stack) or the behavior of the JBoss Modules classloading system.

The syntax for passing in parameters and properties files is the same regardless of whether you are running the domain.sh, standalone.sh, or the Microsoft Windows scriptsdomain.bat or standalone.bat.

The properties file is a standard Java property file containing key=value pairs:

jboss.home.dir=/some/location/wildFly
jboss.domain.config.dir=/some/location/wildFly/custom-domain

System properties can also be set via the xml configuration files. Note however that for a standalone server properties set this way will not be set until the xml configuration is parsed and the commands created by the parser have been executed. So this mechanism should not be used for setting properties whose value needs to be set before this point.

Other command line parameters

The first acceptable format for command line arguments to the WildFly launch scripts is

--name=value

For example:

$JBOSS_HOME/bin/standalone.sh --server-config=standalone-ha.xml

If the parameter name is a single character, it is prefixed by a single '-' instead of two. Some parameters have both a long and short option.

-x=value

For example:

$JBOSS_HOME/bin/standalone.sh -P=/some/location/jboss.properties

For some command line arguments frequently used in previous major releases of WildFly, replacing the "=" in the above examples with a space is supported, for compatibility.

-b 192.168.100.10

If possible, use the -x=value syntax. New parameters will always support this syntax.

The sections below describe the command line parameter names that are available in standalone and domain mode.

Controlling the Bind Address with -b

WildFly binds sockets to the IP addresses and interfaces contained in the <interfaces> elements in standalone.xml, domain.xml and host.xml. (See Interfaces and Socket Bindings for further information on these elements.) The standard configurations that ship with WildFly includes two interface configurations:

<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
    </interface>
    <interface name="public">
       <inet-address value="${jboss.bind.address:127.0.0.1}"/>
    </interface>
</interfaces>

Those configurations use the values of system properties jboss.bind.address.management and jboss.bind.address if they are set. If they are not set, 127.0.0.1 is used for each value.

As noted in Common Parameters, the AS supports the -b and -b<name> command line switches. The only function of these switches is to set system properties jboss.bind.address and jboss.bind.address.<name> respectively. However, because of the way the standard WildFly configuration files are set up, using the -b switches can indirectly control how the AS binds sockets.

If your interface configurations match those shown above, using this as your launch command causes all sockets associated with interface named "public"  to be bound to 192.168.100.10.

$JBOSS_HOME/bin/standalone.sh -b=192.168.100.10

In the standard config files, public interfaces are those not associated with server management. Public interfaces handle normal end-user requests.

To bind the public interfaces to all IPv4 addresses (the IPv4 wildcard address), use the following syntax:

$JBOSS_HOME/bin/standalone.sh -b=0.0.0.0

You can also bind the management interfaces, as follows:

$JBOSS_HOME/bin/standalone.sh -bmanagement=192.168.100.10

In the standard config files, management interfaces are those sockets associated with server management, such as the socket used by the CLI, the HTTP socket used by the admin console, and the JMX connector socket.

<interface name="public">
   <nic name="eth0"/>
</interface>

Controlling the Default Multicast Address with -u

WildFly may use multicast communication for some services, particularly those involving high availability clustering. The multicast addresses and ports used are configured using the socket-binding elements in standalone.xml and domain.xml. (See Socket Bindings for further information on these elements.) The standard HA configurations that ship with WildFly include two socket binding configurations that use a default multicast address:

<socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
<socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>

Those configurations use the values of system property jboss.default.multicast.address if it is set. If it is not set, 230.0.0.4 is used for each value. (The configuration may include other socket bindings for multicast-based services that are not meant to use the default multicast address; e.g. a binding the mod-cluster services use to communicate on a separate address/port with Apache httpd servers.)

As noted in Common Parameters, the AS supports the -u command line switch. The only function of this switch is to set system property jboss.default.multicast.address. However, because of the way the standard AS configuration files are set up, using the -u switches can indirectly control how the AS uses multicast.

If your socket binding configurations match those shown above, using this as your launch command causes the service using those sockets configurations to be communicate over multicast address 230.0.1.2.

$JBOSS_HOME/bin/standalone.sh -u=230.0.1.2

Core Concepts

Wildfly introduces the ability to suspend and resume servers. This can be combined with shutdown to enable the server to gracefully finish processing all active requests and then shut down. When a server is suspended it will immediately stop accepting new requests, but wait for existing request to complete. A suspended server can be resumed at any point, and will begin processing requests immediately. Suspending and resuming has no effect on deployment state (e.g. if a server is suspended singleton EJB's will not be destroyed). As of Wildfly 11 it is also possible to start a server in suspended mode which means it will not accept requests until it has been resumed, servers will also be suspended during the boot process, so no requests will be accepted until the startup process is 100% complete.

Suspend/Resume has no effect on management operations, management operations can still be performed while a server is suspended. If you wish to perform a management operation that will affect the operation of the server (e.g. changing a datasource) you can suspend the server, perform the operation, then resume the server. This allows all requests to finish, and makes sure that no requests are running while the management changes are taking place.

When a server is suspending it goes through four different phases:

• RUNNING - The normal state, the server is accepting requests and running normally

• PRE_SUSPEND - In PRE_SUSPEND the server will notify external parties that it is about to suspend, for example mod_cluster will notify the load balancer that the deployment is suspending. Requests are still accepted in this phase.

• SUSPENDING - All new requests are rejected, and the server is waiting for all active requests to finish. If there are no active requests at suspend time this phase will be skipped.

• SUSPENDED - All requests have completed, and the server is suspended.

Starting Suspended

In order to start into suspended mode when using a standalone server you need to add --start-mode=suspend to the command line. It is also possible to specify the start-mode in the reload operation to cause the server to reload into suspended mode (other possible values for start-mode are normal and admin-only).

In domain mode servers can be started in suspended mode by passing the suspend=true parameter to any command that causes a server to start, restart or reload (e.g. :start-servers(suspend=true)).

The Request Controller Subsystem

Wildfly introduces a new subsystem called the Request Controller Subsystem. This optional subsystem tracks all requests at their entry point, which how the graceful shutdown mechanism know when all requests are done (it also allows you to provide a global limit on the total number of running requests). 

If this subsystem is not present suspend/resume will be limited, in general things that happen in the PRE_SUSPEND phase will work as normal (stopping message delivery, notifying the load balancer), however the server will not wait for all requests to complete and instead move straight to SUSPENDED mode.

There is a small performance penalty associated with the request controller subsystem (about on par with enabling statistics), so if you do not require the suspend/resume functionality this subsystem can be removed to get a small performance boost.

Subsystem Integrations

Suspend/Resume is a service provided by the Wildfly platform that any subsystem may choose to integrate with. Some subsystems integrate directly with the suspend controller, while others integrate through the request controller subsystem.

The following subsystems support graceful shutdown. Note that only subsystems that provide an external entry point to the server need graceful shutdown support, for example the JAX-RS subsystem does not require suspend/resume support as all access to JAX-RS is through the web connector.

• Undertow - Undertow will wait for all requests to finish

• mod_cluster - The mod_cluster subsystem will notify the load balancer that the server is suspending in the PRE_SUSPEND phase.

• EJB - EJB will wait for all remote EJB requests and MDB message deliveries to finish. Delivery to MDB's is stopped in the PRE_SUSPEND phase. EJB timers are suspended, and missed timers will be activated when the server is resumed. 

• Batch - Batch jobs will be stopped at a checkpoint while the server is suspending. They will be restarted from that checkpoint when the server returns to running mode.

• EE Concurrency - The server will wait for all active jobs to finish. All jobs that have already been queued will be skipped.

• Transactions - transaction subsystem waits for all running transactions to finish while server is suspending. During that time server refuses to start any new transaction. But any in-flight transaction will be serviced - e.g. it means that server accepts any incoming remote call which carries context of the transaction already started at the suspending server.
  When you work with EJBs you have to enable the graceful shutdown functionality by setting attribute enable-graceful-txn-shutdown to true.
  (at the ejb3 subsystem xml, for example):
  <enable-graceful-txn-shutdown value="false"/>
  By default graceful shutdown it's disabled for ejb subsystem.
  The reason is that the behavior might be unwelcome in cluster environments, as the server notifies remote clients that the node is no longer available for remote calls only after the transactions are finished. During that brief window of time, the client of a cluster may send a new request to a node that is shutting down and will refuse the request because it is not related to an existing transaction.
  If this attribute enable-graceful-txn-shutdown is set to false, we disable the graceful behavior and EJB clients will not attempt to invoke the node when it suspends, regardless of active transactions.

Standalone Mode

Suspend/Resume can be controlled via the following CLI operations in standalone mode:

:suspend(timeout=z)

Suspends the server. If the timeout is specified it will wait up to the specified number of seconds for all requests to finish. If there is no timeout specified or the value is less than zero it will wait indefinitely. 

:resume

Resumes a previously suspended server. The server should be able to begin serving requests immediately. 

:read-attribute(name=suspend-state)

Returns the current suspend state of the server.

:shutdown(timeout=x)

If a timeout parameter is passed to the shutdown command then a graceful shutdown will be performed. The server will be suspended, and will wait up to the specified number of seconds for all requests to finish before shutting down. A timeout value of less than zero means it will wait indefinitely. 

Domain Mode

Domain mode has similar commands as standalone mode, however they can be applied at both the global and server group levels:

Whole Domain

:suspend-servers(timeout=x)

:resume-servers

:stop-servers(timeout=x)

Server Group

/server-group=main-server-group:suspend-servers(timeout=x)

/server-group=main-server-group:resume-servers

/server-group=main-server-group:stop-servers(timeout=x)

Server 

/host=master/server-config=server-one:suspend(timeout=x)

/host=master/server-config=server-one:resume

/host=master/server-config=server-one:stop(timeout=x)

Managed Domain

In a managed domain the JVM settings can be declared at different scopes: For a specific server group, for a host or for a particular server. If not declared, the settings are inherited from the parent scope. This allows you to customize or extend the JVM settings within every layer.

Let's take a look at the JVM declaration for a server group:

<server-groups>
       <server-group name="main-server-group" profile="default">
           <jvm name="default">
               <heap size="64m" max-size="512m"/>
           </jvm>
           <socket-binding-group ref="standard-sockets"/>
       </server-group>
       <server-group name="other-server-group" profile="default">
           <jvm name="default">
               <heap size="64m" max-size="512m"/>
           </jvm>
           <socket-binding-group ref="standard-sockets"/>
       </server-group>
</server-groups>

_(See domain/configuration/domain.xml₎

In this example the server group "main-server-group" declares a heap size of 64m and a maximum heap size of 512m. Any server that belongs to this group will inherit these settings. You can change these settings for the group as a whole, or a specific server or host:

<servers>
       <server name="server-one" group="main-server-group" auto-start="true">
           <jvm name="default"/>
       </server>
       <server name="server-two" group="main-server-group" auto-start="true">
           <jvm name="default">
               <heap size="64m" max-size="256m"/>
           </jvm>
           <socket-binding-group ref="standard-sockets" port-offset="150"/>
       </server>
       <server name="server-three" group="other-server-group" auto-start="false">
           <socket-binding-group ref="standard-sockets" port-offset="250"/>
       </server>
</servers>

_{(See domain/configuration/host.xml)}

In this case, server-two, belongs to the main-server-group and inherits the JVM settings named default, but declares a lower maximum heap size.

[domain@localhost:9999 /] /host=local/server-config=server-two/jvm=default:read-resource
{
   "outcome" => "success",
   "result" => {
       "heap-size" => "64m",
       "max-heap-size" => "256m",
   }
}

Standalone Server

For a standalone sever you have to pass in the JVM settings either as command line arguments when executing the $JBOSS_HOME/bin/standalone.sh script, or by declaring them in $JBOSS_HOME/bin/standalone.conf. (For Windows users, the script to execute is %JBOSS_HOME%/bin/standalone.bat while the JVM settings can be declared in %JBOSS_HOME%/bin/standalone.conf.bat.)

JSON Formatter

The first thing that needs configuring is the formatter, we currently support outputting log records as JSON. You can define several formatters, for use with different handlers. A log record has the following format, and it is the formatter's job to format the data presented:

2013-08-12 11:01:12 - {
    "type" : "core",
    "r/o" : false,
    "booting" : false,
    "version" : "8.0.0.Alpha4",
    "user" : "$local",
    "domainUUID" : null,
    "access" : "NATIVE",
    "remote-address" : "127.0.0.1/127.0.0.1",
    "success" : true,
    "ops" : [JMX|WFLY8:JMX subsystem configuration],
        "operation" : "write-attribute",
        "name" : "enabled",
        "value" : true,
        "operation-headers" : {"caller-type" : "user"}
    }]
}

It includes an optional timestamp and then the following information in the json record

The json formatter resource has the following attributes:

Handlers

A handler is responsible for taking the formatted data and logging it to a location. There are currently two types of handlers, File and Syslog. You can configure several of each type of handler and use them to log information.

Logger configuration

The final part that needs configuring is the logger for the management operations. This references one or more handlers and is configured at /core-service=management/access=audit/logger=audit-log. The attributes for this resource are:

Then which handlers are used to log the management operations are configured as handler=* children of the logger.

Domain Mode (host specific configuration)

In domain mode audit logging is configured for each host in its host.xml file. This means that when connecting to the DC, the configuration of the audit logging is under the host's entry, e.g. here is the default configuration:

[domain@localhost:9990 /] /host=master/core-service=management/access=audit:read-resource(recursive=true)
{
    "outcome" => "success",
    "result" => {
        "file-handler" => {
            "host-file" => {
                "formatter" => "json-formatter",
                "max-failure-count" => 10,
                "path" => "audit-log.log",
                "relative-to" => "jboss.domain.data.dir"
            },
            "server-file" => {
                "formatter" => "json-formatter",
                "max-failure-count" => 10,
                "path" => "audit-log.log",
                "relative-to" => "jboss.server.data.dir"
            }
        },
        "json-formatter" => {"json-formatter" => {
            "compact" => false,
            "date-format" => "yyyy-MM-dd HH:mm:ss",
            "date-separator" => " - ",
            "escape-control-characters" => false,
            "escape-new-line" => false,
            "include-date" => true
        }},
        "logger" => {"audit-log" => {
            "enabled" => false,
            "log-boot" => true,
            "log-read-only" => false,
            "handler" => {"host-file" => {}}
        }},
        "server-logger" => {"audit-log" => {
            "enabled" => false,
            "log-boot" => true,
            "log-read-only" => false,
            "handler" => {"server-file" => {}}
        }},
        "syslog-handler" => undefined
    }
}

We now have two file handlers, one called host-file used to configure the file to log management operations on the host, and one called server-file used to log management operations executed on the servers. Then logger=audit-log is used to configure the logger for the host controller, referencing the host-file handler. server-logger=audit-log is used to configure the logger for the managed servers, referencing the server-file handler. The attributes for server-logger=audit-log are the same as for server-logger=audit-log in the previous section. Having the host controller and server loggers configured independently means we can control audit logging for managed servers and the host controller independently. 

The cancel-non-progressing-operation operation

The cancel-non-progressing-operation operation instructs the target process to find any operation that isn't proceeding normally and cancel it.

On a standalone server:

[standalone@localhost:9990 /] /core-service=management/service=management-operations:cancel-non-progressing-operation
{
    "outcome" => "success",
    "result" => "-1155777943"
}

The result value is an internal identification number for the operation that was cancelled.

On a managed domain host controller, the equivalent resource is in the host=<hostname> portion of the management resource tree:

[domain@localhost:9990 /] /host=host-a/core-service=management/service=management-operations:cancel-non-progressing-operation
{
    "outcome" => "success",
    "result" => "2156877946"
}

An operation can be cancelled on an individual managed domain server as well:

[domain@localhost:9990 /] /host=host-a/server=server-one/core-service=management/service=management-operations:cancel-non-progressing-operation
{
    "outcome" => "success",
    "result" => "6497786512"
}

An operation is considered to be not proceeding normally if it has been executing with the exclusive operation lock held for longer than 15 seconds. Read-only operations do not acquire the exclusive operation lock, so this operation will not cancel read-only operations. Operations blocking waiting for another operation to release the exclusive lock will also not be cancelled.

If there isn't any operation that is failing to proceed normally, there will be a failure response:

[standalone@localhost:9990 /] /core-service=management/service=management-operations:cancel-non-progressing-operation
{
    "outcome" => "failed",
    "failure-description" => "WFLYDM0089: No operation was found that has been holding the operation execution write lock for long than [15] seconds",
    "rolled-back" => true
}

The find-non-progressing-operation operation

To simply learn the id of an operation that isn't proceeding normally, but not cancel it, use the find-non-progressing-operation operation:

[standalone@localhost:9990 /] /core-service=management/service=management-operations:find-non-progressing-operation
{
    "outcome" => "success",
    "result" => "-1155777943"
}

If there is no non-progressing operation, the outcome will still be success but the result will be undefined.

Once the id of the operation is known, the management resource for the operation can be examined to learn more about its status.

Examining the status of an active operation

There is a management resource for any currently executing operation that can be queried:

[standalone@localhost:9990 /] /core-service=management/service=management-operations/active-operation=-1155777943:read-resource(include-runtime=true)
{
    "outcome" => "success",
    "result" => {
        "access-mechanism" => "undefined",
        "address" => [
            ("deployment" => "example")
        ],
        "caller-thread" => "management-handler-thread - 24",
        "cancelled" => false,
        "exclusive-running-time" => 101918273645L,
        "execution-status" => "awaiting-stability",
        "operation" => "deploy",
        "running-time" => 101918279999L
    }
}

The response includes the following attributes:

The following are the values for the execution-status attribute:

All currently executing operations can be viewed in one request using the read-children-resources operation:

[standalone@localhost:9990 /] /core-service=management/service=management-operations:read-children-resources(child-type=active-operation)
{
    "outcome" => "success",
    "result" => {"-1155777943" => {
        "access-mechanism" => "undefined",
        "address" => [
            ("deployment" => "example")
        ],
        "caller-thread" => "management-handler-thread - 24",
        "cancelled" => false,
        "exclusive-running-time" => 101918273645L,
        "execution-status" => "awaiting-stability",
        "operation" => "deploy",
        "running-time" => 101918279999L
    },
    {"-1246693202" => {
        "access-mechanism" => "undefined",
        "address" => [
            ("core-service" => "management"),
            ("service" => "management-operations")
        ],
        "caller-thread" => "management-handler-thread - 30",
        "cancelled" => false,
        "exclusive-running-time" => -1L,
        "execution-status" => "executing",
        "operation" => "read-children-resources",
        "running-time" => 3356000L
    }}
}

Canceling a specific operation

The cancel-non-progressing-operation operation is a convenience operation for identifying and canceling an operation. However, an administrator can examine the active-operation resources to identify any operation, and then directly cancel it by invoking the cancel operation on the resource for the desired operation.

[standalone@localhost:9990 /] /core-service=management/service=management-operations/active-operation=-1155777943:cancel
{
    "outcome" => "success",
    "result" => undefined
}

Controlling operation blocking time

As an operation executes, the execution thread may block at various points, particularly while waiting for the service container to stabilize following any changes. Since an operation may be holding the exclusive execution lock while blocking, in WildFly execution behavior was changed to ensure that blocking will eventually time out, resulting in roll back of the operation.

The default blocking timeout is 300 seconds. This is intentionally long, as the idea is to only trigger a timeout when something has definitely gone wrong with the operation, without any false positives.

An administrator can control the blocking timeout for an individual operation by using the blocking-timeout operation header. For example, if a particular deployment is known to take an extremely long time to deploy, the default 300 second timeout could be increased:

[standalone@localhost:9990 /] deploy /tmp/mega.war --headers={blocking-timeout=450}

Note the blocking timeout is not a guaranteed maximum execution time for an operation. If it only a timeout that will be enforced at various points during operation execution.

Snapshots

In addition to the backups taken by the server as described above you can manually take take snapshots which will be stored in the snapshot folder under the _xml_history folder, the automatic backups described above are subject to automatic house keeping so will eventually be automatically removed, the snapshots on the other hand can be entirely managed by the administrator.

You may also take your own snapshots using the CLI:

[standalone@localhost:9990 /] :take-snapshot
{
    "outcome" => "success",
    "result" => {"name" => "/Users/kabir/wildfly/standalone/configuration/standalone_xml_history/snapshot/20110630-172258657standalone.xml"}
}

You can also use the CLI to list all the snapshots

[standalone@localhost:9990 /] :list-snapshots
{
    "outcome" => "success",
    "result" => {
        "directory" => "/Users/kabir/wildfly/standalone/configuration/standalone_xml_history/snapshot",
        "names" => [
            "20110630-165714239standalone.xml",
            "20110630-165821795standalone.xml",
            "20110630-170113581standalone.xml",
            "20110630-171411463standalone.xml",
            "20110630-171908397standalone.xml",
            "20110630-172258657standalone.xml"
        ]
    }
}

 To delete a particular snapshot:

[standalone@localhost:9990 /] :delete-snapshot(name="20110630-165714239standalone.xml")
{"outcome" => "success"}

and to delete all snapshots:

[standalone@localhost:9990 /] :delete-snapshot(name="all")
{"outcome" => "success"}

In domain mode executing the snapshot operations against the root node will work against the domain model. To do this for a host model you need to navigate to the host in question:

[domain@localhost:9990 /] /host=master:list-snapshots
{
    "outcome" => "success",
    "result" => {
        "domain-results" => {"step-1" => {
            "directory" => "/Users/kabir/wildfly/domain/configuration/host_xml_history/snapshot",
            "names" => [
                "20110630-141129571host.xml",
                "20110630-172522225host.xml"
            ]
        }},
        "server-operations" => undefined
    }
}

Subsequent Starts

For subsequent server starts it may be desirable to take the state of the server back to one of the previously known states, for a number of items an abbreviated reverence to the file can be used:

In addition to this the --server-config parameter can always be used to specify a configuration relative to the jboss.server.config.dir and finally if no matching configuration is found an attempt to locate the configuration as an absolute path will be made.