In the current RHQ schema all communication from Agent to Server is "payload only" meaning that e.g. for Configurations, the agent can only signal if a configuration change was successful or not. It is not possible to e.g. signal on top of this payload that the target WildFly server needs to be issued a "reload" command in order to make this configuration active.
There are actually other use cases that can make use of such a back channel.
The back channel does not make sense on its own by its pure existence, but needs to have receivers that can use the information sent. The following graphic illustrates the basic idea:
On between Agent and Server is a channel that forwards messages. "Everyone" in the Agent can put messages on that Backchannel, which get then forwarded to the server. This is shown in orange. Inside the server there are a number of receivers (magenta), that subscribe to such messages and that are then able to do further actions like storing the data in the database for existing subsystems to use. And/or there could be a push mechanism that is forwarding such a message to the UI to be displayed in the Message Center (Keep in mind though, that this is not enough in many cases, as many users do not use the UI on a daily basis, but rather the CLI, the REST-api or "headless".
When you look at the drawing, then this looks a lot like JMS where plugins can put ObjectMessage s into a JMS-Topic. On the server side Subscribers subscribe to the Topic and get all messages. With the help of a selector, they can then receive the messages they are interested in and act upon.
MAZZ: I would go a step further and examine if its possible to have ALL messages from the agent to the server go through this async message-queue style communications layer.
Plus, not only can the agent use this message Q to send messages, it could also use it to receive messages pub-sub style and thus allowing us to provide uni-directional comm and eliminate the need for direct server->agent communications. This has the benefit of not requiring all servers to be able to resolve all agent DNS hostnames and not requiring firewalls running on agents to have an agent port (16163) open.
Use cases revisited
AS7/WildFly "dirty state"
WildFly often does not directly apply changes to configuration attributes, but only when a :reload command is issued on the server. The reason is that one can prepare a few changes individually and then apply them all together (e.g. when installing an update of an installed application).
The downside in the current RHQ implementation is that there is no real way exists to let the user know that such a reload is required. We have a workaround that if you go on the current configuration subtab, this "dirty" state is displayed in the message bar.
This workaround is brittle on one side, as it requires the user to check a specific place to find out about the "dirty" state. Also this information is neither available over REST or in the CLI.
Event subsystem
Right now, when you add a new logfile as event source, there is no way of finding out if the logfile location is invalid (i.e. contains a valid path and no typo) or if the logfile just has no content and is never written to.
In use case, the setup of the Logfile tailer can check for the existence of the file and send a warning back if the file does not exist. The warning could perhaps be more detailed to include if the directory exists or not
New Logfile in WildFly
When you set up a new Log file handler in WildFly, the user needs afterwards go to the Server resource and enable tailing of the logfile there, which may result in all the issues shown in the previous point.
With the backchannel, the AS7 plugin can signal that such a new logfile handler has been added and an actor on the server can add this specific logfile as Event source for tailing.
Changes in Plugin-Api
The plugin api would get an additional method sendMessage() that would get the message passed along with an identifier, that will work as selector on the receiving side. The method itself needs to be asynchronous.
MAZZ: We should not expose server-side, remote APIs to plugins directly. This has always been frowned upon and we never do this (in fact, plugins today have no way of doing this - they don't have the remote sender to send info directly to the server). Rather, what you will want to do is expose some API on, for example, the resource context which itself should provide an API to call into the plugin container. The plugin container is the only thing that should be sending messages to the server. This way, we have one place (the plugin container) that can dictate how and when messages actually get sent up to the server, get retried on failures, get spooled when necessary, get delayed due to concurrency limits, etc.
Changes in Server
On the server side a mechanism to subscribe to those messages need to be provided. If we were using JMS for Agent-Server comms, this could be implemented by standard MessageDrivenBeans (MDB), that get an annotation denoting the selector.
It will be allowed that more than one actor can get messages with the same selector. Order of deliver is undefined in this case.
If the same message is sent over and over again (e.g. after each interaction with a "dirty" Wildfly instance, it is up to the individual actor on how to react. Sending the same message twice in a row is explicitly allowed.
Changes in selected plugins
Those need to be broken out into sub-pages that are more detailed on message format and other semantics
as7 plugin
Whenever a command to AS7 returns with a "reload required" is returned, the plugin would put such a "dirty" message that includes the resource id, on the backchannel. We may opt to store the state and only send the message after state change or every n-th time to reduce the sending overhead. When the managed WildFly server then no longer reports that reload-needed flag, the plugin needs to send a "clean" message.
RHQ Server side we'd store the "dirty" state along the resource so that it can be retrieved via UI, REST and CLI.