There are going to be environments where the servers cannot always talk to all the agents (but all agents can talk directly to the servers). Usually it is because the agents all sit behind firewalls and the servers cannot tunnel through those firewalls.
It would be nice to be able to support managing these kinds of networks.
The general solution will need to involve the agent's comm layer periodically polling the server and pulling down messages that the server wants to deliver. The server will need to queue up messages it wants to send to agents and only when the agents "check in" with the server will those messages be delivered.
There are many problems that need to be solved in order to provide this functionality.
There will be no such thing as "synchronous" or "rpc" like calls from server to agent. For example, the server will no longer be able to "ping" the agent synchronously via an RPC-like call to the remote agent. The server will have to assume the agent is available and not rely on a ping command to determine this. Any message a server wants to send to an agent must be asynchronous and that message must be queued up in a agent queue. There might be (I'm pretty sure there are) instances today where the server must make an RPC call to the agent because the server needs to know the return value of the RPC call. We need to figure out a way to now require this.
The number of agents may require a large amount of memory or diskspace on the server. If a single server is dealing with hundreds or thousands of agents, it effectively has hundreds or thousands of agent queues potentially filled with hundreds (or more) messages. We need to make the storage and retrieval of these queued messages efficient and fast.
We need fault tolerance if server goes down - which will probably mean a way to persist the queued messages to avoid losing those messages in the event the server goes down.
Today, the servers and agents require bi-directional flow - there are cases when the server needs to initial a workflow or continue a workflow that requires it to send a message to the agent. We need to design the message workflow to avoid this.
This agent queue mechanism must work in HA. If there are 2 servers in the cloud, and one server had messages for agent N but that server goes down before agent N could retrieve the, the agent will need to failover to the second server in the cloud but stil manaage to retrieve the messages that were to be delivered by the first server.
The agent needs to add this capability to periodically poll the server for messages. Today, the agent does poll the server, but it only does this to check the server's availability. We'd have to add something similar for the ability to pull down agent messages from the server.