CompatibleGroupId::ResourceTypeId1:ResourceKey1::ResourceTypeId2:ResourceKey2:: ... ::ResourceTypeIdN:ResourceKeyN
"Cluster" monitoring is an extension to what the present monitor tab provides for a compatible group. Cluster support falls into a few buckets:
Identify a set of resources that form a logical cluster and aggregate them
Create a compatible group of Apache, Tomcat, or JBAS instances manually or via DynaGroups
All cluster management requirements can be satisfied by enhancing the view/display of data for a compatible group. So, any compatible group can be managed as a cluster as appropriate.
There will not be a separate cluster/non-cluster view for a compatible group, a single new layout will serve all purposes.
The mock-up of the new, cluster group enabled, layout for compatible groups can be seen here.
Cluster Hierarchy (i.e. Parallel Hierarchy)
A compatible group will be viewed in the left navigation area as a hierarchy, analogous to a resource hierarchy. But, the root of the hierarchy is the compatible group (a set of resources of the same resource type). The hierarchy will still be broken down by category. But where a resource hierarchy generates autogroups of child resources with the same resource type, the cluster hierarchy generates autoclusters of resources with the same resource type and resource key. The autocluster contains resources with different, but analogous ancestry. Each resource in the autocluster will have as its oldest ancestor a resource in the root compatible group. The root compatible group is also now known as the cluster group.
For example, a compatible group of Postgres Databases would autocluster all Table resources with the same table name. This allows you to manage like tables across the databases in your group, without having to generate an explicit group of tables.
Note that it is important that plugins wisely follow the guidelines for resource key generation. In this example, table name is the resource key for resource type Table.
allow navigation to group (any tab [operations, config history, alerts]) via cluster view (see "Navigation" item 4 below)
DO NOT SHOW compatible groups in the resource browser for autoclusters
however, under the covers, in order to cope with how group functionality has been written to date, it might require creating a compatible group lazily...but this group must be hidden from the UI
Group configuration enhancements (provide a convenient user interface mechanism for displaying and changing configuration differences across a group or autocluster)
aggregate configuration will enhance the current config UI component
homogeneous versus heterogeneous property value display
if all values of some property are the same, the "real" renderer is used and the appropriate UIInput control is displayed with the homogeneous value
otherwise, UIInput control is suppressed, and an informative msg is displayed telling the user to use the context menu to value and/or edit the values
context menu (when right-clicking any part of the table row)
quick edit
there exist a sub-context menu for each property value that allows setting of all properties across the group to the selected value
full edit
open new window / frame allow simultaneous editing of individual property values across the group or autocluster
does not support pagination - all resources should be displayed
table is vertical where first column is resource and second column is the value
value will ALWAYS use the "real" renderer and display the appropriate UIInput control for the given property being edited
restrictions
context-menu should only show up to N individual values; if there are more than N items in the group, maybe use "..." and have the user use the explicit "edit" option that pops open the full table
User interface markers
Rows with values that have been changed will be marked differently somehow - some sort of icon that reminds the user that specific properties have changed and haven't been saved yet
The user interface needs to render the value column as appropriate to match user interactions;
if the result, after some user interaction, is that the property values are homogeneous, then the "real" renderer must be used;
likewise, if the result, as some user interaction, is that the property values are heterogeneous, then the renderer must be suppress and the informative msg displayed instead
note to implementors: this can be ajax (ideal user interaction) or a slower, full page reload (if it makes the implementation simpler)
Deprecations
get rid of override button, that was only important when we couldn't handle updating members in the group when they had heterogeneous configuration property value
mixed value columns well
Group Operation enhancements (provide a convenient user interface mechanism for scheduling operations across the resource cluster)
TBD
Navigation
Navigation mockups can be seen here.
Instead of binding the resource hierarchy to the monitor tab, break it out into it's own area
Left-hand, hierarchical navigation (horizontal real estate is not an issue)
Traits will be shown on the RHS of this navigational element (as opposed to above it, spanning the width of the UI like they do in 1.1 and earlier)
javascripty expand/collapse
show 1-level of children only, and all nodes at that level are collapsed by default
using navigational controls (opening/closing children nodes) does not change the page context; clicking on node data (resource name, auto-group name, cluster name) will change the page context (i.e., full page reload)
note: there is no restriction on how many levels of data or even how many siblings can be expanded simultaneously
For compatible groups - it would be the Cluster hierarchy
Root node is the compatible group itself, and children descend from there
For single resources - it would be a hierarchy of autogroups
Root node is always the associated platform (i.e., no matter how deeply nested the resource, for the sake of consistency always display the hierarchy up to the platform)
Common requirements
Aim for UI consistency between resource view (autogroups) and cluster view (parallel hierarchies)
Right-click context menu
MICA icons embedded, for quicker navigation to the appropriate subtab of the cluster and/or autogroup
Metrics > Add Chart (from schedule list)
Inventory > Create resource (from res type list)
Inventory > Manually add resource (from res type list)
Inventory > Delete resource (harder, since it requires selecting from a potentially large set)
Operations > Execute (from ops list)
Alert > Create new
Content > Deploy
Use a visual marker to show which node in the hierarchy you're actively viewing (alternate font style or size, row highlighting, blink tag
)
Cluster Control / Manipulation - which cluster files do we expose through JON Configuration UI
for JBoss resources:
JGroups config
ClusterPartition (extends / embedded a JGroups config - defines the "cluster")
HASessionStateService (for SFSB / session repl)
HANamingService
UnifiedInvokerHA
JGCacheInvalidationBridge (for distrib cache invalidation)
Each node in the cluster hierarchy is an autocluster, a compatible set of resources with the same resource type and resource key. The cluster key identifies the resource membership of the autocluster. Its definition is recursive in that the membership of any autocluster is dependent on its parent autocluster. This allows for resources with physically different resource ancestry to be identified as part of the same autocluster.
The autocluster should not contain a compatible group name as this name can change. It should use the compatible group id. Furthermore, a ResourceTypeId is a suitable replacement for a PluginName|ResourceTypeName combination and may also provide name change safety. Although not human readable, defining the cluster key using ids provides a safer, shorter key, which can still be converted easily to a human readable format.
So, the cluster key String format is:
CompatibleGroupId::ResourceTypeId1:ResourceKey1::ResourceTypeId2:ResourceKey2:: ... ::ResourceTypeIdN:ResourceKeyN
Every autocluster has minimal depth or 1, depth 0 would just be the compatible group, which is an explicit resource cluster, not an autocluster.
Expressed recursively the autocluster key format is:
Depth-1
CompatibleGroupId::ResourceTypeId1:ResourceKey1
ParentClusterKey::ResourceTypeIdN:ResourceKeyN
An example can be seen here. This cluster hierarchy has potential ambiguity but the cluster key uniquely identifies the appropriate resources (Note, the example uses a human readable form, for clarity). Although expressed recursively above, the fully expanded key will be stored when necessary. So, a single fetch and get will make the cluster key available. This also allows us to store an autocluster anywhere in the hierarchy, regardless of whether parents have been stored.
Although maximum sized names would limit the hierarchy depth able to be stored in a standard varchar2 it is believed that average resource key lengths and hierarchy depths are such that the size limit will be reasonable. Also, see Store Resource Keys in There Own Table
Given the key, it should typically take a single query to determine autocluster membership.
Although a subquery approach would allow for larger resource cardinality, the limit on IN clause is most likely not going to be an issue.
The query built from a cluster key may return different group membership on different evaluations. The membership is dynamic given that resources can come and go anywhere in the cluster hierarchy. A dynamic view of the data is most likely preferable and when navigating and viewing the cluster hierarchy membership will be computed as needed. As such autoclusters will not, in general, be persisted. But, there are times when the autocluster needs to be persisted. Specifically, when an action is performed against the autocluster that generates history. In this case that history relates specifically the the group membership at that time.
Since an autocluster is a resource group it is intended that the ResourceGroup entity be enhanced to be able to support the needs of an autocluster. The backing ResourceGroup is implicit and not visible via the GUI. It exists under the covers to coordinate generation of group history, whether it be configuration change or operation execution of the autocluster. It will be a compatible group. The ResourceGroup entity will need to be enhanced with:
An optional FK to the Cluster Group (the root Compatible Group) since removal of the Cluster should remove all associated autoclusters.
A cluster key field. (max varchar2). This defines the autocluster and will be unique among all backing groups. NOTE - this will be stored as the ResourceGroup name, not a new field.
A visibility flag. This will be set false to ensure the backing group is not presented to the user.
There are two options when a backing ResourceGroup already exists at autocluster persist-time:
Create a new group for each persistence.
Re-use the existing backing group.
A new group on each persisted action doesn't serve much purpose. Although the group membership at the time of the action would be persisted, this is actually already reflected in the group history. The ResourceGroup keeps a list of group operation and group config history objects, each of those in turn maintain the list of relevant resource history, with reference back to the affected Resource. So, the membership snapshot is maintained in the history itself.
This begs the question as to whether the backing group needs to maintain the resource membership at all, or whether it is simply a container for group history. Since the group membership is never displayed it may not be necessary to maintain it, since the membership will be computed dynamically at each persist action. If group membership is persisted it would be updated at persist-time to reflect the current membership.
use text, not images for tabs and sub-tabs
get rid of sub-sub-tabs
monitor: graphs (default, used to be called indicators), metric data, configuration
events (new tab, in between monitor and inventory?): history (default, used to be called events), timeline
keep subsystem tabs (config, ops, alerts) visible, but gray and unclickable
always keep the same options in the context menus, but gray and disable them
No clustered view will naturally take care of this; the cluster view will show aggregate metrics thus hiding the 0s, but you can still navigate to each resource to see how the singleton leader rotated around the cluster over time
There has been some discussion around our current notion of clustered resources not adequately expressing an end user view or expectation of a cluster. It may be semantics but the question is whether we are appropriately using the term cluster. In the example a user's idea of cluster management may extend past the navigation and grouping of like resource instances. But will include also related resources, of different types. For example, an AS may always be tied to a specific apache instance. And those entities must be considered together when truly managing a cluster of AS resources.
Today, mixed groups addresses certain aspects of the issue and this design for clustered resources should be complementary. We may still need to go further to cohesively present what a user may see as a logical cluster of resources, but that is currently out of scope and most likely will be based on user feedback.
Below is proposed alternate terminology that we've rejected. The notion of clustered resources and the current terminology will stand going forward.
current name |
proposed name |
Notes |
ClusterGroup |
Compatible Group |
Just continue to call this a Compatible Group. If the user wants to name it "XXX Cluster" that's up to them |
AutoCluster |
InstanceGroup |
grouping like resource instances, same key |
|
AutoGroup |
the interpretation would be context specific. Typed AutoGroup vs Keyed or Instance AutoGroup |
ClusterKey |
Instancekey |
|
|
GroupKey |
This should be possible. Meaning, we need to be able to incorporate editing of this type into a single page that is actively editing other property types as well.
abridged from dev discussion: "Today on the main view/edit we only display an abridged form of the maps, which only displays the member props w/ summary="true". So, drilling down to the map page not only lets you edit the map, but also see all member props as well as descriptions for the member props. We should be able to show multiple map entries on the same page. So, instead of trying to compare map to map maybe we do map-element to map-element by name so the detail page pop-up would be a list of map-elements from the same relative maps across resources in question for the cluster group."
(3:05:36 PM) joseph: right, but there's nothing preventing us from showing multiple map entries on the same page, right
(3:05:43 PM) joseph: so instead of trying to compare map to map
(3:05:44 PM) fmoquete left the room (quit: Ping timeout: 622 seconds).
(3:05:50 PM) joseph: maybe we do map-element to map-element by name
(3:06:21 PM) joseph: so the detail page pop-up would be a list of map-elements from
No. It is possible, although unlikely, that autocluster membership could change in the database after the resource set is presented in the GUI and before actions, like config update, take place. This is initially going to be acceptable, only the presented resources will be involved in the update. It may be required in the future to be more proactive about informing the user of membership changes for manipulated clustergroups , either at action time or between visits to manipulated autoclusters, but this is not seen as a requirement at this time.
Add some DynaGroup templates to rhq-plugin.xml for various plugins with well-known elements that can group resources into logical clusters (for example, JBossAS can be aggregated by the PartitionName attribute of the ClusterPartition service/mbean)
Or, instead of extending the plugin descriptor, add raw row records to the default data inserted by dbsetup
If we had DynaGroup auto-calculation / auto-recalculation, this would be a more compelling feature
The question here is whether the resulting compatible group would also be a backing ResourceGroup, with visibility on, or whether this would just be a convenience for generating a standard compatible group seeded with the current members of the clustergroup. The latter would be very clear, making a backing group visible could have some subtle issues around group membership:
should group membership be managed by the system, the user, or both?
By moving Resource.resourceKey values out of Resource and into its own value table we could:
Improve autocluster query performance.
When navigating the cluster hierarchy we must determine the autoclusters at depth N+1 when are current view is depth N. To do this we must query against the children of all resources at Depth N, grouping by resource type and resource key. Long textual keys are (we expect) inefficient for this query. Integer values would speed things up.
Eliminate the danger of a cluster key extending past its storage capacity. See the following analysis (provided by jmarques)
if we use the schema we have today, then the resourceKeys can be very long. in fact, they can be extremely long. i'm actually thinking that max length might not be enough in all cases when we're talking about a deeply nested auto-cluster.
maybe we could break resource key out into it's own table (separate from the rhq_resource table). it would have a OneToOne map back to the Resource entity, and then we'd have the luxury of using the resourceKeyId (an auto-generated seq number) instead of a long key. this would give us deterministic size of the name of auto-clusters:
= groupId :: upToSixOf ( reosurceTypeId + : + resourceKeyId + :: )
= sizeof(groupId) + 2 + 6*( sizeof(resourceTypeId) + 1 + sizeof(resourceKeyId) + 2 )
= sizeof(number) + 2 + 6*( sizeof(number) + 1 + sizeof(number) + 2 )
= sizeof(number) + 2 + 6*( 2*sizeof(number) + 3 )
= sizeof(number) + 2 + 12*sizeof(number) + 18
= 13*sizeof(number) + 20
today we only use int for the ids (as opposed to long), so we have a magnitude of ~2B, which is 10 decimal places. so we have 13*10 + 8 or a required width of 150 characters - very reasonable. even if we were someday to update the entities to use long (263) that's only 19 places as a decimal, or 13*19 + 8 = 267 characters. still very reasonable.
We have decided to push this out initially since there may be some subtle issues given that resource keys are widely used. Still, the risk should be acceptable for future consideration.
support "revert" option for individual rollback to properties that have changed, but not yet persisted