CompatibleGroupId::ResourceTypeId1:ResourceKey1::ResourceTypeId2:ResourceKey2:: ... ::ResourceTypeIdN:ResourceKeyN
"Cluster" monitoring is an extension to what the present monitor tab provides for a compatible group. Cluster support falls into a few buckets:
-
Identify a set of resources that form a logical cluster and aggregate them
-
Create a compatible group of Apache, Tomcat, or JBAS instances manually or via DynaGroups
-
All cluster management requirements can be satisfied by enhancing the view/display of data for a compatible group. So, any compatible group can be managed as a cluster as appropriate.
-
There will not be a separate cluster/non-cluster view for a compatible group, a single new layout will serve all purposes.
-
The mock-up of the new, cluster group enabled, layout for compatible groups can be seen here.
-
-
-
-
Cluster Hierarchy (i.e. Parallel Hierarchy)
-
A compatible group will be viewed in the left navigation area as a hierarchy, analogous to a resource hierarchy. But, the root of the hierarchy is the compatible group (a set of resources of the same resource type). The hierarchy will still be broken down by category. But where a resource hierarchy generates autogroups of child resources with the same resource type, the cluster hierarchy generates autoclusters of resources with the same resource type and resource key. The autocluster contains resources with different, but analogous ancestry. Each resource in the autocluster will have as its oldest ancestor a resource in the root compatible group. The root compatible group is also now known as the cluster group.
-
For example, a compatible group of Postgres Databases would autocluster all Table resources with the same table name. This allows you to manage like tables across the databases in your group, without having to generate an explicit group of tables.
-
Note that it is important that plugins wisely follow the guidelines for resource key generation. In this example, table name is the resource key for resource type Table.
-
-
-
allow navigation to group (any tab [operations, config history, alerts]) via cluster view (see "Navigation" item 4 below)
-
DO NOT SHOW compatible groups in the resource browser for autoclusters
-
however, under the covers, in order to cope with how group functionality has been written to date, it might require creating a compatible group lazily...but this group must be hidden from the UI
-
-
-
Group configuration enhancements (provide a convenient user interface mechanism for displaying and changing configuration differences across a group or autocluster)
-
aggregate configuration will enhance the current config UI component
-
homogeneous versus heterogeneous property value display
-
if all values of some property are the same, the "real" renderer is used and the appropriate UIInput control is displayed with the homogeneous value
-
otherwise, UIInput control is suppressed, and an informative msg is displayed telling the user to use the context menu to value and/or edit the values
-
-
context menu (when right-clicking any part of the table row)
-
quick edit
-
there exist a sub-context menu for each property value that allows setting of all properties across the group to the selected value
-
-
full edit
-
open new window / frame allow simultaneous editing of individual property values across the group or autocluster
-
does not support pagination - all resources should be displayed
-
table is vertical where first column is resource and second column is the value
-
value will ALWAYS use the "real" renderer and display the appropriate UIInput control for the given property being edited
-
-
restrictions
-
context-menu should only show up to N individual values; if there are more than N items in the group, maybe use "..." and have the user use the explicit "edit" option that pops open the full table
-
-
-
User interface markers
-
Rows with values that have been changed will be marked differently somehow - some sort of icon that reminds the user that specific properties have changed and haven't been saved yet
-
The user interface needs to render the value column as appropriate to match user interactions;
-
if the result, after some user interaction, is that the property values are homogeneous, then the "real" renderer must be used;
-
likewise, if the result, as some user interaction, is that the property values are heterogeneous, then the renderer must be suppress and the informative msg displayed instead
-
note to implementors: this can be ajax (ideal user interaction) or a slower, full page reload (if it makes the implementation simpler)
-
-
-
Deprecations
-
get rid of override button, that was only important when we couldn't handle updating members in the group when they had heterogeneous configuration property value
mixed value columns well
-
-
-
Group Operation enhancements (provide a convenient user interface mechanism for scheduling operations across the resource cluster)
-
TBD
-
-
Navigation
-
Navigation mockups can be seen here.
-
Instead of binding the resource hierarchy to the monitor tab, break it out into it's own area
-
Left-hand, hierarchical navigation (horizontal real estate is not an issue)
-
Traits will be shown on the RHS of this navigational element (as opposed to above it, spanning the width of the UI like they do in 1.1 and earlier)
-
-
javascripty expand/collapse
-
show 1-level of children only, and all nodes at that level are collapsed by default
-
using navigational controls (opening/closing children nodes) does not change the page context; clicking on node data (resource name, auto-group name, cluster name) will change the page context (i.e., full page reload)
-
note: there is no restriction on how many levels of data or even how many siblings can be expanded simultaneously
-
-
For compatible groups - it would be the Cluster hierarchy
-
Root node is the compatible group itself, and children descend from there
-
-
For single resources - it would be a hierarchy of autogroups
-
Root node is always the associated platform (i.e., no matter how deeply nested the resource, for the sake of consistency always display the hierarchy up to the platform)
-
-
Common requirements
-
Aim for UI consistency between resource view (autogroups) and cluster view (parallel hierarchies)
-
Right-click context menu
-
MICA icons embedded, for quicker navigation to the appropriate subtab of the cluster and/or autogroup
-
Metrics > Add Chart (from schedule list)
-
Inventory > Create resource (from res type list)
-
Inventory > Manually add resource (from res type list)
-
Inventory > Delete resource (harder, since it requires selecting from a potentially large set)
-
Operations > Execute (from ops list)
-
Alert > Create new
-
Content > Deploy
-
-
Use a visual marker to show which node in the hierarchy you're actively viewing (alternate font style or size, row highlighting, blink tag
)
-
-
-
Cluster Control / Manipulation - which cluster files do we expose through JON Configuration UI
-
for JBoss resources:
-
JGroups config
-
ClusterPartition (extends / embedded a JGroups config - defines the "cluster")
-
HASessionStateService (for SFSB / session repl)
-
HANamingService
-
UnifiedInvokerHA
-
JGCacheInvalidationBridge (for distrib cache invalidation)
-
-
Detailed Design
Cluster Key
Each node in the cluster hierarchy is an autocluster, a compatible set of resources with the same resource type and resource key. The cluster key identifies the resource membership of the autocluster. Its definition is recursive in that the membership of any autocluster is dependent on its parent autocluster. This allows for resources with physically different resource ancestry to be identified as part of the same autocluster.
The autocluster should not contain a compatible group name as this name can change. It should use the compatible group id. Furthermore, a ResourceTypeId is a suitable replacement for a PluginName|ResourceTypeName combination and may also provide name change safety. Although not human readable, defining the cluster key using ids provides a safer, shorter key, which can still be converted easily to a human readable format.
So, the cluster key String format is:
Every autocluster has minimal depth or 1, depth 0 would just be the compatible group, which is an explicit resource cluster, not an autocluster.
Expressed recursively the autocluster key format is:
Depth-1
CompatibleGroupId::ResourceTypeId1:ResourceKey1
Depth-N
ParentClusterKey::ResourceTypeIdN:ResourceKeyN
An example can be seen here. This cluster hierarchy has potential ambiguity but the cluster key uniquely identifies the appropriate resources (Note, the example uses a human readable form, for clarity). Although expressed recursively above, the fully expanded key will be stored when necessary. So, a single fetch and get will make the cluster key available. This also allows us to store an autocluster anywhere in the hierarchy, regardless of whether parents have been stored.
Although maximum sized names would limit the hierarchy depth able to be stored in a standard varchar2 it is believed that average resource key lengths and hierarchy depths are such that the size limit will be reasonable. Also, see Store Resource Keys in There Own Table
Query
Given the key, it should typically take a single query to determine autocluster membership.
Although a subquery approach would allow for larger resource cardinality, the limit on IN clause is most likely not going to be an issue.
AutoCluster Persistence
The query built from a cluster key may return different group membership on different evaluations. The membership is dynamic given that resources can come and go anywhere in the cluster hierarchy. A dynamic view of the data is most likely preferable and when navigating and viewing the cluster hierarchy membership will be computed as needed. As such autoclusters will not, in general, be persisted. But, there are times when the autocluster needs to be persisted. Specifically, when an action is performed against the autocluster that generates history. In this case that history relates specifically the the group membership at that time.
Since an autocluster is a resource group it is intended that the ResourceGroup entity be enhanced to be able to support the needs of an autocluster. The backing ResourceGroup is implicit and not visible via the GUI. It exists under the covers to coordinate generation of group history, whether it be configuration change or operation execution of the autocluster. It will be a compatible group. The ResourceGroup entity will need to be enhanced with:
-
An optional FK to the Cluster Group (the root Compatible Group) since removal of the Cluster should remove all associated autoclusters.
-
A cluster key field. (max varchar2). This defines the autocluster and will be unique among all backing groups. NOTE - this will be stored as the ResourceGroup name, not a new field.
-
A visibility flag. This will be set false to ensure the backing group is not presented to the user.
There are two options when a backing ResourceGroup already exists at autocluster persist-time:
-
Create a new group for each persistence.
-
Re-use the existing backing group.
A new group on each persisted action doesn't serve much purpose. Although the group membership at the time of the action would be persisted, this is actually already reflected in the group history. The ResourceGroup keeps a list of group operation and group config history objects, each of those in turn maintain the list of relevant resource history, with reference back to the affected Resource. So, the membership snapshot is maintained in the history itself.
This begs the question as to whether the backing group needs to maintain the resource membership at all, or whether it is simply a container for group history. Since the group membership is never displayed it may not be necessary to maintain it, since the membership will be computed dynamically at each persist action. If group membership is persisted it would be updated at persist-time to reflect the current membership.
Other
Related Changes / Things to Consider
change the subsystem tabs
-
use text, not images for tabs and sub-tabs
-
get rid of sub-sub-tabs
-
monitor: graphs (default, used to be called indicators), metric data, configuration
-
events (new tab, in between monitor and inventory?): history (default, used to be called events), timeline
don't make UI bounce around
-
-
keep subsystem tabs (config, ops, alerts) visible, but gray and unclickable
-
always keep the same options in the context menus, but gray and disable them
FAQ
Do we need to handle singleton services across the cluster, where only one resource in the cluster will report will metrics at any given point in time and the others return 0 and/or appear down?
No clustered view will naturally take care of this; the cluster view will show aggregate metrics thus hiding the 0s, but you can still navigate to each resource to see how the singleton leader rotated around the cluster over time
Do we need A broader interpretation of "Cluster"
There has been some discussion around our current notion of clustered resources not adequately expressing an end user view or expectation of a cluster. It may be semantics but the question is whether we are appropriately using the term cluster. In the example a user's idea of cluster management may extend past the navigation and grouping of like resource instances. But will include also related resources, of different types. For example, an AS may always be tied to a specific apache instance. And those entities must be considered together when truly managing a cluster of AS resources.
Today, mixed groups addresses certain aspects of the issue and this design for clustered resources should be complementary. We may still need to go further to cohesively present what a user may see as a logical cluster of resources, but that is currently out of scope and most likely will be based on user feedback.
Below is proposed alternate terminology that we've rejected. The notion of clustered resources and the current terminology will stand going forward.
|
current name |
proposed name |
Notes |
|
ClusterGroup |
Compatible Group |
Just continue to call this a Compatible Group. If the user wants to name it "XXX Cluster" that's up to them |
|
AutoCluster |
InstanceGroup |
grouping like resource instances, same key |
|
|
AutoGroup |
the interpretation would be context specific. Typed AutoGroup vs Keyed or Instance AutoGroup |
|
ClusterKey |
Instancekey |
|
|
|
GroupKey |
For group resource config, Can we edit a list of maps? a la events in the AS plugin config.
This should be possible. Meaning, we need to be able to incorporate editing of this type into a single page that is actively editing other property types as well.
abridged from dev discussion: "Today on the main view/edit we only display an abridged form of the maps, which only displays the member props w/ summary="true". So, drilling down to the map page not only lets you edit the map, but also see all member props as well as descriptions for the member props. We should be able to show multiple map entries on the same page. So, instead of trying to compare map to map maybe we do map-element to map-element by name so the detail page pop-up would be a list of map-elements from the same relative maps across resources in question for the cluster group."
(3:05:36 PM) joseph: right, but there's nothing preventing us from showing multiple map entries on the same page, right
(3:05:43 PM) joseph: so instead of trying to compare map to map
(3:05:44 PM) fmoquete left the room (quit: Ping timeout: 622 seconds).
(3:05:50 PM) joseph: maybe we do map-element to map-element by name
(3:06:21 PM) joseph: so the detail page pop-up would be a list of map-elements from
Is it necessary to check for group membership change at config update commit time?
No. It is possible, although unlikely, that autocluster membership could change in the database after the resource set is presented in the GUI and before actions, like config update, take place. This is initially going to be acceptable, only the presented resources will be involved in the update. It may be required in the future to be more proactive about informing the user of membership changes for manipulated clustergroups , either at action time or between visits to manipulated autoclusters, but this is not seen as a requirement at this time.
Out of Scope / Future or Nice to Have
of-the-box experience
-
Add some DynaGroup templates to rhq-plugin.xml for various plugins with well-known elements that can group resources into logical clusters (for example, JBossAS can be aggregated by the PartitionName attribute of the ClusterPartition service/mbean)
-
Or, instead of extending the plugin descriptor, add raw row records to the default data inserted by dbsetup
-
If we had DynaGroup auto-calculation / auto-recalculation, this would be a more compelling feature
-
Cluster Hierarchy
Give the user some way to create a compatible group from the autocluster view
The question here is whether the resulting compatible group would also be a backing ResourceGroup, with visibility on, or whether this would just be a convenience for generating a standard compatible group seeded with the current members of the clustergroup. The latter would be very clear, making a backing group visible could have some subtle issues around group membership:
-
should group membership be managed by the system, the user, or both?
Store Resource Keys in There Own Table
By moving Resource.resourceKey values out of Resource and into its own value table we could:
-
Improve autocluster query performance.
When navigating the cluster hierarchy we must determine the autoclusters at depth N+1 when are current view is depth N. To do this we must query against the children of all resources at Depth N, grouping by resource type and resource key. Long textual keys are (we expect) inefficient for this query. Integer values would speed things up. -
Eliminate the danger of a cluster key extending past its storage capacity. See the following analysis (provided by jmarques)
if we use the schema we have today, then the resourceKeys can be very long. in fact, they can be extremely long. i'm actually thinking that max length might not be enough in all cases when we're talking about a deeply nested auto-cluster.
maybe we could break resource key out into it's own table (separate from the rhq_resource table). it would have a OneToOne map back to the Resource entity, and then we'd have the luxury of using the resourceKeyId (an auto-generated seq number) instead of a long key. this would give us deterministic size of the name of auto-clusters:
= groupId :: upToSixOf ( reosurceTypeId + : + resourceKeyId + :: )
= sizeof(groupId) + 2 + 6*( sizeof(resourceTypeId) + 1 + sizeof(resourceKeyId) + 2 )
= sizeof(number) + 2 + 6*( sizeof(number) + 1 + sizeof(number) + 2 )
= sizeof(number) + 2 + 6*( 2*sizeof(number) + 3 )
= sizeof(number) + 2 + 12*sizeof(number) + 18
= 13*sizeof(number) + 20
today we only use int for the ids (as opposed to long), so we have a magnitude of ~2B, which is 10 decimal places. so we have 13*10 + 8 or a required width of 150 characters - very reasonable. even if we were someday to update the entities to use long (263) that's only 19 places as a decimal, or 13*19 + 8 = 267 characters. still very reasonable.
We have decided to push this out initially since there may be some subtle issues given that resource keys are widely used. Still, the risk should be acceptable for future consideration.
Group Config
fine-grained cancel
support "revert" option for individual rollback to properties that have changed, but not yet persisted