Utilize the DynaGroups feature by setting up GroupDefinitions that can automatically group resources with certain attributes. Set an update period to have the groups automatically recalculate themselves.
JBossAS Clusters
Use the template definition for "JBoss AS Clusters in the system" to automatically create a separate group for each set of jboss nodes in a cluster.
Down Resources
Create a handy group of down resources with the "All Down Resources" template.
JBossAS Server : JVM Free Memory
Alert on this getting < 10MB to indicate the app server may be about to go out of memory.
Data Source: Available Connections
Alert on this getting too low, to know when your app is about to run out of free database connections
Tomcat Connector: Error Count
Alert on this being > 0 to find out when your web apps are responding with errors to requests. See also 'Web Application (WAR) : Errors while processing' for information on individual web apps.
Tomcat Connector: Maximum Request Time
Alert on this going above say 15sec, to indicate your web apps are running slow. See also 'Web Application (WAR) : Max. Servlet Resp. Time' for information on individual web apps.
Tomcat Connector: Request count per Minute
Monitor this to see how load on your app changes during the day/week. See also 'Web Application (WAR) : Requests served per Minute' for information on individual web apps.
EJB3 Session Bean : Method Invocation Time
Collect the minimum, maximum, and average invocation times for each of the methods exposed by an EJB. Useful to monitor responsiveness of critical backend systems, e.g. EJBs calling into DBs or EISs.
JMS Queue : Messages in Queue
Create an alert with a dampening rule on this metric to see if messages are backing up in the queu. Look at JMS Topic : All Message Counter' for the equivalent for topics.
Hibernate : viewQueries operation
Schedule this operation and review its output regularly to determine the most used/expensive SQL in your hibernate app.
Management invoker security
always secure invoker endpoints
configuration
symlinking is common for externalizing configs (mostly xml), makes
upgrading easierparametrized configurations by further externalizing environment variables
how to leverage check config against init files
application management
use deploy dir or farm, sometimes symlinks to common deploy dir company
wide
turn autodeploy off so that accidental touches to files do not cause
redeploys
application upgrade - side-by-side or rolling restart, scripted solution
and coordinate with load balancer
running out of file handles
http threads, work thread, conn pools
incomplete deployments / undeployments
running out of sockets
lock contention / jvm deadlock detection
run out of jdbc connections
slow memory leaks
load (for physical machine) and latency
lockup detection
determine final/last state just before failure: heap, seg fault, remote
restart, system was rebooted, etc
have "test" application that you can hit in an isolated format to verify
top-to-bottom health of connection from http entry to db and back
delta analysis - highlight metric spikes to signify unusual /
unanticipated things
rate of change analysis - acceleration of metric value deltas over time
load simulation against some app
look for error counts, number of times so-and-so occurred
transaction rollback messages
inability to connect to database errors
any network timeout issues, any
lower-level issues such as syslog messages, kernel panic, daemons dying
log / list of remote JMX commands sent to app server