This page is to work on our next version of a storage system for call time and response time data.

Requirements

• Hierarchical data of a limited depth

• Limited attached summed numeric data

• Scalable to default enablement

• Time series display of key indicators

The intention is to enhance the call time to be a superset of its current functionality. This would support data collection scenarios such as workflow instrumentation or the experimental byteman plugin grabing data at specific tiers. The hierarchy would support a simplified trace model (not intended for full traces, but for critical component traces). The most obvious examples would the http request path, the service component layers (seam, ejb3), and the data access tier (JPA, JDBC, Hibernate). This would cover the primary tiers necessary for production KPI monitoring.

The attached data would be numerical summable data across a set of transactions. Examples include the number of requests of a given http response code, the number of rows returned from a jdbc call, etc.

Scalability requirements mean this has to be low overhead to the instrumented application and have minimal storage and lookup impact on the server. Initially targeting a nominal capacity for long term running storage of 1000 monitored instances. This would include data purge and potential compression if feasible.

This system should support integration with alerting system for the most power.

System Differences

One major change from the old system that this implies is detaching the call time metrics from each separate resource so they can be collected at the application level into the hierarchy. The need for a new collection system that supports tracing through different parts of the stack requires this model and I don't believe we lose much. In fact it may be an improvement to pull this data up to a more visible location. This model would allow stack tracing of any JVM based plugin as well. 

Data Example

• Tier: HTTP request

  • Seam Compoent

    • EJB3 SLSB

      • JPA

        • JDBC

          Tier	Context	Key
          HTTP Request		URL Path	Timings	Success/Fail
          Seam		component.method	"	"
          EJB3 SLSB		component.method	"	"
          JPA	CallPath (class.method)	JPQL	"	"	rows
          JDBC	CallPath (class.method)	SQL	"	"	rows

          The above shows the information collected at each tier of the app. Collection would be hiearchical such that you can trace a give SQL and its timings to each of the HTTP requests that caused it. Assuming the collection of any given level could be enabled/disabled via the byteman collection system. Also assume it could be configured to collect arbitrary method executions via byteman rules so that further integration can be supported for any component technology or implementation strategy. Each tier would manage tracking a set of UIDs for that VM that match to the context of the call. This would allow for  data surrogate key at any tier to be its parent context id + context id. context id's would be main globaly unique and mapped for reduced data collection impact and via dictionary that is consistent for VM execution. 

Instrumented Classes

• HttpServlet

• EJB3 (@Stateless, @Stateful)

• JDBC (statement)

• JPA (query)

• HttpUrlConnection

Standard Deviations

In order to offer standard deviation information on the response times, I want to enhance the the bucket storage and transfer entities to add the necessary fields for a running variance calculation. For all location buckets it would mean storing the current counts, the mean and the moment (sum of the squares of the differences to the mean). We can store these in our typical bucket model and merge them to show deviations in larger bucketed times.

Persistence Options

There are a few options on persistence for this data. Historically, we've stored it in the db, like other metric data. It's been a two table system of keys and time series values for those keys. This system has been reasonable so far, but we've not performed a ton of scalability testing on it. The new features will add more load on the system and so may require alternate thinking. We'll want to be able to show time series data for specific tracepoints or traces across calculated time periods with bucket merging. And query against min/max/counts/std-dev, etc as part of reporting and data viewing. We'll also want merged views of the traces across cluster and then variance comparisons between them.

Like elsewhere we could consider RRD storage models instead, but the querying and on the fly calculation we're talking about here would likely suffer as compared to relation storage. Like the rest of the metric system, partitioned relational storage is also a viable option. To start with, I plan to perform some testing and analysis to see what the data levels will be and where the performance edges are for storage. 

In Memory Considerations

Key to this system will be the ability to correlate the instrument collected data and get it out of VM without impacting the monitored application too much. I'm going to put the result of some high level testing here to calculate some usage scenarios using the rhq code base. 

Data

• URLs: 230 url paths (gwt UI), Average path length is 83 chars, storage sum is 19515 chars

• ejb methods: 385 tracked so far, Avg method name is 23 chars,   sum is 8826 chars  (but this will need to become FQNs)

• Hibernate queries: 450 tracked so far, avg length 240 chars,  sum is 107377

I don't yet have data on the number of paths through this info, but with contexts there could be many thousands of points monitored which would result in a lot of data tracked in memory just to keep the key mappings. Points that match to code points don't need their context's stored separately (i.e. we don't need to store the method name and can statically track a surrogate temporary lookup key). But for the queries, we need some way to match and aggregate them. In order to reduce the data we could compress the keys or more likely store a hash of them such as SHA-1 which would reduce the storage to 40 bytes per key. This would reduce the in memory mappings by about 1/6th as tested, but result in the additional cpu overhead from hashing. Any hashing or lookup model would also require a model to match the lookups as stored somewhere outside the vm.

Another alternative is to reduce the stored data by tracking more limited data. This could be reduced to info such as the primary table in a query and the type of query. This wouldn't give detailed data access info, but could at least let you know that you selected from table x 20 times in a transaction. This would incur the cpu overhead of parsing the data.

The other primary data in memory would be the metrics. This could either be continuously tracked and read / reset by the client for bucketing, or be bucketed in VM and then output to disk for reading by the client.