Package com.bigdata.counters.store

This package provides a persistence mechanism for performance counters.

See: Description

Class Summary

Class	Description
CounterSetBTree	An API encapsulating for writing and querying counter sets.
CounterSetBTree.CounterSetBTreeTupleSerializer	Encapsulates key and value formation.
CounterSetBTree.Entry	A representation of a timestamped performance counter value as stored in the CounterSetBTree.
TestAll	Aggregates tests in dependency order.
TestCounterSetBTree

Package com.bigdata.counters.store Description

This package provides a persistence mechanism for performance counters. Counters can be stored at a granularity of up to one second and queried efficiently for a given time range, granularity, and filtered path. For example, a query could request the average value of IOWAIT for all hosts over the last 12 hours.

The counters are written into a BTree backed by a local file. The schema is:

    [minute][path][timestamp] : value
    

where minute is the floor of the minute for the timestamp associated with the counter value, where path is the fully qualified path of the counter, and where value is the datum reported for that counter value.

The keys can be easily compressed since many keys will share a common prefix. At the same time it is possible to store counters which are sampled more frequently than once per second. Finally, since the minutes appear first in the key, we can scan only the time range of interest. Of course, when scanning a large period of time we will still have to page through a lot of data.

When querying, we materialize a view of the time period at an appropriate granularity.

Minutes

If we want the last minute we scan that minute and filter the counter paths selecting the paths and their timestamped values of interest.

Hours or Days

For higher aggregations we allocate a History of the appropriate length, e.g., 12 hours, and scan the row store over the specified time period. The values are inserted into the history for each counter and the History automatically reports the correct aggregate.

Regardless, the httpd query interprets a counter set having data only for the desired time period and at the desired granularity.