com.bigdata.counters.ganglia

Class BigdataMetadataFactory

java.lang.Object

com.bigdata.ganglia.AbstractMetrics

com.bigdata.ganglia.GangliaCoreMetricDecls

com.bigdata.counters.ganglia.BigdataMetadataFactory

All Implemented Interfaces:

IGangliaAttributes, IGangliaMetadataFactory

public class BigdataMetadataFactory
extends GangliaCoreMetricDecls
implements IGangliaMetadataFactory

A factory which integrates the bigdata hierarchical counter set model with the ganglia metric model.

The bigdata counter set model is hierarchical and collects both per-host and per-service metrics. This is illustrated below.

 /192.168.1.10/CPU/% IO Wait=0.0
 /192.168.1.10/CPU/% Processor Time=0.24
 /192.168.1.10/CPU/% System Time=0.08
 /192.168.1.10/CPU/% User Time=0.16
 /192.168.1.10/Info/Architecture=amd64
 /192.168.1.10/Info/Number of Processors=4
 /192.168.1.10/Info/Operating System Name=Linux
 /192.168.1.10/Info/Operating System Version=2.6.38-11-server
 /192.168.1.10/Info/Processor Info=Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz Family 6 Model 42 Stepping 7, GenuineIntel
 /192.168.1.10/Memory/Bytes Free=5.031514112E9
 /192.168.1.10/Memory/Major Page Faults Per Second=0.0
 /192.168.1.10/Memory/Swap Bytes Used=0.0
 /192.168.1.10/PhysicalDisk/Bytes Read Per Second=0.0
 /192.168.1.10/PhysicalDisk/Bytes Written Per Second=6144.0
 /192.168.1.10/test-context-7/CPU/% Processor Time=0.0
 /192.168.1.10/test-context-7/CPU/% System Time=0.0
 /192.168.1.10/test-context-7/CPU/% User Time=0.0020
 /192.168.1.10/test-context-7/Memory/Major Faults per Second=0.0
 /192.168.1.10/test-context-7/Memory/Minor Faults per Second=0.0
 /192.168.1.10/test-context-7/Memory/Percent Memory Size=0.0091
 /192.168.1.10/test-context-7/Memory/Resident Set Size=76304384
 /192.168.1.10/test-context-7/Memory/Virtual Size=2449391616
 /192.168.1.10/test-context-7/PhysicalDisk/Bytes Read per Second=0.0
 /192.168.1.10/test-context-7/PhysicalDisk/Bytes Written per Second=0.0
 

In order to map this information onto ganglia, the host and service names must be identified in the counter path. This allows us to recognize the local name of the metric, which may itself include some path components. For example, Memory/Resident Set Size is the local name of the metric whose value is the resident set size of the service process. Some metrics appear in both the per-host and per-service sections of the counter set hierarchy. For example, CPU/% Processor Time is reported both for the host and for the service (when using pidstat to monitor the service).

For the per-host metrics, bigdata and ganglia sometimes collect what is essentially the same metric. For example, CPU/% User Time is essentially the same as cpu_user. However, sometimes the units in which the metric is collected differ. For example, bigdata collects Memory/Bytes Free in bytes using a double counter while ganglia collects mem_free in KBs using a float metric. In these cases, we need to scale the metric and also change the name of the metric in order to report it as if it were a core ganglia metric.

Another difference is that bigdata tends to put the units into the name of the metric while ganglia has explicit metadata about the units. For metrics such as CPU/% System Time we need to register the appropriate units (note that ganglia also can not handle the % in the name of the metric). TODO Service name, service iface, service UUID. TODO Right now metrics which we are not aligning with Ganglia are being reported out in other top-level groups (Bytes Free and Swap Bytes Used are two good examples). [Bytes Free has been fixed.] TODO Rather than munging a "%" to a "Percent", it would be better to rename the metric and specify "%" as the units.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	serviceName

Fields inherited from class com.bigdata.ganglia.GangliaCoreMetricDecls

heartbeatInterval

Fields inherited from class com.bigdata.ganglia.AbstractMetrics

dmax, emptyMap, hostName, slope, tmax

Fields inherited from interface com.bigdata.ganglia.IGangliaAttributes

ATTR_DESC, ATTR_GROUP, ATTR_TITLE, GROUP_CORE, GROUP_CPU, GROUP_DISK, GROUP_LOAD, GROUP_MEMORY, GROUP_NETWORK, GROUP_PROCESS, GROUP_SYSTEM

Constructor Summary

Constructors

Constructor and Description
BigdataMetadataFactory(String hostName, String serviceName, GangliaSlopeEnum slope, int tmax, int dmax, int heartbeatInterval)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	addHostCounter(String metricName, IGangliaMetadataMessage decl) Register a per-host metric.
protected void	addHostCounter(String metricName, IGangliaMetadataMessage decl, boolean prefer)
protected void	addHostCPUMetrics()
protected void	addHostDiskMetrics()
protected void	addHostInfoMetrics()
protected void	addHostMemoryMetrics()
protected void	addHostMetrics() Hook to declare host metrics.
protected void	addServiceCounter(String metricName, IGangliaMetadataMessage decl) Register a per-service metric.
protected void	addServiceCounter(String metricName, IGangliaMetadataMessage decl, boolean prefer)
protected void	addServiceCPUMetrics()
protected void	addServiceDiskMetrics()
protected void	addServiceInfoMetrics()
protected void	addServiceMemoryMetrics()
protected void	addServiceMetrics() Hook to declare service metrics.
protected String	getServiceMetricName(String group, String label)
IGangliaMetadataMessage	newDecl(String hostName, String metricName, Object value) Factory for new declarations.
IGangliaMetadataMessage	resolve(IGangliaMetadataMessage decl) If this class has a replacement declaration for the metric, then it is returned.

Methods inherited from class com.bigdata.ganglia.GangliaCoreMetricDecls

boottime, bytes_in, bytes_out, cpu_aidle, cpu_idle, cpu_intr, cpu_nice, cpu_num, cpu_sintr, cpu_speed, cpu_system, cpu_user, cpu_wio, declare, disk_free, disk_total, gexec, heartbeat, heartbeat, load_fifteen, load_five, load_one, location, machine_type, mem_buffers, mem_cached, mem_free, mem_shared, mem_total, os_name, os_release, part_max_used, pkts_in, pkts_out, proc_run, proc_total, swap_free, swap_total

Methods inherited from class com.bigdata.ganglia.AbstractMetrics

getMap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

serviceName

protected final String serviceName

Constructor Detail

BigdataMetadataFactory

public BigdataMetadataFactory(String hostName,
                      String serviceName,
                      GangliaSlopeEnum slope,
                      int tmax,
                      int dmax,
                      int heartbeatInterval)

Parameters:

hostName - The name of this host.

serviceName - The name of the service running on this host.

slope - The default value to use in the declarations.

tmax - The value of tmax to use in the declarations.

dmax - The value of dmax to use in the declarations.

heartbeatInteval - The heartbeat interval in seconds -or- ZERO (0) if we will not be sending out the ganglia host heartbeat.

Method Detail

addHostCounter

protected void addHostCounter(String metricName,
                  IGangliaMetadataMessage decl)

Register a per-host metric.

Parameters:

metricName - The name of the metric.

decl - The declaration for the metric.

addHostCounter

protected void addHostCounter(String metricName,
                  IGangliaMetadataMessage decl,
                  boolean prefer)

addServiceCounter

protected void addServiceCounter(String metricName,
                     IGangliaMetadataMessage decl)

Register a per-service metric.

Parameters:

metricName - The name of the metric.

decl - The declaration for the metric.

addServiceCounter

protected void addServiceCounter(String metricName,
                     IGangliaMetadataMessage decl,
                     boolean prefer)

newDecl

public IGangliaMetadataMessage newDecl(String hostName,
                              String metricName,
                              Object value)

Factory for new declarations.

Attempts to resolve a host counter and then a service counter based on the localName.

Specified by:

newDecl in interface IGangliaMetadataFactory

Parameters:

hostName - The name of this host.

metricName - The ganglia metric name (suitably munged if necessary for use by ganglia).

value - The metric value (NOT null).

Returns:

The declaration if one exists for that localName.

resolve

public IGangliaMetadataMessage resolve(IGangliaMetadataMessage decl)

If this class has a replacement declaration for the metric, then it is returned. Otherwise the caller's argument is returned. TODO We could actually meld the data generating a new declaration having the desired behavior but the same data as the caller's argument. That would require a new constructor (or factory method) on a rich metadata class.

Specified by:

resolve in interface IGangliaMetadataFactory

Parameters:

decl - A declaration (typically received over the wire).

Returns:

The declaration to be used.

addHostMetrics

protected void addHostMetrics()

Hook to declare host metrics.

See Also:

IHostCounters

addHostInfoMetrics

protected void addHostInfoMetrics()

addHostCPUMetrics

protected void addHostCPUMetrics()

addHostMemoryMetrics

protected void addHostMemoryMetrics()

addHostDiskMetrics

protected void addHostDiskMetrics()

addServiceMetrics

protected void addServiceMetrics()

Hook to declare service metrics.

See Also:

IProcessCounters

getServiceMetricName

protected String getServiceMetricName(String group,
                          String label)

addServiceInfoMetrics

protected void addServiceInfoMetrics()

addServiceCPUMetrics

protected void addServiceCPUMetrics()

addServiceMemoryMetrics

protected void addServiceMemoryMetrics()

addServiceDiskMetrics

protected void addServiceDiskMetrics()