com.bigdata.bop.solutions

Class PipelinedAggregationOp

java.lang.Object

com.bigdata.bop.CoreBaseBOp

com.bigdata.bop.BOpBase

com.bigdata.bop.PipelineOp

com.bigdata.bop.solutions.GroupByOp

com.bigdata.bop.solutions.PipelinedAggregationOp

All Implemented Interfaces:

BOp, ISingleThreadedOp, IPropertySet, Serializable, Cloneable

public class PipelinedAggregationOp
extends GroupByOp
implements ISingleThreadedOp

A pipelined aggregation operator based on an in memory hash table associating with per-group state for each aggregate expression (it can also handle the degenerate case where all solutions form a single implicit group). This operator is highly efficient, but may only be used if (a) DISTINCT is NOT specified for any aggregate and (b) aggregates do not embed other aggregates.

Note: This implementation is a pipelined operator which inspects each chunk of solutions as they arrive. The state is shared across invocations of the operator for each source chunk. The operator waits until the last chunk has been consumed before writing the output solutions. In order to observe the lastInvocation signal, the operator MUST be single threaded ( PipelineOp.Annotations#MAX_PARALLEL:=1) and running on the query controller.

Note: Since this operator evaluates IAggregates incrementally (one input solution at a time), it relies on IAggregate's contract for "sticky" errors. See IAggregate.get(IBindingSet) and IAggregate.done().

Note: This this operator will be invoked multiple times, and potentially on multiple nodes in a cluster, it is critical that the anonymous variables assigned by the GroupByRewriter are stable across all invocations on any node of the cluster (this caution also applies for a single node where the operator can still be invoked multiple times).

Author:

Bryan Thompson

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	PipelinedAggregationOp.Annotations

Field Summary

Fields inherited from class com.bigdata.bop.CoreBaseBOp

DEFAULT_INITIAL_CAPACITY

Fields inherited from interface com.bigdata.bop.BOp

NOANNS, NOARGS

Constructor Summary

Constructors

Constructor and Description
PipelinedAggregationOp(BOp[] args, Map<String,Object> annotations) Required shallow copy constructor.
PipelinedAggregationOp(PipelinedAggregationOp op) Constructor required for com.bigdata.bop.BOpUtility#deepCopy(FilterNode).

Method Summary

Methods

Modifier and Type	Method and Description
FutureTask<Void>	eval(BOpContext<IBindingSet> context) Return a FutureTask which computes the operator against the evaluation context.
int	getInitialCapacity()
float	getLoadFactor()
boolean	isPipelinedAggregationOp() Return true iff this operator supports pipelined aggregation
BOpStats	newStats() Return a new object which can be used to collect statistics on the operator evaluation.

Methods inherited from class com.bigdata.bop.solutions.GroupByOp

getGroupByRewrite, getGroupByState

Methods inherited from class com.bigdata.bop.PipelineOp

assertAtOnceJavaHeapOp, assertMaxParallelOne, getChunkCapacity, getChunkOfChunksCapacity, getChunkTimeout, getMaxMemory, getMaxParallel, isAtOnceEvaluation, isBlockedEvaluation, isLastPassRequested, isPipelinedEvaluation, isReorderSolutions, isSharedState

Methods inherited from class com.bigdata.bop.BOpBase

__replaceArg, _clearProperty, _set, _setProperty, annotations, annotationsCopy, annotationsEqual, annotationsRef, argIterator, args, argsCopy, arity, clearAnnotations, clearProperty, deepCopy, deepCopy, get, getProperty, setArg, setProperty, setUnboundProperty, toArray, toArray

Methods inherited from class com.bigdata.bop.CoreBaseBOp

annotationsEqual, annotationsToString, annotationsToString, annotationValueToString, checkArgs, clone, equals, getEvaluationContext, getId, getProperty, getRequiredProperty, hashCode, indent, isController, mutation, shortenName, toShortString, toString, toString

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

PipelinedAggregationOp

public PipelinedAggregationOp(PipelinedAggregationOp op)

Constructor required for com.bigdata.bop.BOpUtility#deepCopy(FilterNode).

PipelinedAggregationOp

public PipelinedAggregationOp(BOp[] args,
                      Map<String,Object> annotations)

Required shallow copy constructor.

Method Detail

isPipelinedAggregationOp

public boolean isPipelinedAggregationOp()

Return true iff this operator supports pipelined aggregation

Note: Pipelined aggregation is only possible when (a) DISTINCT is not used in the IAggregates; and (b) IAggregates are not nested within other IAggregates. An implementation which returns true for this method MUST NOT be used for aggregation operations which would violate either of these constraints.

Returns true. This is a pipelined aggregation operator and MAY NOT be used to evaluate aggregation requests which use DISTINCT or which nest IAggregates in other IAggregates.

Specified by:

isPipelinedAggregationOp in class GroupByOp

getInitialCapacity

public int getInitialCapacity()

See Also:

HashMapAnnotations.INITIAL_CAPACITY

getLoadFactor

public float getLoadFactor()

See Also:

HashMapAnnotations.LOAD_FACTOR

newStats

public BOpStats newStats()

Description copied from class: PipelineOp

Return a new object which can be used to collect statistics on the operator evaluation. This may be overridden to return a more specific class depending on the operator.

Overrides:

newStats in class PipelineOp

eval

public FutureTask<Void> eval(BOpContext<IBindingSet> context)

Description copied from class: PipelineOp

Return a FutureTask which computes the operator against the evaluation context. The caller is responsible for executing the FutureTask (this gives them the ability to hook the completion of the computation).

Specified by:

eval in class PipelineOp

Parameters:

context - The evaluation context.

Returns:

The FutureTask which will compute the operator's evaluation.