com.bigdata.bop.fed

Class FederationChunkHandler<E>

java.lang.Object

com.bigdata.bop.engine.StandaloneChunkHandler

com.bigdata.bop.fed.FederationChunkHandler<E>

Type Parameters:

E - The generic type of the objects in the relation.

All Implemented Interfaces:

IChunkHandler

public class FederationChunkHandler<E>
extends StandaloneChunkHandler

The base class is extended to organize the output from one operator so in order to make it available to another operator running on a different node. There are several cases which have to be handled and which are identified by the BOp.getEvaluationContext(). In addition, we need to handle low latency and high data volume queries somewhat differently. Except for BOpEvaluationContext.ANY, all of these cases wind up writing the intermediate results onto a direct ByteBuffer and notifying the receiving service that there are intermediate results which it can pull when it is ready to process them. This pattern allows the receiver to impose flow control on the producer.

Version:

$Id: FederationChunkHandler.java 6038 2012-02-17 17:43:26Z thompsonbry $

Author:

Bryan Thompson

See Also:

Vector query engine messages per node

Field Summary

Fields

Modifier and Type	Field and Description
static IChunkHandler	INSTANCE FIXME Debug the NIO chunk message materialization logic (it is currently disabled by the setting of the nioThreshold parameter to the constructor).
static IChunkHandler	TEST_INSTANCE Instance used by some test suites to avoid a dependency on the RDF data model.

Fields inherited from class com.bigdata.bop.engine.StandaloneChunkHandler

MANAGED_HEAP_INSTANCE, NATIVE_HEAP_INSTANCE

Constructor Summary

Constructors

Constructor and Description
FederationChunkHandler(int nioThreshold, boolean usePOJO)

Method Summary

Methods

Modifier and Type	Method and Description
int	handleChunk(IRunningQuery query, int bopId, int sinkId, IBindingSet[] chunk) Take an IBindingSet[] chunk generated by some pass over an operator and make it available to the target operator.
protected void	sendChunkMessage(FederatedRunningQuery q, UUID serviceUUID, int sinkId, int partitionId, DirectBufferPoolAllocator.IAllocationContext allocationContext, IBindingSet[] source) Create and send an IChunkMessage from some intermediate results.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INSTANCE

public static final IChunkHandler INSTANCE

FIXME Debug the NIO chunk message materialization logic (it is currently disabled by the setting of the nioThreshold parameter to the constructor).

See Also:

ResourceService should use NIO for file and buffer transfers, Support NIO solution set interchange on the cluster

TEST_INSTANCE

public static final IChunkHandler TEST_INSTANCE

Instance used by some test suites to avoid a dependency on the RDF data model. All messages will use LocalChunkMessage which uses POJO serialization.

Constructor Detail

FederationChunkHandler

public FederationChunkHandler(int nioThreshold,
                      boolean usePOJO)

Parameters:

nioThreshold - The threshold above which the intermediate solutions are shipped using NIO rather than RMI. This is ignored if usePOJO:=true.

usePOJO - When true, the LocalChunkMessage will be used for all messages. This allows the test cases to avoid RDF specific logic in the IChunkMessage serialization.

Method Detail

handleChunk

public int handleChunk(IRunningQuery query,
              int bopId,
              int sinkId,
              IBindingSet[] chunk)

Take an IBindingSet[] chunk generated by some pass over an operator and make it available to the target operator. How this is done depends on whether the query is running against a standalone database or the scale-out database.

Note: The return value is used as part of the termination criteria for the query which depends on (a) the #of running operator tasks and (b) the #of IChunkMessages generated (available) and consumed. The return value of this method increases the #of IChunkMessages available to the query.

Specified by:

handleChunk in interface IChunkHandler

Overrides:

handleChunk in class StandaloneChunkHandler

Parameters:

query - The query.

bopId - The operator which wrote on the sink.

sinkId - The identifier of the target operator.

chunk - The intermediate results to be passed to that target operator.

Returns:

The #of IChunkMessage sent. This will always be ONE (1) for scale-up. For scale-out, there will be at least one IChunkMessage per index partition over which the intermediate results were mapped.

sendChunkMessage

protected void sendChunkMessage(FederatedRunningQuery q,
                    UUID serviceUUID,
                    int sinkId,
                    int partitionId,
                    DirectBufferPoolAllocator.IAllocationContext allocationContext,
                    IBindingSet[] source)

Create and send an IChunkMessage from some intermediate results. Various optimizations are employed depending on the amount of data to be moved and whether or not the target is this service.

Parameters:

serviceUUID - The UUID of the IQueryPeer who is the recipient.

sinkId - The identifier of the target BOp.

allocationContext - The allocation context within which the ByteBuffers will be managed for this NIOChunkMessage.

source - The binding sets to be formatted onto a buffer.