com.bigdata.relation.rule.eval.pipeline

Class DistributedJoinTask

java.lang.Object

com.bigdata.relation.rule.eval.pipeline.JoinTask

com.bigdata.relation.rule.eval.pipeline.DistributedJoinTask

All Implemented Interfaces:

Callable<Void>

public class DistributedJoinTask
extends JoinTask

Implementation used by scale-out deployments. There will be one instance of this task per index partition on which the rule will read. Those instances will be in-process on the DataService hosting that index partition. Instances are created on the DataService using the JoinTaskFactoryTask helper class.

Version:

$Id$

Author:

Bryan Thompson

Nested Class Summary

Nested classes/interfaces inherited from class com.bigdata.relation.rule.eval.pipeline.JoinTask

JoinTask.AccessPathTask, JoinTask.BindingSetConsumerTask, JoinTask.ChunkTask, JoinTask.ThreadLocalFactory<T extends IBuffer<E>,E>

Field Summary

Fields

Modifier and Type	Field and Description
protected AbstractScaleOutFederation<?>	fed The federation is used to obtain locator scans for the access paths.
protected IJoinNexus	fedJoinNexus The IJoinNexus for the IBigdataFederation.
protected Future<Void>	futureProxy A (proxy for) the Future for this DistributedJoinTask.

Fields inherited from class com.bigdata.relation.rule.eval.pipeline.JoinTask

DEBUG, firstCause, halt, INFO, joinNexus, lastJoin, log, masterProxy, masterUUID, orderIndex, partitionId, predicate, relation, requiredVars, rule, tailCount, tailIndex, WARN

Constructor Summary

Constructors

Constructor and Description
DistributedJoinTask(IRule rule, IJoinNexus joinNexus, int[] order, int orderIndex, int partitionId, AbstractScaleOutFederation<?> fed, IJoinMaster master, UUID masterUUID, IAsynchronousIterator<IBindingSet[]> src, IKeyOrder[] keyOrders, DataService dataService, IVariable[][] requiredVars)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	addSource(IAsynchronousIterator<IBindingSet[]> source) Adds a source from which this DistributedJoinTask will read IBindingSet chunks.
protected void	cancelSinks() Cancel all DistributedJoinTasks that are sinks for this DistributedJoinTask.
protected void	closeSources() Sets a flag preventing new sources from being declared and closes all known sources and removes this task from the Session.
protected IBindingSet[]	combineChunks(List<IBindingSet[]> chunks, int bindingSetCount) Combine the chunk(s) into a single chunk.
protected void	flushAndCloseBuffersAndAwaitSinks() Notifies each sink that this DistributedJoinTask will no longer generate new IBindingSet chunks and then waits for the sink task(s) to complete.
protected JoinTaskSink	getSink(PartitionLocator locator) Return the sink on which we will write IBindingSet for the index partition associated with the specified locator.
protected IBuffer<ISolution[]>	getSolutionBuffer() The buffer on which the last predicate in the evaluation order will write its ISolutions.
protected void	logCallError(Throwable t) Logs an error in JoinTask.call() on the local log file and adds some metadata about the operation which was being executed.
protected AbstractUnsynchronizedArrayBuffer<IBindingSet>	newUnsyncOutputBuffer() A method used by the JoinTask.threadLocalBufferFactory to create new output buffer as required.
protected IBindingSet[]	nextChunk() Returns a chunk of IBindingSets by combining chunks from the various source JoinTasks.

Methods inherited from class com.bigdata.relation.rule.eval.pipeline.JoinTask

call, consumeSources, getTailIndex, halt, reportOnce, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

fed

protected final AbstractScaleOutFederation<?> fed

The federation is used to obtain locator scans for the access paths.

fedJoinNexus

protected final IJoinNexus fedJoinNexus

The IJoinNexus for the IBigdataFederation. This is mainly used to setup the solutionBuffer since it needs to write on the scale-out index while the AccessPathTask will read on the local index partition view.

futureProxy

protected Future<Void> futureProxy

A (proxy for) the Future for this DistributedJoinTask.

Constructor Detail

DistributedJoinTask

public DistributedJoinTask(IRule rule,
                   IJoinNexus joinNexus,
                   int[] order,
                   int orderIndex,
                   int partitionId,
                   AbstractScaleOutFederation<?> fed,
                   IJoinMaster master,
                   UUID masterUUID,
                   IAsynchronousIterator<IBindingSet[]> src,
                   IKeyOrder[] keyOrders,
                   DataService dataService,
                   IVariable[][] requiredVars)

Method Detail

addSource

public boolean addSource(IAsynchronousIterator<IBindingSet[]> source)

Adds a source from which this DistributedJoinTask will read IBindingSet chunks.

Parameters:

source - The source.

Returns:

true iff the source was accepted.

Throws:

IllegalArgumentException - if the source is null.

getSolutionBuffer

protected final IBuffer<ISolution[]> getSolutionBuffer()

Description copied from class: JoinTask

The buffer on which the last predicate in the evaluation order will write its ISolutions.

Specified by:

getSolutionBuffer in class JoinTask

Returns:

The buffer.

closeSources

protected void closeSources()

Sets a flag preventing new sources from being declared and closes all known sources and removes this task from the Session.

nextChunk

protected IBindingSet[] nextChunk()
                           throws InterruptedException

Returns a chunk of IBindingSets by combining chunks from the various source JoinTasks.

Specified by:

nextChunk in class JoinTask

Returns:

A chunk assembled from one or more chunks from one or more of the source JoinTasks.

Throws:

InterruptedException

combineChunks

protected IBindingSet[] combineChunks(List<IBindingSet[]> chunks,
                          int bindingSetCount)

Combine the chunk(s) into a single chunk.

Parameters:

chunks - A list of chunks read from the sources.

bindingSetCount - The #of bindingSets in those chunks.

Returns:

newUnsyncOutputBuffer

protected AbstractUnsynchronizedArrayBuffer<IBindingSet> newUnsyncOutputBuffer()

Description copied from class: JoinTask

A method used by the JoinTask.threadLocalBufferFactory to create new output buffer as required. The output buffer will be used to aggregate IBindingSets generated by this JoinTask.

Note: A different implementation class must be used depending on whether or not this is the last join dimension for the query (when it is, then we write on the solution buffer) and whether or not the target join index is key-range partitioned (when it is, each binding set is mapped across the sink JoinTask(s)).

Specified by:

newUnsyncOutputBuffer in class JoinTask

flushAndCloseBuffersAndAwaitSinks

protected void flushAndCloseBuffersAndAwaitSinks()
                                          throws InterruptedException,
                                                 ExecutionException

Notifies each sink that this DistributedJoinTask will no longer generate new IBindingSet chunks and then waits for the sink task(s) to complete.

Note: Closing the BlockingBuffer from which a sink JoinTask is reading will cause the source iterator for that sink task to eventually return false indicating that it is exhausted (assuming that the sink keeps reading on the iterator).

Specified by:

flushAndCloseBuffersAndAwaitSinks in class JoinTask

Throws:

InterruptedException - if interrupted while awaiting the future for a sink.

ExecutionException

cancelSinks

protected void cancelSinks()

Cancel all DistributedJoinTasks that are sinks for this DistributedJoinTask.

Specified by:

cancelSinks in class JoinTask

getSink

protected JoinTaskSink getSink(PartitionLocator locator)
                        throws InterruptedException,
                               RuntimeException

Return the sink on which we will write IBindingSet for the index partition associated with the specified locator. The sink will be backed by a DistributedJoinTask running on the IDataService that is host to that index partition. The scale-out index will be the scale-out index for the next IPredicate in the evaluation order.

Parameters:

locator - The locator for the index partition.

Returns:

The sink.

Throws:

RuntimeException - If the JoinTaskFactoryTask fails.

InterruptedException - If the JoinTaskFactoryTask is interrupted.

logCallError

protected void logCallError(Throwable t)

Logs an error in JoinTask.call() on the local log file and adds some metadata about the operation which was being executed. This does not imply that the error originates with this join task. You have to inspect the error messages, the order in which the joins were being evaluated, and even correlate the JoinTask.masterUUID in order to figure out what really happened.

Overrides:

logCallError in class JoinTask