H - The generic type of the value returned by Callable.call()
for the master.O - The generic type for unserialized value objects.E - The generic type of the elements in the chunks stored in the
BlockingBuffer.S - The generic type of the subtask implementation class.L - The generic type of the key used to lookup a subtask in the
internal map (must be unique and must implement hashCode() and
equals() per their contracts).HS - The generic type of the value returned by Callable.call()
for the subtask.T - The generic type of the CTOR for the procedure used to write on
the index.R - The type of the result from applying the index procedure to a
single Split of data.A - The type of the aggregated result.public abstract class IndexWriteTask<H extends IndexAsyncWriteStats<L,HS>,O,E extends KVO<O>,S extends IndexPartitionWriteTask,L extends PartitionLocator,HS extends IndexPartitionWriteStats,T extends IKeyArrayIndexProcedure,R,A> extends AbstractMasterTask<H,E,S,L>
BlockingBuffer containing KVO[] chunks, splits
the chunks based on the separator keys for the scale-out index, and then
assigns each chunk to per-index partition BlockingBuffer which is in
turned drained by an IndexPartitionWriteTask that writes onto a
specific index partition.
If the task is interrupted, it will refuse additional writes by closing its
BlockingBuffer and will cancel any sub-tasks and discard any buffered
writes.
| Modifier and Type | Class and Description |
|---|---|
static class |
IndexWriteTask.M<T extends IKeyArrayIndexProcedure,O,R,A>
Concrete master hides most of the generic types leaving you with only
those that are meaningfully parameterize for applications using the
streaming write API.
|
| Modifier and Type | Field and Description |
|---|---|
protected AbstractKeyArrayIndexProcedureConstructor<T> |
ctor |
protected IDuplicateRemover<O> |
duplicateRemover |
protected IScaleOutClientIndex |
ndx |
protected IResultHandler<R,A> |
resultHandler |
protected int |
sinkChunkSize |
protected long |
sinkChunkTimeoutNanos |
protected int |
sinkQueueCapacity |
buffer, log, sinkIdleTimeoutNanos, sinkPollTimeoutNanos, src, stats| Constructor and Description |
|---|
IndexWriteTask(IScaleOutClientIndex ndx,
long sinkIdleTimeoutNanos,
long sinkPollTimeoutNanos,
int sinkQueueCapacity,
int sinkChunkSize,
long sinkChunkTimeoutNanos,
IDuplicateRemover<O> duplicateRemover,
AbstractKeyArrayIndexProcedureConstructor<T> ctor,
IResultHandler<R,A> resultHandler,
H stats,
BlockingBuffer<E[]> buffer) |
| Modifier and Type | Method and Description |
|---|---|
protected void |
handleChunk(E[] a,
boolean reopen)
Splits the chunk according to the current index partitions and transfers
each split to the appropriate sink.
|
protected S |
newSubtask(L locator,
BlockingBuffer<E[]> out)
Factory for a new subtask.
|
protected BlockingBuffer<E[]> |
newSubtaskBuffer()
Factory for a new buffer for a subtask.
|
protected void |
submitSubtask(FutureTask<? extends AbstractSubtaskStats> subtask)
Submit the subtask to an
Executor. |
String |
toString() |
addToOutputBuffer, call, getBuffer, getRedirectQueueSize, getSink, getStats, mapOperationOverSubtasks, moveSinkToFinishedQueueAtomically, nothingPending, notifySubtaskDone, redirectChunk, willShutdownhalt, haltedprotected final IScaleOutClientIndex ndx
protected final int sinkQueueCapacity
protected final int sinkChunkSize
protected final long sinkChunkTimeoutNanos
protected final IResultHandler<R,A> resultHandler
protected final IDuplicateRemover<O> duplicateRemover
protected final AbstractKeyArrayIndexProcedureConstructor<T extends IKeyArrayIndexProcedure> ctor
public IndexWriteTask(IScaleOutClientIndex ndx, long sinkIdleTimeoutNanos, long sinkPollTimeoutNanos, int sinkQueueCapacity, int sinkChunkSize, long sinkChunkTimeoutNanos, IDuplicateRemover<O> duplicateRemover, AbstractKeyArrayIndexProcedureConstructor<T> ctor, IResultHandler<R,A> resultHandler, H stats, BlockingBuffer<E[]> buffer)
ndx - The client's view of the scale-out index.sinkIdleTimeoutNanos - The time in nanoseconds after which an idle sink will be
closed. Any buffered writes are flushed when the sink is
closed. This must be GTE the sinkChunkTimeout
otherwise the sink will decide that it is idle when it was
just waiting for enough data to prepare a full chunk.sinkPollTimeoutNanos - The time in nanoseconds that the sink
will wait inside of the IAsynchronousIterator when it
polls the iterator for a chunk. This value should be
relatively small so that the sink remains responsible rather
than blocking inside of the IAsynchronousIterator for
long periods of time.sinkQueueCapacity - The capacity of the internal queue for the per-sink output
buffer.sinkChunkSize - The desired size of the chunks written that will be written by
the sink.sinkChunkTimeoutNanos - The maximum amount of time in nanoseconds that a sink will
combine smaller chunks so that it can satisfy the desired
sinkChunkSize.duplicateRemover - Removes duplicate key-value pairs from the (optional).ctor - The ctor instantiates an IIndexProcedure for each
chunk written on an index partition.resultHandler - Aggregates results across the individual index partition write
operations (optional).stats - The index statistics object.buffer - The buffer on which the application will write.protected void handleChunk(E[] a, boolean reopen) throws InterruptedException
handleChunk in class AbstractMasterTask<H extends IndexAsyncWriteStats<L,HS>,E extends KVO<O>,S extends IndexPartitionWriteTask,L extends PartitionLocator>a - A chunk.reopen - When false it is an error if the output buffer
has been closed. When true the output buffer
will be (re-)opened as necessary. This will be
false when invoked by AbstractMasterTask.call() (since
the output buffers are not closed until the master's buffer is
closed) and should be true if you are handling
redirects.InterruptedExceptionprotected S newSubtask(L locator, BlockingBuffer<E[]> out)
AbstractMasterTasknewSubtask in class AbstractMasterTask<H extends IndexAsyncWriteStats<L,HS>,E extends KVO<O>,S extends IndexPartitionWriteTask,L extends PartitionLocator>locator - The unique key for the subtask.out - The BlockingBuffer on which the master will write for
that subtask.protected BlockingBuffer<E[]> newSubtaskBuffer()
The queue capacity, chunk size and chunk timeout are taken from the ctor parameters.
newSubtaskBuffer in class AbstractMasterTask<H extends IndexAsyncWriteStats<L,HS>,E extends KVO<O>,S extends IndexPartitionWriteTask,L extends PartitionLocator>protected void submitSubtask(FutureTask<? extends AbstractSubtaskStats> subtask)
AbstractMasterTaskExecutor.submitSubtask in class AbstractMasterTask<H extends IndexAsyncWriteStats<L,HS>,E extends KVO<O>,S extends IndexPartitionWriteTask,L extends PartitionLocator>subtask - The FutureTask used to execute thee subtask.Copyright © 2006–2019 SYSTAP, LLC DBA Blazegraph. All rights reserved.