com.bigdata.bop.engine

Interface IChunkAccessor<E>

public interface IChunkAccessor<E>

API providing a variety of ways to access chunks of data (data are typically elements or binding sets). TODO This could be much more flexible and should be harmonized to support high volume operators, GPU operators, etc. probably the right thing to do is introduce another interface here with a getChunk():IChunk where IChunk let's you access the chunks data in different ways (and chunks can be both IBindingSet []s and element[]s so we might need to raise that into the interfaces and/or generics as well). TODO It is likely that we can convert to the use of BlockingQueue instead of BlockingBuffer in the operators and then handle the logic for combining chunks inside of the QueryEngine. E.g., by scanning this list for chunks for the same bopId and combining them logically into a single chunk.

For scale-out, chunk combination will naturally occur when the node on which the operator will run requests the ByteBuffers from the source nodes. Those will get wrapped up logically into a source for processing. For selective operators, those chunks can be combined before we execute the operator. For unselective operators, we are going to run over all the data anyway. TODO Expose an IChunkedIterator, which handles both element at a time and chunk at a time. TODO Expose a mechanism to visit the direct ByteBuffer slices in which the data are stored. For an operator which executes on a GPU, we want to transfer the data from the direct ByteBuffer in which it was received into a direct ByteBuffer which is a slice onto its VRAM. (And obviously we need to do the reverse with the outputs of a GPU operator).

Version:

$Id$

Author:

Bryan Thompson

See Also:

Optimize serialization for query messages on cluster

Method Summary

Methods

Modifier and Type	Method and Description
ICloseableIterator<E[]>	iterator() Visit the binding sets in the chunk.

Method Detail

iterator

ICloseableIterator<E[]> iterator()

Visit the binding sets in the chunk.