com.bigdata.bop.join

Interface PipelineJoin.Annotations

All Superinterfaces:

AccessPathJoinAnnotations, BOp.Annotations, BufferAnnotations, JoinAnnotations, PipelineOp.Annotations

Enclosing class:

PipelineJoin<E>

public static interface PipelineJoin.Annotations
extends AccessPathJoinAnnotations

Field Summary

Fields

Modifier and Type	Field and Description
static String	COALESCE_DUPLICATE_ACCESS_PATHS When true, binding sets observed in the same chunk which have the binding pattern on the variables for the access path will be coalesced into a single access path (default true).
static boolean	DEFAULT_COALESCE_DUPLICATE_ACCESS_PATHS
static int	DEFAULT_MAX_PARALLEL_CHUNKS
static int	DEFAULT_MIN_DATAPOINTS_PER_TASK
static int	DEFAULT_NUM_TASKS_PER_THREAD
static boolean	DEFAULT_REORDER_ACCESS_PATHS
static String	MAX_PARALLEL_CHUNKS The maximum parallelism with which the pipeline will consume the source IBindingSet[] chunk.
static String	MIN_DATAPOINTS_PER_TASK The minimum number of (estimated) data points assigned to a task.
static String	NUM_TASKS_PER_THREAD Desired number of access path tasks generated per thread in case the range is large enough.
static String	REORDER_ACCESS_PATHS When true, access paths will be reordered to maximize locality.

Fields inherited from interface com.bigdata.bop.join.AccessPathJoinAnnotations

PREDICATE

Fields inherited from interface com.bigdata.bop.join.JoinAnnotations

CONSTRAINTS, DEFAULT_LIMIT, JOIN_TYPE, LIMIT, SELECT

Fields inherited from interface com.bigdata.bop.PipelineOp.Annotations

ALT_SINK_REF, DEFAULT_LAST_PASS, DEFAULT_MAX_MEMORY, DEFAULT_MAX_MESSAGES_PER_TASK, DEFAULT_MAX_PARALLEL, DEFAULT_PIPELINE_QUEUE_CAPACITY, DEFAULT_PIPELINED, DEFAULT_REORDER_SOLUTIONS, DEFAULT_SHARED_STATE, LAST_PASS, MAX_MEMORY, MAX_MESSAGES_PER_TASK, MAX_PARALLEL, PIPELINE_QUEUE_CAPACITY, PIPELINED, REORDER_SOLUTIONS, SHARED_STATE, SINK_REF

Fields inherited from interface com.bigdata.bop.BOp.Annotations

BOP_ID, CONTROLLER, DEFAULT_CONTROLLER, DEFAULT_EVALUATION_CONTEXT, DEFAULT_TIMEOUT, EVALUATION_CONTEXT, NAMESPACE, TIMEOUT

Fields inherited from interface com.bigdata.bop.BufferAnnotations

CHUNK_CAPACITY, CHUNK_OF_CHUNKS_CAPACITY, CHUNK_TIMEOUT, chunkTimeoutUnit, DEFAULT_CHUNK_CAPACITY, DEFAULT_CHUNK_OF_CHUNKS_CAPACITY, DEFAULT_CHUNK_TIMEOUT

Field Detail

MAX_PARALLEL_CHUNKS

static final String MAX_PARALLEL_CHUNKS

The maximum parallelism with which the pipeline will consume the source IBindingSet[] chunk.

Note: When ZERO (0), everything will run in the caller's Thread, but there will still be one thread per pipeline join task which is executing concurrently against different source chunks. When GT ZERO (0), tasks will run on an ExecutorService with the specified maximum parallelism.

Note: This is NOT the same as PipelineOp.Annotations#MAX_PARALLEL. This option ( MAX_PARALLEL_CHUNKS limits the #of chunks for a single task which may be processed concurrently. PipelineOp.Annotations#MAX_PARALLEL limits the #of task instances which may run concurrently.

See Also:

DEFAULT_MAX_PARALLEL_CHUNKS

DEFAULT_MAX_PARALLEL_CHUNKS

static final int DEFAULT_MAX_PARALLEL_CHUNKS

See Also:

Constant Field Values

COALESCE_DUPLICATE_ACCESS_PATHS

static final String COALESCE_DUPLICATE_ACCESS_PATHS

When true, binding sets observed in the same chunk which have the binding pattern on the variables for the access path will be coalesced into a single access path (default true). This option increases the efficiency of the join since it reads the access path once per set of input binding sets which are coalesced. This option does NOT reduce the #of solutions generated.

This option can cause some error in the join hit ratio when it is estimated from a cutoff join.

See Also:

PipelineJoinStats.getJoinHitRatio()

DEFAULT_COALESCE_DUPLICATE_ACCESS_PATHS

static final boolean DEFAULT_COALESCE_DUPLICATE_ACCESS_PATHS

See Also:

Constant Field Values

REORDER_ACCESS_PATHS

static final String REORDER_ACCESS_PATHS

When true, access paths will be reordered to maximize locality.

Note: This needs to be turned off when the RTO uses row identifiers to correlate the input and output solutions for complex joins (those which required materialization of RDF Values for FILTER evaluation).

DEFAULT_REORDER_ACCESS_PATHS

static final boolean DEFAULT_REORDER_ACCESS_PATHS

See Also:

Constant Field Values

MIN_DATAPOINTS_PER_TASK

static final String MIN_DATAPOINTS_PER_TASK

The minimum number of (estimated) data points assigned to a task. This basically defines the threshold upon which parallelization starts to pay out. Currently only implemented for the geospatial feature (see BLZG-1478), but this might be implemented more generally for the pipelined join in the future

DEFAULT_MIN_DATAPOINTS_PER_TASK

static final int DEFAULT_MIN_DATAPOINTS_PER_TASK

See Also:

Constant Field Values

NUM_TASKS_PER_THREAD

static final String NUM_TASKS_PER_THREAD

Desired number of access path tasks generated per thread in case the range is large enough. The default is one task per thread, which is a good choice if there's not much skew in the data. If there's a lot of skew, it might be beneficial to increase this parameter to avoid a situation where a certain thread gets assigned longer-running tasks s.t. other threads are finished earlier (leading to, effectively, decreased parallelism). On the downside, increasing this parameter leads to overhead in setting up additional subtasks, so values >5 are generally discouraged. Note that this is an upper bound only: we always consider the MIN_DATAPOINTS_PER_TASK threshold in order to avoid generating threads with a low workload. Currently only implemented for the geospatial feature (see BLZG-1478), but this might be implemented more generally for the pipelined join in the future. Must be a value >= 1.

DEFAULT_NUM_TASKS_PER_THREAD

static final int DEFAULT_NUM_TASKS_PER_THREAD

See Also:

Constant Field Values