com.bigdata.resources

Interface OverflowManager.Options

All Superinterfaces:

IndexManager.Options, IServiceShutdown.Options, Options, StoreManager.Options

All Known Subinterfaces:

DataService.Options, EmbeddedClient.Options, MetadataService.Options, ResourceManager.Options, StressTestConcurrent.TestOptions

Enclosing class:

OverflowManager

public static interface OverflowManager.Options
extends IndexManager.Options, IServiceShutdown.Options

Options understood by the OverflowManager.

Version:

$Id$

Author:

Bryan Thompson

Field Summary

Fields

Modifier and Type	Field and Description
static String	ACCELERATE_SPLIT_THRESHOLD The #of index partitions below which we will accelerate the decision to split an index partition (default "20").
static String	BUILD_SERVICE_CORE_POOL_SIZE The #of threads in the pool handling index segment builds from the old journal.
static String	COPY_INDEX_THRESHOLD Index partitions having no more than this many entries as reported by a range count will be copied to the new journal during synchronous overflow processing rather than building a new index segment from the buffered writes (default "1000").
static String	DEFAULT_ACCELERATE_SPLIT_THRESHOLD
static String	DEFAULT_BUILD_SERVICE_CORE_POOL_SIZE
static String	DEFAULT_COPY_INDEX_THRESHOLD
static String	DEFAULT_HOT_SPLIT_THRESHOLD
static String	DEFAULT_JOINS_ENABLED
static String	DEFAULT_MAXIMUM_BUILD_SEGMENTS_BYTES
static String	DEFAULT_MAXIMUM_JOURNALS_PER_VIEW
static String	DEFAULT_MAXIMUM_MOVE_PERCENT_OF_SPLIT
static String	DEFAULT_MAXIMUM_MOVES
static String	DEFAULT_MAXIMUM_MOVES_PER_TARGET
static String	DEFAULT_MAXIMUM_SEGMENTS_PER_VIEW
static String	DEFAULT_MERGE_SERVICE_CORE_POOL_SIZE
static String	DEFAULT_MINIMUM_ACTIVE_INDEX_PARTITIONS
static String	DEFAULT_MOVE_PERCENT_CPU_TIME_THRESHOLD
static String	DEFAULT_NOMINAL_SHARD_SIZE
static String	DEFAULT_OPTIONAL_COMPACTING_MERGES_PER_OVERFLOW
static String	DEFAULT_OVERFLOW_CANCELLED_WHEN_JOURNAL_FULL
static String	DEFAULT_OVERFLOW_ENABLED
static String	DEFAULT_OVERFLOW_MAX_COUNT
static String	DEFAULT_OVERFLOW_TASKS_CONCURRENT
static String	DEFAULT_OVERFLOW_THRESHOLD
static String	DEFAULT_OVERFLOW_TIMEOUT The default timeout in milliseconds for asynchronous overflow processing (equivalent to 10 minutes).
static String	DEFAULT_PERCENT_OF_SPLIT_THRESHOLD
static String	DEFAULT_SCATTER_SPLIT_ENABLED
static String	DEFAULT_TAIL_SPLIT_THRESHOLD
static String	HOT_SPLIT_THRESHOLD Deprecated. Hot splits are not implemented and this option does not do anything. It will be going away soon.
static String	JOINS_ENABLED Option may be used to disable index partition joins.
static String	MAXIMUM_BUILD_SEGMENT_BYTES Option limits the #of IndexSegmentStore bytes that an OverflowActionEnum.Build operation will process (default "20971520").
static String	MAXIMUM_JOURNALS_PER_VIEW Deprecated. merges are now performed in priority order while time remains in a given asynchronous overflow cycle.
static String	MAXIMUM_MOVE_PERCENT_OF_SPLIT This is the maximum percentage (in [0:2]) of a full index partition which will be considered for a move (default ".8").
static String	MAXIMUM_MOVES Deprecated. Moves are now decided on a case by case basis. An alternative parameter might be introduced in the future to restrict the rate at which a DS can shed shards by moving them to other nodes.
static String	MAXIMUM_MOVES_PER_TARGET Deprecated. Moves are now decided on a case by case basis. An alternative parameter might be introduced in the future to restrict the rate at which a DS can shed shards by moving them to other nodes. Note: This is also used to disable moves by some of the unit tests so we need a way to replace that functionality before this can be taken out.
static String	MAXIMUM_OPTIONAL_MERGES_PER_OVERFLOW Deprecated. merges are now performed in priority order while time remains in a given asynchronous overflow cycle.
static String	MAXIMUM_SEGMENTS_PER_VIEW Deprecated. merges are now performed in priority order while time remains in a given asynchronous overflow cycle.
static String	MERGE_SERVICE_CORE_POOL_SIZE The #of threads in the pool handling index partition merges.
static String	MINIMUM_ACTIVE_INDEX_PARTITIONS The minimum #of active index partitions on a data service before the resource manager will consider moving an index partition to another service (default "1").
static String	MOVE_PERCENT_CPU_TIME_THRESHOLD The threshold for a service to consider itself sufficiently loaded that it will consider moving an index partition (default ".7").
static String	NOMINAL_SHARD_SIZE The nominal size on the size of a full index partition (~200MB).
static String	OVERFLOW_CANCELLED_WHEN_JOURNAL_FULL Deprecated. Asynchronous overflow processing should run to completion with a minimum goal of an incremental build for each index partition having data on the previous journal.
static String	OVERFLOW_ENABLED Boolean property determines whether or not IResourceManager.overflow() processing is enabled (default "true").
static String	OVERFLOW_MAX_COUNT Deprecated. This is no longer used, even for testing.
static String	OVERFLOW_TASKS_CONCURRENT Deprecated. by MERGE_SERVICE_CORE_POOL_SIZE and BUILD_SERVICE_CORE_POOL_SIZE.
static String	OVERFLOW_THRESHOLD Floating point property specifying the percentage of the maximum extent at which synchronous overflow processing will be triggered (default DEFAULT_OVERFLOW_THRESHOLD).
static String	OVERFLOW_TIMEOUT Deprecated. Asynchronous overflow processing should run to completion with a minimum goal of an incremental build for each index partition having data on the previous journal.
static String	PERCENT_OF_SPLIT_THRESHOLD The minimum percentage (where 1.0 corresponds to 100 percent) that an index partition must constitute of a nominal index partition before a head or tail split will be considered (default ".9").
static String	SCATTER_SPLIT_ENABLED Boolean option indicates whether or not scatter splits are allowed (default ) on this service.
static String	TAIL_SPLIT_THRESHOLD The minimum percentage (in [0:1]) of leaf splits which must be in the tail of the index partition before a tail split of an index partition will be considered (default ".4").

Fields inherited from interface com.bigdata.resources.IndexManager.Options

DEFAULT_INDEX_CACHE_CAPACITY, DEFAULT_INDEX_CACHE_TIMEOUT, DEFAULT_INDEX_SEGMENT_CACHE_CAPACITY, DEFAULT_INDEX_SEGMENT_CACHE_TIMEOUT, INDEX_CACHE_CAPACITY, INDEX_CACHE_TIMEOUT, INDEX_SEGMENT_CACHE_CAPACITY, INDEX_SEGMENT_CACHE_TIMEOUT

Fields inherited from interface com.bigdata.resources.StoreManager.Options

ACCELERATE_OVERFLOW_THRESHOLD, DATA_DIR, DEFAULT_ACCELERATE_OVERFLOW_THRESHOLD, DEFAULT_IGNORE_BAD_FILES, DEFAULT_PURGE_OLD_RESOURCES_DURING_STARTUP, DEFAULT_STORE_CACHE_CAPACITY, DEFAULT_STORE_CACHE_TIMEOUT, IGNORE_BAD_FILES, PURGE_OLD_RESOURCES_DURING_STARTUP, STORE_CACHE_CAPACITY, STORE_CACHE_TIMEOUT

Fields inherited from interface com.bigdata.journal.Options

ALTERNATE_ROOT_BLOCK, BUFFER_MODE, CREATE, CREATE_TEMP_FILE, CREATE_TIME, DEFAULT_BUFFER_MODE, DEFAULT_CREATE, DEFAULT_CREATE_TEMP_FILE, DEFAULT_DELETE_ON_CLOSE, DEFAULT_DELETE_ON_EXIT, DEFAULT_DOUBLE_SYNC, DEFAULT_FILE_LOCK_ENABLED, DEFAULT_FORCE_ON_COMMIT, DEFAULT_FORCE_WRITES, DEFAULT_HALOG_COMPRESSOR, DEFAULT_HISTORICAL_INDEX_CACHE_CAPACITY, DEFAULT_HISTORICAL_INDEX_CACHE_TIMEOUT, DEFAULT_HOT_CACHE_SIZE, DEFAULT_HOT_CACHE_THRESHOLD, DEFAULT_INITIAL_EXTENT, DEFAULT_LIVE_INDEX_CACHE_CAPACITY, DEFAULT_LIVE_INDEX_CACHE_TIMEOUT, DEFAULT_MAXIMUM_EXTENT, DEFAULT_MINIMUM_EXTENSION, DEFAULT_READ_CACHE_BUFFER_COUNT, DEFAULT_READ_ONLY, DEFAULT_USE_DIRECT_BUFFERS, DEFAULT_VALIDATE_CHECKSUM, DEFAULT_WRITE_CACHE_BUFFER_COUNT, DEFAULT_WRITE_CACHE_COMPACTION_THRESHOLD, DEFAULT_WRITE_CACHE_ENABLED, DEFAULT_WRITE_CACHE_MIN_CLEAN_LIST_SIZE, DELETE_ON_CLOSE, DELETE_ON_EXIT, DOUBLE_SYNC, FILE, FILE_LOCK_ENABLED, FORCE_ON_COMMIT, FORCE_WRITES, HALOG_COMPRESSOR, HISTORICAL_INDEX_CACHE_CAPACITY, HISTORICAL_INDEX_CACHE_TIMEOUT, HOT_CACHE_SIZE, HOT_CACHE_THRESHOLD, IGNORE_BAD_ROOT_BLOCK, INITIAL_EXTENT, JNL, LIVE_INDEX_CACHE_CAPACITY, LIVE_INDEX_CACHE_TIMEOUT, MAXIMUM_EXTENT, MEM_MAX_EXTENT, MINIMUM_EXTENSION, minimumInitialExtent, minimumMinimumExtension, OFFSET_BITS, OTHER_MAX_EXTENT, READ_CACHE_BUFFER_COUNT, READ_ONLY, RW_MAX_EXTENT, SEG, TMP_DIR, UPDATE_ICU_VERSION, USE_DIRECT_BUFFERS, VALIDATE_CHECKSUM, WRITE_CACHE_BUFFER_COUNT, WRITE_CACHE_COMPACTION_THRESHOLD, WRITE_CACHE_ENABLED, WRITE_CACHE_MIN_CLEAN_LIST_SIZE

Fields inherited from interface com.bigdata.service.IServiceShutdown.Options

DEFAULT_SHUTDOWN_TIMEOUT, SHUTDOWN_TIMEOUT

Field Detail

OVERFLOW_ENABLED

static final String OVERFLOW_ENABLED

Boolean property determines whether or not IResourceManager.overflow() processing is enabled (default "true"). When disabled the journal will grow without bounds, IndexSegments will never be generated and index partitions will not be split, joined nor moved away from this ResourceManager.

DEFAULT_OVERFLOW_ENABLED

static final String DEFAULT_OVERFLOW_ENABLED

See Also:

Constant Field Values

OVERFLOW_MAX_COUNT

static final String OVERFLOW_MAX_COUNT

Deprecated. This is no longer used, even for testing.

Option may be used to permit a fixed number of synchronous overflow operations after which overflow is disabled (default "0"). When ZERO (0) there is no limit on the #of synchronous overflow operations. This option is mainly used for testing, but it can be enabled if you want higher throughput (for a while) and you know that the data will be well distributed on the federation after N overflows. Once synchronous overflow is disabled, all future writes will be buffered by the live journal and index partition builds, merges, splits, joins, and moves will no longer be executed. Eventually the live journal extent will grow large enough that throughput will drop (due to IOWAIT on random seeks against the journal) and it is possible that the maximum possible journal extent can be exceeded unless you also configure Options.OFFSET_BITS for scale-up.

DEFAULT_OVERFLOW_MAX_COUNT

static final String DEFAULT_OVERFLOW_MAX_COUNT

See Also:

Constant Field Values

OVERFLOW_THRESHOLD

static final String OVERFLOW_THRESHOLD

Floating point property specifying the percentage of the maximum extent at which synchronous overflow processing will be triggered (default DEFAULT_OVERFLOW_THRESHOLD). The value is multiplied into the configured Options.MAXIMUM_EXTENT. If the result is GTE the current extend of the live journal, then synchronous overflow processing will be triggered. However, note that synchronous overflow processing can not be triggered until asynchronous overflow processing for the last journal is complete. Therefore if asynchronous overflow processing takes a long time, the overflow threshold might not be checked until after it has already been exceeded.

The main purpose of this property is to trigger overflow processing before the maximum extent is exceeded. The trigger needs to lead the maximum extent somewhat since overflow processing can not proceed until there is an exclusive lock on the write service, and tasks already running will continue to write on the live journal. Overflowing the maximum extent is not a problem as long as the BufferMode supports transparent extension of the journal. However, some BufferModes do not and therefore they can not be used reliably with the overflow manager.

DEFAULT_OVERFLOW_THRESHOLD

static final String DEFAULT_OVERFLOW_THRESHOLD

See Also:

Constant Field Values

COPY_INDEX_THRESHOLD

static final String COPY_INDEX_THRESHOLD

Index partitions having no more than this many entries as reported by a range count will be copied to the new journal during synchronous overflow processing rather than building a new index segment from the buffered writes (default "1000"). When ZERO (0), index partitions will not be copied during overflow processing (unless they are empty). While it is important to keep down the latency of synchronous overflow processing, small indices can be copied so quickly that it is worth it to avoid the heavier index segment build operation.

See Also:

DEFAULT_COPY_INDEX_THRESHOLD

DEFAULT_COPY_INDEX_THRESHOLD

static final String DEFAULT_COPY_INDEX_THRESHOLD

See Also:

Constant Field Values

ACCELERATE_SPLIT_THRESHOLD

static final String ACCELERATE_SPLIT_THRESHOLD

The #of index partitions below which we will accelerate the decision to split an index partition (default "20"). When a new scale-out index is created there is by default only a single index partition on a single IDataService. Since each index (partition) is single threaded for writes, we can increase the potential concurrency if we split the initial index partition. We accelerate decisions to split index partitions by reducing the minimum and target #of tuples per index partition for an index with fewer than the #of index partitions specified by this parameter. When ZERO (0) this feature is disabled and we do not count the #of index partitions.

DEFAULT_ACCELERATE_SPLIT_THRESHOLD

static final String DEFAULT_ACCELERATE_SPLIT_THRESHOLD

See Also:

Constant Field Values

PERCENT_OF_SPLIT_THRESHOLD

static final String PERCENT_OF_SPLIT_THRESHOLD

The minimum percentage (where 1.0 corresponds to 100 percent) that an index partition must constitute of a nominal index partition before a head or tail split will be considered (default ".9"). Values near to and greater than 1.0 are permissible and imply that the post-split leftSibling index partition will be approximately a nominal index partition. However the maximum percentage may not be greater than 2.0 (200 percent).

DEFAULT_PERCENT_OF_SPLIT_THRESHOLD

static final String DEFAULT_PERCENT_OF_SPLIT_THRESHOLD

See Also:

Constant Field Values

TAIL_SPLIT_THRESHOLD

static final String TAIL_SPLIT_THRESHOLD

The minimum percentage (in [0:1]) of leaf splits which must be in the tail of the index partition before a tail split of an index partition will be considered (default ".4").

DEFAULT_TAIL_SPLIT_THRESHOLD

static final String DEFAULT_TAIL_SPLIT_THRESHOLD

See Also:

Constant Field Values

HOT_SPLIT_THRESHOLD

static final String HOT_SPLIT_THRESHOLD

Deprecated. Hot splits are not implemented and this option does not do anything. It will be going away soon.

The minimum percentage (in [0:2]) of a nominal split before an index partition will be "hot split" (default "2.0"). Hot splits are taken by hosts which are more heavily utilized than their peers but not heavily utilized in terms of their own resources. This is basically an acceleration factor for index partition splits when a host has a relatively higher workload than its peers. The purpose of a "hot split" is to increase the potential concurrency by breaking an active index partition into two index partitions. If the writes on the index partition are evenly distributed, then this can double the concurrency if the host has spare cycles. Reasonable values are on the order of [.25:.75]. Hot splits may be effectively disabled by raising the percent of split to GTE .

DEFAULT_HOT_SPLIT_THRESHOLD

static final String DEFAULT_HOT_SPLIT_THRESHOLD

See Also:

Constant Field Values

SCATTER_SPLIT_ENABLED

static final String SCATTER_SPLIT_ENABLED

Boolean option indicates whether or not scatter splits are allowed (default ) on this service.

See Also:

IndexMetadata.Options#SCATTER_SPLIT_ENABLED

DEFAULT_SCATTER_SPLIT_ENABLED

static final String DEFAULT_SCATTER_SPLIT_ENABLED

See Also:

Constant Field Values

JOINS_ENABLED

static final String JOINS_ENABLED

Option may be used to disable index partition joins. FIXME Joins are being triggered by the scatter split and/or ACCELERATE_SPLIT_THRESHOLD behaviors since the target for the split size increases as a function of the #of index partitions. For example, a scatter split can cause the adjust nominal size of a shard to jump to its configured setting, which will cause the shards to be "undercapacity" and hence drive JOINs. In order to fix this we have to somehow discount joins, either by requiring deletes on the index partition or by waiting some #of overflows since the split, etc. Alternatively, joins could be ignored unless there are more partitions of a given index than were (or would be) produced by a scatter split. For the moment joins are disabled by default.

DEFAULT_JOINS_ENABLED

static final String DEFAULT_JOINS_ENABLED

See Also:

Constant Field Values

MINIMUM_ACTIVE_INDEX_PARTITIONS

static final String MINIMUM_ACTIVE_INDEX_PARTITIONS

The minimum #of active index partitions on a data service before the resource manager will consider moving an index partition to another service (default "1").

Note: This makes sure that we don't do a move if there are only a few active index partitions on this service. This value is also used to place an upper bound on the #of index partitions that can be moved away from this service - if we move too many (or too many at once) then this service stands a good chance of becoming under-utilized and index partitions will just bounce around which is very inefficient.

Note: Even when only a single index partition for a new scale-out index is initially allocated on this service, if it is active and growing it will eventually split into enough index partitions that we will begin to re-distribute those index partitions across the federation.

Note: Index partitions are considered to be "active" iff ITx.UNISOLATED or ITx.READ_COMMITTED operations are run against the index partition during the life cycle of the live journal. There may be many other index partitions on the same service that either are never read or are subject only to historical reads. However, since only the current state of the index partition is moved, not its history, moving index partitions which are only the target for historical reads will not reduce the load on the service. Instead, read burdens are reduced using replication.

See Also:

DEFAULT_MINIMUM_ACTIVE_INDEX_PARTITIONS

DEFAULT_MINIMUM_ACTIVE_INDEX_PARTITIONS

static final String DEFAULT_MINIMUM_ACTIVE_INDEX_PARTITIONS

See Also:

Constant Field Values

MAXIMUM_MOVES

static final String MAXIMUM_MOVES

Deprecated. Moves are now decided on a case by case basis. An alternative parameter might be introduced in the future to restrict the rate at which a DS can shed shards by moving them to other nodes.

This is the maximum #of index partitions that the resource manager is willing to move in a given overflow operations across all of the identified under-utilized services (default "3").

Note: Index partition moves MAY be disabled by setting this property to ZERO (0).

See Also:

DEFAULT_MAXIMUM_MOVES

DEFAULT_MAXIMUM_MOVES

static final String DEFAULT_MAXIMUM_MOVES

See Also:

Constant Field Values

MAXIMUM_MOVES_PER_TARGET

static final String MAXIMUM_MOVES_PER_TARGET

Deprecated. Moves are now decided on a case by case basis. An alternative parameter might be introduced in the future to restrict the rate at which a DS can shed shards by moving them to other nodes.

Note: This is also used to disable moves by some of the unit tests so we need a way to replace that functionality before this can be taken out.

This is the maximum #of index partitions that the resource manager is willing to move in a given overflow operation onto each identified under-utilized service (default "2").

Note: Index partitions are moved to the identified under-utilized services using a round-robin approach which aids in distributing the load across the federation.

Note: Index partition moves MAY be disabled by setting this property to ZERO (0).

See Also:

DEFAULT_MAXIMUM_MOVES_PER_TARGET

DEFAULT_MAXIMUM_MOVES_PER_TARGET

static final String DEFAULT_MAXIMUM_MOVES_PER_TARGET

See Also:

Constant Field Values

MAXIMUM_MOVE_PERCENT_OF_SPLIT

static final String MAXIMUM_MOVE_PERCENT_OF_SPLIT

This is the maximum percentage (in [0:2]) of a full index partition which will be considered for a move (default ".8").

See Also:

DEFAULT_MAXIMUM_MOVE_PERCENT_OF_SPLIT

DEFAULT_MAXIMUM_MOVE_PERCENT_OF_SPLIT

static final String DEFAULT_MAXIMUM_MOVE_PERCENT_OF_SPLIT

See Also:

Constant Field Values

MOVE_PERCENT_CPU_TIME_THRESHOLD

static final String MOVE_PERCENT_CPU_TIME_THRESHOLD

The threshold for a service to consider itself sufficiently loaded that it will consider moving an index partition (default ".7"). This threshold IS NOT considered for scatter splits, since the goal there is to distribute the data evenly across the federation.

DEFAULT_MOVE_PERCENT_CPU_TIME_THRESHOLD

static final String DEFAULT_MOVE_PERCENT_CPU_TIME_THRESHOLD

See Also:

Constant Field Values

MAXIMUM_OPTIONAL_MERGES_PER_OVERFLOW

static final String MAXIMUM_OPTIONAL_MERGES_PER_OVERFLOW

Deprecated. merges are now performed in priority order while time remains in a given asynchronous overflow cycle.

The maximum #of optional compacting merge operations that will be performed during a single overflow event (default "2").

Once this #of optional compacting merge tasks have been identified for a given overflow event, the remainder of the index partitions that are neither split, joined, moved, nor copied will use incremental builds. An incremental build is generally cheaper since it only copies the data on the mutable BTree for the lastCommitTime rather than the fused view. A compacting merge permits the older index segments to be released and results in a simpler view with view IndexSegments. Either a compacting merge or an incremental build will permit old journals to be released once the commit points on those journals are no longer required.

Note: Mandatory compacting merges are identified based on MAXIMUM_JOURNALS_PER_VIEW and MAXIMUM_SEGMENTS_PER_VIEW. There is NO limit the #of mandatory compacting merges that will be performed during an asynchronous overflow event. However, each mandatory compacting merge does count towards the maximum #of optional merges. Therefore if the #of mandatory compacting merges is greater than this parameter then NO optional compacting merges will be selected in a given overflow cycle.

DEFAULT_OPTIONAL_COMPACTING_MERGES_PER_OVERFLOW

static final String DEFAULT_OPTIONAL_COMPACTING_MERGES_PER_OVERFLOW

See Also:

Constant Field Values

MAXIMUM_JOURNALS_PER_VIEW

static final String MAXIMUM_JOURNALS_PER_VIEW

Deprecated. merges are now performed in priority order while time remains in a given asynchronous overflow cycle.

A compacting merge will be triggered when the #of journals in an index partition view is GTE to this value (default "3"). The minimum value is TWO (2) since there will be two journals in a view when an index partition overflows and OverflowActionEnum.Copy is not selected. As long as index partition splits, builds or merges are performed the #of journals in the view WILL NOT exceed 2 and will always be ONE (1) after an asynchronous overflow in which a split, build or merge was performed.

It is extremely important to perform compacting merges in order to release dependencies on old resources (both journals and index segments) and keep down the #of sources in a view. This is especially true when those sources are journals. Journals are organized by write access, not read access. Once the backing buffer for a journal is released there will be large spikes in IOWAIT when reading on an old journal as reads are more or less random.

Note: The MAXIMUM_OPTIONAL_MERGES_PER_OVERFLOW will be ignored if a compacting merge is recommended for an index partition based on this parameter.

Note: Synchronous overflow will refuse to copy tuples for an index partition whose mutable BTree otherwise satisfies the COPY_INDEX_THRESHOLD if the #of sources in the view exceeds thresholds which demand a compacting merge.

DEFAULT_MAXIMUM_JOURNALS_PER_VIEW

static final String DEFAULT_MAXIMUM_JOURNALS_PER_VIEW

See Also:

Constant Field Values

MAXIMUM_SEGMENTS_PER_VIEW

static final String MAXIMUM_SEGMENTS_PER_VIEW

Deprecated. merges are now performed in priority order while time remains in a given asynchronous overflow cycle.

A compacting merge will be triggered when the #of index segments in an index partition view is GTE to this value (default "6").

It is extremely important to perform compacting merges in order to release dependencies on old resources (both journals and index segments) and keep down the #of sources in a view. However, this is less important when those resources are IndexSegments since they are very efficient for read operations. In this case the main driver is to reduce the complexity of the view, to require fewer open index segments (and associated resources) in order to materialize the view, and to make it possible to release index segments and thus have less of a footprint on the disk.

Note: The MAXIMUM_OPTIONAL_MERGES_PER_OVERFLOW will be ignored if a compacting merge is recommended for an index partition based on this parameter.

Note: Synchronous overflow will refuse to copy tuples for an index partition whose mutable BTree otherwise satisfies the COPY_INDEX_THRESHOLD if the #of sources in the view exceeds thresholds which demand a compacting merge.

DEFAULT_MAXIMUM_SEGMENTS_PER_VIEW

static final String DEFAULT_MAXIMUM_SEGMENTS_PER_VIEW

See Also:

Constant Field Values

MAXIMUM_BUILD_SEGMENT_BYTES

static final String MAXIMUM_BUILD_SEGMENT_BYTES

Option limits the #of IndexSegmentStore bytes that an OverflowActionEnum.Build operation will process (default "20971520"). Given that the nominal size of an index partition is 200M, a reasonable value for this might be 1/10th to 1/5th of that, so 20-40M. The key is to keep the builds fast so they should not do too much work while reducing the frequency with which we must do a compacting merge. This option only effects the #of IndexSegments that will be incorporated into an OverflowActionEnum.Build operation. When ZERO (0L), OverflowActionEnum.Build operations will only include the data from the historical journal.

DEFAULT_MAXIMUM_BUILD_SEGMENTS_BYTES

static final String DEFAULT_MAXIMUM_BUILD_SEGMENTS_BYTES

See Also:

Constant Field Values

OVERFLOW_TIMEOUT

static final String OVERFLOW_TIMEOUT

Deprecated. Asynchronous overflow processing should run to completion with a minimum goal of an incremental build for each index partition having data on the previous journal.

The timeout in milliseconds for asynchronous overflow processing to complete (default DEFAULT_OVERFLOW_TIMEOUT). Any overflow task that does not complete within this timeout will be canceled.

Asynchronous overflow processing is responsible for splitting, moving, and joining index partitions. The asynchronous overflow tasks are written to fail "safe". Also, each task may succeed or fail on its own. Iff the task succeeds, then its effect is made restart safe. Otherwise clients continue to use the old view of the index partition.

If asynchronous overflow processing DOES NOT complete each time then we run several very serious and non-sustainable risks, including: (a) the #of sources in a view can increase without limit; and (b) the #of journal that must be retained can increase without limit.

DEFAULT_OVERFLOW_TIMEOUT

static final String DEFAULT_OVERFLOW_TIMEOUT

The default timeout in milliseconds for asynchronous overflow processing (equivalent to 10 minutes).

See Also:

Constant Field Values

OVERFLOW_TASKS_CONCURRENT

static final String OVERFLOW_TASKS_CONCURRENT

Deprecated. by MERGE_SERVICE_CORE_POOL_SIZE and BUILD_SERVICE_CORE_POOL_SIZE.

The #of threads used to execute the asynchronous overflow tasks in parallel, ZERO (0) to execute ALL asynchronous overflow tasks in parallel, or ONE (1) to execute the asynchronous overflow tasks sequentially (default "0").

DEFAULT_OVERFLOW_TASKS_CONCURRENT

static final String DEFAULT_OVERFLOW_TASKS_CONCURRENT

See Also:

Constant Field Values

OVERFLOW_CANCELLED_WHEN_JOURNAL_FULL

static final String OVERFLOW_CANCELLED_WHEN_JOURNAL_FULL

Deprecated. Asynchronous overflow processing should run to completion with a minimum goal of an incremental build for each index partition having data on the previous journal.

Cancel an existing asynchronous overflow process (interrupting any running tasks) if the live journal is again approaching its maximum extent (default "true").

DEFAULT_OVERFLOW_CANCELLED_WHEN_JOURNAL_FULL

static final String DEFAULT_OVERFLOW_CANCELLED_WHEN_JOURNAL_FULL

See Also:

Constant Field Values

BUILD_SERVICE_CORE_POOL_SIZE

static final String BUILD_SERVICE_CORE_POOL_SIZE

The #of threads in the pool handling index segment builds from the old journal.

DEFAULT_BUILD_SERVICE_CORE_POOL_SIZE

static final String DEFAULT_BUILD_SERVICE_CORE_POOL_SIZE

See Also:

Constant Field Values

MERGE_SERVICE_CORE_POOL_SIZE

static final String MERGE_SERVICE_CORE_POOL_SIZE

The #of threads in the pool handling index partition merges.

DEFAULT_MERGE_SERVICE_CORE_POOL_SIZE

static final String DEFAULT_MERGE_SERVICE_CORE_POOL_SIZE

See Also:

Constant Field Values

NOMINAL_SHARD_SIZE

static final String NOMINAL_SHARD_SIZE

The nominal size on the size of a full index partition (~200MB). Index partitions are split once they reach or exceed this size. The space on the journal is not considered when making this decision since it can not readily be attributed to any given index partition.

Note: If you modify this, you may also need to modify the size of the buffers in the DirectBufferPool used to fully buffer the nodes region of the index segment file.

DEFAULT_NOMINAL_SHARD_SIZE

static final String DEFAULT_NOMINAL_SHARD_SIZE

See Also:

Constant Field Values