public abstract class OverflowManager extends IndexManager
WriteExecutorServices for the journal are paused and all running
tasks on those services are allowed to complete and commit. Once no writers
are running, the WriteExecutorService triggers synchronous overflow.
Synchronous overflow is a low-latency process which creates a new journal to
absorb future writes, re-defines the views for all index partitions found on
the old journal to include the new journal as their first source, and
initiates a background thread performing asynchronous overflow
post-processing.
Asynchronous overflow post-processing is responsible for identifying index
partitions overflow (resulting in a split into two or more index partitions),
index partition underflow (resulting in the join of the under-capacity index
partition with its rightSibling), index partition moves (the index partition
is moved to a different DataService), and index partition builds (an
IndexSegment is created from the current view in what is effectively
a compacting merge). Overflow processing is suspended during asynchronous
post-processing, but is automatically re-enabled once post-processing
completes.
| Modifier and Type | Class and Description |
|---|---|
static interface |
OverflowManager.IIndexPartitionTaskCounters
Performance counters for the index partition tasks.
|
static interface |
OverflowManager.IOverflowManagerCounters
Performance counters for the
OverflowManager. |
static interface |
OverflowManager.Options
Options understood by the
OverflowManager. |
static class |
OverflowManager.ResourceScores
Helper class reports performance counters of interest for this service.
|
IndexManager.IIndexManagerCounters, IndexManager.IndexSegmentStatsStoreManager.IStoreManagerCounters, StoreManager.ManagedJournal| Modifier and Type | Field and Description |
|---|---|
protected int |
accelerateSplitThreshold |
protected AtomicBoolean |
asyncOverflowEnabled
A flag used to disable the asynchronous overflow processing for some unit
tests.
|
protected int |
buildServiceCorePoolSize
The #of threads which will execute index partition build operations.
|
AtomicBoolean |
compactingMerge
A flag that may be set to force the next asynchronous overflow to perform
a compacting merge for all indices that are not simply copied over to the
new journal (the use of this flag significantly raises the time
required for asynchronous overflow processing as all shard views must be
made compact and SHOULD NOT be used for deployed federations).
|
protected boolean |
compactingMergeWithAfterAction
FIXME This is a temporary flag used to (dis|en)able the logic for
executing various index partition operations as after actions for a
compacting merge.
|
protected int |
copyIndexThreshold |
AtomicBoolean |
forceOverflow
Flag may be set to force overflow processing during the next group
commit.
|
protected boolean |
joinsEnabled |
protected static org.apache.log4j.Logger |
log
Logger.
|
protected long |
maximumBuildSegmentBytes |
protected int |
maximumJournalsPerView
Deprecated.
merges are now performed in priority order while time remains
in a given asynchronous overflow cycle.
|
protected double |
maximumMovePercentOfSplit |
protected int |
maximumMoves
Deprecated.
Moves are now decided on a case by case basis. An alternative
parameter might be introduced in the future to restrict the
rate at which a DS can shed shards by moving them to other
nodes.
|
protected int |
maximumMovesPerTarget
Deprecated.
Moves are now decided on a case by case basis. An alternative
parameter might be introduced in the future to restrict the
rate at which a DS can shed shards by moving them to other
nodes.
Note: This is also used to disable moves by some of the unit tests so we need a way to replace that functionality before this can be taken out. |
protected int |
maximumOptionalMergesPerOverflow
Deprecated.
merges are now performed in priority order while time remains
in a given asynchronous overflow cycle.
|
protected int |
maximumSegmentsPerView
Deprecated.
merges are now performed in priority order while time remains
in a given asynchronous overflow cycle.
|
protected int |
mergeServiceCorePoolSize
The #of threads which will execute index partition merge operations.
|
protected int |
minimumActiveIndexPartitions |
protected double |
movePercentCpuTimeThreshold |
long |
nominalShardSize
Index partitions are split when they approach this size on the disk.
|
protected AtomicBoolean |
overflowAllowed
A flag used to disable overflow of the live journal until asynchronous
post-processing of the old journal has been completed.
|
protected boolean |
overflowCancelledWhenJournalFull |
protected OverflowCounters |
overflowCounters
The "live" overflow counters which are maintained by the service.
|
protected int |
overflowTasksConcurrent
Deprecated.
|
protected double |
overflowThreshold |
protected long |
overflowTimeout
The timeout for asynchronous overflow processing.
|
protected double |
percentOfJoinThreshold
FIXME configuration option.
|
protected double |
percentOfSplitThreshold |
protected boolean |
scatterSplitEnabled |
protected String |
serviceName
The name of the service (iff available).
|
double |
shardOverextensionLimit
If an index partition refuses to split it will be disabled once its size
on disk (for a compact view) is greater than this multiplier.
|
protected double |
tailSplitThreshold |
buildTasks, concurrentBuildTaskCount, concurrentMergeTaskCount, staleLocatorCacheaccelerateOverflowThreshold, bytesDeleted, bytesUnderManagement, dataDir, indexCacheLock, journalBytesUnderManagement, journalDeleteCount, journalReopenCount, journalsDir, lastCommitTimePreserved, lastOverflowTime, liveJournalRef, maximumJournalSizeAtOverflow, purgeResourcesMillis, segmentBytesUnderManagement, segmentsDir, segmentStoreDeleteCount, segmentStoreReopenCount, storeCache, tmpDir| Constructor and Description |
|---|
OverflowManager(Properties properties) |
| Modifier and Type | Method and Description |
|---|---|
protected OverflowMetadata |
doSynchronousOverflow()
Synchronous overflow processing.
|
long |
getAsynchronousOverflowCount()
#of asynchronous overflows that have taken place.
|
protected double |
getHostCounter(String path,
double defaultValue)
Return the value of a host counter.
|
OverflowCounters |
getOverflowCounters()
Return a copy of the
OverflowCounters. |
protected double |
getServiceCounter(String path,
double defaultValue)
Return the value of a service counter.
|
long |
getSynchronousOverflowCount()
#of synchronous overflows that have taken place.
|
boolean |
isOverflowAllowed()
true unless an overflow event is currently being
processed. |
boolean |
isOverflowEnabled()
true if overflow processing is enabled and
false if overflow processing was disabled as a
configuration option or if a maximum overflow count was configured and
has been satisfied, in which case the live journal will NOT overflow. |
Future<Object> |
overflow()
Core method for overflow with post-processing.
|
boolean |
shouldOverflow()
An overflow condition is recognized when the journal is within some
declared percentage of
Options.MAXIMUM_EXTENT. |
void |
shutdown()
The service will no longer accept new requests, but existing requests
will be processed (sychronous).
|
void |
shutdownNow()
The service will no longer accept new requests and will make a best
effort attempt to terminate all existing requests and return ASAP.
|
buildIndexSegment, disableWrites, enableWrites, getIndex, getIndexCacheCapacity, getIndexCacheSize, getIndexCounters, getIndexCounters, getIndexOnStore, getIndexPartitionGone, getIndexRetentionTime, getIndexSegmentCacheCapacity, getIndexSegmentCacheSize, getIndexSources, getIndexSources, getStaleLocatorCount, isDisabledWrites, listIndexPartitions, markAndGetDelta, setIndexPartitionGoneaddResource, assertNotOpen, assertOpen, assertRunning, awaitRunning, deleteResource, deleteResources, getBytesUnderManagement, getCommitTimeStrictlyGreaterThan, getConcurrencyManager, getDataDir, getDataDirFreeSpace, getIndexSegmentFile, getIndexSegmentFile, getJournal, getJournalBytesUnderManagement, getLiveJournal, getManagedJournalCount, getManagedSegmentCount, getProperties, getReleaseTime, getResourceService, getResourcesForTimestamp, getSegmentBytesUnderManagement, getStoreCacheSize, getStoreCounters, getTempDirFreeSpace, getTmpDir, isOpen, isRunning, isStarting, isTransient, munge, newFileFilter, nextTimestamp, openStore, overrideJournalExtent, purgeOldResources, purgeOldResources, retentionSetAdd, retentionSetRemove, setConcurrencyManager, setReleaseTimecloseJournal, closeTx, closeUnisolatedIndex, deleteJournal, dropUnisolatedIndex, extendJournal, isolateIndex, openJournal, openTx, openUnisolatedIndexclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitgetCounters, getDataService, getDataServiceUUID, getFederationprotected static final org.apache.log4j.Logger log
protected final boolean compactingMergeWithAfterAction
protected final int copyIndexThreshold
protected final int accelerateSplitThreshold
protected final double percentOfSplitThreshold
protected final double percentOfJoinThreshold
protected final double tailSplitThreshold
protected final boolean scatterSplitEnabled
protected final boolean joinsEnabled
protected final int minimumActiveIndexPartitions
protected final int maximumMoves
protected final int maximumMovesPerTarget
Note: This is also used to disable moves by some of the unit tests so we need a way to replace that functionality before this can be taken out.
protected final double maximumMovePercentOfSplit
protected final double movePercentCpuTimeThreshold
protected final int maximumOptionalMergesPerOverflow
protected final int maximumJournalsPerView
protected final int maximumSegmentsPerView
protected final long maximumBuildSegmentBytes
protected final int buildServiceCorePoolSize
protected final int mergeServiceCorePoolSize
protected final String serviceName
protected final double overflowThreshold
protected final AtomicBoolean overflowAllowed
AsynchronousOverflowTaskprotected final AtomicBoolean asyncOverflowEnabled
public final AtomicBoolean forceOverflow
overflow().public final AtomicBoolean compactingMerge
protected final OverflowCounters overflowCounters
protected final long overflowTimeout
protected final int overflowTasksConcurrent
protected final boolean overflowCancelledWhenJournalFull
public final long nominalShardSize
public final double shardOverextensionLimit
ISimpleSplitHandler implementation
provided by the application when it registered the index. By disallowing
further writes on the shard we prevent it from dragging down performance
for the entire data service and push the problem back on the application.
In order to remedy this issue on a pre-existing index you must fix the
split handler, register the new split handler on the MDS and on each
shard on the index, and then re-enable writes for the index.public OverflowManager(Properties properties)
properties - public OverflowCounters getOverflowCounters()
OverflowCounters.public long getSynchronousOverflowCount()
getOverflowCounters()public long getAsynchronousOverflowCount()
getOverflowCounters()public boolean isOverflowEnabled()
true if overflow processing is enabled and
false if overflow processing was disabled as a
configuration option or if a maximum overflow count was configured and
has been satisfied, in which case the live journal will NOT overflow.public boolean isOverflowAllowed()
true unless an overflow event is currently being
processed.public void shutdown()
IServiceShutdownIServiceShutdown.Options.SHUTDOWN_TIMEOUT. Implementations SHOULD be
synchronized. If the service is aleady shutdown, then
this method should be a NOP.shutdown in interface IServiceShutdownshutdown in class StoreManagerpublic void shutdownNow()
IServiceShutdownshutdownNow in interface IServiceShutdownshutdownNow in class StoreManagerpublic boolean shouldOverflow()
Options.MAXIMUM_EXTENT. However, this
method will return false if overflow has been disabled
or if there is an asynchronous overflow operation in progress.true if overflow processing should occur.public Future<Object> overflow()
Note: This method does not test preconditions based on the extent of the journal.
Note: The caller is responsible for ensuring that this method is invoked with an exclusive lock on the write service.
Preconditions:
WriteExecutorServiceisOverflowAllowed()Post-conditions:
PostProcessOldJournal task was submitted.isOverflowAllowed() was set false and will
remain false until PostProcessOldJournalFuture for the task handling post-processing of the
old journal.protected OverflowMetadata doSynchronousOverflow()
This is invoked once all preconditions have been satisfied.
Index partitions that have fewer than some threshold #of index entries will be copied onto the new journal. Otherwise the view of the index will be re-defined to place writes on the new journal and read historical data from the old journal.
This uses StoreManager.purgeOldResources() to delete old resources from the
local file system that are no longer required as determined by
StoreManager.setReleaseTime(long) and #getEffectiveReleaseTime().
Note: This method does NOT start a AsynchronousOverflowTask.
Note: You MUST have an exclusive lock on the WriteExecutorService
before you invoke this method!
protected double getHostCounter(String path, double defaultValue)
path - The path (relative to the host root).defaultValue - The default value to use if the counter was not found.protected double getServiceCounter(String path, double defaultValue)
path - The path (relative to the service root).defaultValue - The default value to use if the counter was not found.Copyright © 2006–2019 SYSTAP, LLC DBA Blazegraph. All rights reserved.