com.bigdata.service

Class AbstractScaleOutFederation<T>

java.lang.Object

com.bigdata.service.AbstractFederation<T>

com.bigdata.service.AbstractScaleOutFederation<T>

Type Parameters:

T - The generic type of the client or service.

All Implemented Interfaces:

ICounterSetAccess, IGISTManager, IIndexManager, IIndexStore, IBigdataFederation<T>, IFederationDelegate<T>

Direct Known Subclasses:

AbstractDistributedFederation, EmbeddedFederation

public abstract class AbstractScaleOutFederation<T>
extends AbstractFederation<T>

Abstract base class for federation implementations using the scale-out index architecture (federations that support key-range partitioned indices).

Version:

$Id$

Author:

Bryan Thompson

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AbstractScaleOutFederation.ForceOverflowTask Task forces immediate overflow of the specified data service, returning once both synchronous AND asynchronous overflow are complete.
static class	AbstractScaleOutFederation.PurgeResourcesTask Task directs a DataService to purge any unused resources and to optionally truncate the extent of the live journal.

Nested classes/interfaces inherited from class com.bigdata.service.AbstractFederation

AbstractFederation.ReportTask, AbstractFederation.StartDeferredTasksTask

Field Summary

Fields

Modifier and Type	Field and Description
protected AbstractScaleOutClient.MetadataIndexCachePolicy	metadataIndexCachePolicy

Fields inherited from class com.bigdata.service.AbstractFederation

log

Constructor Summary

Constructors

Constructor and Description
AbstractScaleOutFederation(IBigdataClient<T> client)

Method Summary

Methods

Modifier and Type	Method and Description
UUID[]	awaitServices(int minDataServices, long timeout) Await the availability of an IMetadataService and the specified minimum #of IDataServices.
void	forceOverflow(boolean compactingMerge, boolean truncateJournal) Force overflow of each data service in the scale-out federation (only scale-out federations support overflow processing).
ClientIndexView	getIndex(String name, long timestamp) Strengthens the return type.
protected IndexCache	getIndexCache() Return the cache for IIndex objects.
IMetadataIndex	getMetadataIndex(String name, long timestamp) Return a read-only view onto an IMetadataIndex.
protected MetadataIndexCache	getMetadataIndexCache() Return the cache for IMetadataIndex objects.
boolean	isScaleOut() Return true.
Iterator<PartitionLocator>	locatorScan(String name, long timestamp, byte[] fromKey, byte[] toKey, boolean reverseScan) Returns an iterator that will visit the PartitionLocators for the specified scale-out index key range.
void	shutdown() Normal shutdown allows any existing client requests to federation services to complete but does not schedule new requests, disconnects from the federation, and then terminates any background processing that is being performed on the behalf of the client (service discovery, etc).
void	shutdownNow() Immediate shutdown terminates any client requests to federation services, disconnects from the federation, and then terminate any background processing that is being performed on the behalf of the client (service discovery, etc).

Methods inherited from class com.bigdata.service.AbstractFederation

addScheduledTask, assertOpen, destroy, didStart, dropIndex, getClient, getCollectPlatformStatistics, getCollectQueueStatistics, getCounters, getDataServices, getExecutorService, getGangliaService, getGlobalFileSystem, getGlobalRowStore, getGlobalRowStore, getHostCounterSet, getHttpdPort, getHttpdURL, getIndexCounters, getResourceLocator, getScheduledExecutorService, getService, getServiceCounterPathPrefix, getServiceCounterPathPrefix, getServiceCounterSet, getServiceIface, getServiceName, getServiceUUID, getTaskCounters, getTempStore, indexNameScan, isGroupCommit, isOpen, isServiceReady, newHttpd, reattachDynamicCounters, registerIndex, registerIndex, registerIndex, reportCounters, sendEvent, serviceJoin, serviceLeave

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.bigdata.service.IBigdataFederation

getAnyDataService, getDataService, getDataServiceByName, getDataServiceUUIDs, getLastCommitTime, getLoadBalancerService, getMetadataService, getTransactionService, isDistributed, isJiniFederation, isStable

Methods inherited from interface com.bigdata.journal.IIndexStore

getResourceLockService

Field Detail

metadataIndexCachePolicy

protected final AbstractScaleOutClient.MetadataIndexCachePolicy metadataIndexCachePolicy

Constructor Detail

AbstractScaleOutFederation

public AbstractScaleOutFederation(IBigdataClient<T> client)

Parameters:

client -

Method Detail

getIndex

public ClientIndexView getIndex(String name,
                       long timestamp)

Strengthens the return type. Applies an AbstractIndexCache and strengthens the return type. Obtain a view on a partitioned index.

Specified by:

getIndex in interface IIndexManager

Specified by:

getIndex in interface IBigdataFederation<T>

Overrides:

getIndex in class AbstractFederation<T>

Parameters:

name - The index name.

timestamp - A transaction identifier, ITx.UNISOLATED for the unisolated index view, ITx.READ_COMMITTED, or timestamp for a historical view no later than the specified timestamp.

Returns:

The index or null if the index does not exist.

See Also:

IGISTLocalManager.getIndexLocal(String, long)

shutdown

public void shutdown()

Normal shutdown allows any existing client requests to federation services to complete but does not schedule new requests, disconnects from the federation, and then terminates any background processing that is being performed on the behalf of the client (service discovery, etc).

Note: concrete implementations MUST extend this method.

Note: Clients use IBigdataClient.disconnect(boolean) to disconnect from a federation. The federation implements that disconnect using either AbstractFederation.shutdown() or AbstractFederation.shutdownNow().

The implementation must be a NOP if the federation is already shutdown.

Extended to clear the various caches.

Overrides:

shutdown in class AbstractFederation<T>

shutdownNow

public void shutdownNow()

Immediate shutdown terminates any client requests to federation services, disconnects from the federation, and then terminate any background processing that is being performed on the behalf of the client (service discovery, etc).

Note: concrete implementations MUST extend this method to either disconnect from the remote federation or close the embedded federation and then clear the #fed reference so that the client is no longer "connected" to the federation.

Note: Clients use IBigdataClient.disconnect(boolean) to disconnect from a federation. The federation implements that disconnect using either AbstractFederation.shutdown() or AbstractFederation.shutdownNow().

The implementation must be a NOP if the federation is already shutdown.

Extended to clear the various caches.

Overrides:

shutdownNow in class AbstractFederation<T>

getMetadataIndex

public IMetadataIndex getMetadataIndex(String name,
                              long timestamp)

Return a read-only view onto an IMetadataIndex.

Parameters:

name - The name of the scale-out index.

timestamp - The timestamp for the view.

Returns:

The IMetadataIndex for the named scale-out index -or- null iff there is no such scale-out index.

locatorScan

public Iterator<PartitionLocator> locatorScan(String name,
                                     long timestamp,
                                     byte[] fromKey,
                                     byte[] toKey,
                                     boolean reverseScan)

Returns an iterator that will visit the PartitionLocators for the specified scale-out index key range.

The method fetches a chunk of locators at a time from the metadata index. Unless the #of index partitions spanned is very large, this will be an atomic read of locators from the metadata index. When the #of index partitions spanned is very large, then this will allow a chunked approach.

Note: It is possible that a split, join or move could occur during the process of mapping the procedure across the index partitions. When the view is ITx.UNISOLATED or ITx.READ_COMMITTED this could make the set of mapped index partitions inconsistent in the sense that it might double count some parts of the key range or that it might skip some parts of the key range. In order to avoid this problem the caller MUST use read-consistent semantics. If the ClientIndexView is not already isolated by a transaction, then the caller MUST create a read-only transaction use the global last commit time of the federation.

Parameters:

name - The name of the scale-out index.

timestamp - The timestamp of the view. It is the responsibility of the caller to choose timestamp so as to provide read-consistent semantics for the locator scan.

fromKey - The scale-out index first key that will be visited (inclusive). When null there is no lower bound.

toKey - The first scale-out index key that will NOT be visited (exclusive). When null there is no upper bound.

reverseScan - true if you need to visit the index partitions in reverse key order (this is done when the partitioned iterator is scanning backwards).

Returns:

The iterator.

Throws:

IllegalArgumentException - if name is null.

NoSuchIndexException - if there is no such scale-out index at the specified timestamp

isScaleOut

public final boolean isScaleOut()

Return true.

See Also:

IndexMetadata

getIndexCache

protected IndexCache getIndexCache()

Description copied from class: AbstractFederation

Return the cache for IIndex objects.

Specified by:

getIndexCache in class AbstractFederation<T>

getMetadataIndexCache

protected MetadataIndexCache getMetadataIndexCache()

Return the cache for IMetadataIndex objects.

awaitServices

public UUID[] awaitServices(int minDataServices,
                   long timeout)
                     throws InterruptedException,
                            TimeoutException

Await the availability of an IMetadataService and the specified minimum #of IDataServices.

Parameters:

minDataServices - The minimum #of data services.

timeout - The timeout (ms).

Returns:

An array #of the UUIDs of the IDataServices that have been discovered by this client. Note that at least minDataServices elements will be present in this array but that ALL discovered data services MAY be reported.

Throws:

IllegalArgumentException - if minDataServices is non-positive.

IllegalArgumentException - if timeout is non-positive.

IllegalStateException - if the client is not connected to the federation.

InterruptedException - if this thread is interrupted while awaiting the availability of the MetadataService or the specified #of DataServices.

TimeoutException - If a timeout occurs.

forceOverflow

public void forceOverflow(boolean compactingMerge,
                 boolean truncateJournal)

Force overflow of each data service in the scale-out federation (only scale-out federations support overflow processing). This method is synchronous. It will not return until all DataServices have initiated and completed overflow processing. Any unused resources (as determined by the StoreManager) will have been purged.

This is a relatively fast operation when compactingMerge := false. By specifying both compactingMerge := false and truncateJournal := false you can cause the data services to close out their current journals against further writes. While this is not a global synchronous operation, it can provide a basis to obtain a "near synchronous" snapshot from the federation consisting of all writes up to the point where overflow was triggered on each data service.

Parameters:

compactingMerge - When true, each shard on each IDataService will undergo a compacting merge. Synchronous parallel compacting merge of all shards is an expensive operation. This parameter shoudl normally be false unless you are requesting a compacting merge for specific purposes, such as benchmarking when all data is known to exist in one IndexSegment per shard.

truncateJournal - When true, the live journal will be truncated to its minimum extent (all writes will be preserved but there will be no free space left in the journal). This may be used to force the DataService to its minimum possible footprint.