public abstract class DataService extends AbstractService implements IDataService, IServiceShutdown, ISession
IDataService. The service is
started using the DataServer class. Operations are submitted using an
IConcurrencyManager.submit(AbstractTask) and will run with the
appropriate concurrency controls as imposed by that method.DataServer, which is used to start this service.| Modifier and Type | Class and Description |
|---|---|
static class |
DataService.DataServiceFederationDelegate
Delegate handles custom counters for the
ResourceManager, local
AbstractTransactionService and the ConcurrencyManager, dynamic
re-attachment of counters, etc. |
class |
DataService.DataServiceTransactionManager
Concrete implementation manages the local state of transactions executing
on a
DataService. |
static class |
DataService.GetIndexMetadataTask
Retrieves the
IndexMetadata for the named index as of the
specified timestamp. |
static interface |
DataService.IDataServiceCounters
Interface defines and documents the counters and counter namespaces
reported by the
DataService and the various services which it
uses. |
static interface |
DataService.Options
Options understood by the
DataService. |
protected static class |
DataService.RangeIteratorTask
Task for running a rangeIterator operation.
|
protected static class |
DataService.ReadBlockCounters |
| Modifier and Type | Field and Description |
|---|---|
protected static org.apache.log4j.Logger |
log |
| Modifier | Constructor and Description |
|---|---|
protected |
DataService(Properties properties)
Core constructor - you MUST
start() the DataService
before it can be used. |
| Modifier and Type | Method and Description |
|---|---|
void |
abort(long tx)
Request abort of the transaction by the data service.
|
void |
destroy()
Destroy the service.
|
void |
dropIndex(String name)
Drops the named index.
|
void |
forceOverflow(boolean immediate,
boolean compactingMerge)
Method sets a flag that will force overflow processing during the next
group commit and optionally forces a group commit (Note: This
method exists primarily for unit tests and benchmarking activities and
SHOULD NOT be used on a deployed federation as the overhead associated
with a compacting merge of each index partition can be significant).
|
long |
getAsynchronousOverflowCounter()
The #of asynchronous overflows that have taken place on this data service
(the counter is not restart safe).
|
ConcurrencyManager |
getConcurrencyManager()
The object used to control access to the local resources.
|
protected File |
getHTTPDURLFile()
The file on which the URL of the embedded httpd service is written.
|
IndexMetadata |
getIndexMetadata(String name,
long timestamp)
Return the metadata for the named index.
|
static String |
getIndexPartitionName(String name,
int partitionId)
Forms the name of the index corresponding to a partition of a named
scale-out index as name#partitionId.
|
ILocalTransactionManager |
getLocalTransactionManager()
The object used to coordinate transactions executing against local
resources.
|
Properties |
getProperties()
An object wrapping the properties specified to the ctor.
|
IQueryPeer |
getQueryEngine()
The object used to support distributed query against an
IBigdataFederation. |
ResourceManager |
getResourceManager()
The object used to manage the local resources.
|
Class |
getServiceIface()
Returns either
IDataService or IMetadataService as
appropriate. |
Session |
getSession()
A transient and dynamic property set (aka session).
|
boolean |
isOpen()
Note: "open" is judged by the
ConcurrencyManager.isOpen() but the
DataService is not usable until StoreManager.isStarting()
returns false (there is asynchronous processing involved
in reading the existing store files or creating the first store file and
you can not use the DataService until that processing has been
completed). |
boolean |
isOverflowActive()
Return
true iff the data service is currently engaged in
overflow processing. |
protected IResourceManager |
newResourceManager(Properties properties)
Returns the
IResourceManager. |
void |
prepare(long tx,
long revisionTime)
Request that the
IDataService participate in a 3-phase commit. |
boolean |
purgeOldResources(long timeout,
boolean truncateJournal)
This attempts to pause the service accepting
ITx.UNISOLATED
writes and then purges any resources that are no longer required based on
the StoreManager.Options#MIN_RELEASE_AGE. |
ResultSet |
rangeIterator(long tx,
String name,
byte[] fromKey,
byte[] toKey,
int capacity,
int flags,
IFilter filter)
Streaming traversal of keys and/or values in a key range.
|
IBlock |
readBlock(IResourceMetadata resource,
long addr)
Read a low-level record from the described
IRawStore described by
the IResourceMetadata. |
void |
registerIndex(String name,
IndexMetadata metadata)
Register a named mutable index on the
DataService. |
void |
setReleaseTime(long releaseTime)
Notify a data service that it MAY release data required to support views
for up to the specified releaseTime .
|
void |
shutdown()
Polite shutdown does not accept new requests and will shutdown once the
existing requests have been processed.
|
void |
shutdownNow()
Shutdown attempts to abort in-progress requests and shutdown as soon as
possible.
|
long |
singlePhaseCommit(long tx)
Note: This is basically identical to the standalone journal case.
|
DataService |
start()
Starts the
DataService. |
Future<? extends Object> |
submit(Callable<? extends Object> task)
Note: When the
DataService is accessed via RMI the Future
MUST be a proxy. |
<T> Future<T> |
submit(long tx,
String name,
IIndexProcedure<T> proc)
Note: This chooses
ITx.READ_COMMITTED if the the index has
ITx.UNISOLATED isolation and the IIndexProcedure is an
read-only operation. |
clearLoggingContext, getFederation, getHostname, getServiceName, getServiceUUID, setServiceUUID, setupLoggingContextclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitgetHostname, getServiceName, getServiceUUIDprotected DataService(Properties properties)
start() the DataService
before it can be used.properties - The configuration properties.DataService.Options,
start()public ResourceManager getResourceManager()
public IQueryPeer getQueryEngine()
IBigdataFederation.getQueryEngine in interface IDataServicepublic ConcurrencyManager getConcurrencyManager()
public ILocalTransactionManager getLocalTransactionManager()
protected IResourceManager newResourceManager(Properties properties)
IResourceManager.properties - Properties to configure that object.IResourceManager.public Properties getProperties()
public Session getSession()
ISessiongetSession in interface ISessionpublic boolean isOpen()
ConcurrencyManager.isOpen() but the
DataService is not usable until StoreManager.isStarting()
returns false (there is asynchronous processing involved
in reading the existing store files or creating the first store file and
you can not use the DataService until that processing has been
completed). The ConcurrencyManager will block for a while waiting
for the StoreManager startup to complete and will reject tasks if
startup processing does not complete within a timeout.isOpen in interface IServiceShutdownpublic DataService start()
DataService.start in class AbstractServicepublic void shutdown()
shutdown in interface IServiceShutdownshutdown in class AbstractServicepublic void shutdownNow()
shutdownNow in interface IServiceShutdownshutdownNow in class AbstractServicepublic void destroy()
IServiceDestroyAdmin#destroy().destroy in interface IServicedestroy in class AbstractServiceprotected File getHTTPDURLFile()
public void setReleaseTime(long releaseTime)
ITxCommitProtocolsetReleaseTime in interface ITxCommitProtocolreleaseTime - The new release time (strictly advanced by the transaction
manager).public long singlePhaseCommit(long tx)
throws ExecutionException,
InterruptedException,
IOException
singlePhaseCommit in interface ITxCommitProtocoltx - The transaction identifier.ExecutionException - This will wrap a ValidationError if validation fails.InterruptedException - if interrupted.IOException - if there is an RMI problem.JournalTransactionService#commitImpl(long)}.public void prepare(long tx,
long revisionTime)
throws ExecutionException,
InterruptedException,
IOException
ITxCommitProtocolIDataService participate in a 3-phase commit.
When the IDataService is sent the ITxCommitProtocol.prepare(long, long)
message it executes a task which will handle commit processing for the
transaction. That task MUST hold exclusive locks for the unisolated
indices to which the transaction write sets will be applied. While
holding those locks, the task must first validate the transaction's write
set and then merge down the write set onto the corresponding unisolated
indices using the specified revisionTime and checkpoint the
indices in order to reduce all possible sources of latency. Note that
each IDataService is able to independently prepare exactly those
parts of the transaction's write set which are mapped onto index
partitions hosted by a given IDataService.
Once validation is complete and all possible steps have been taken to
reduce sources of latency (e.g., checkpoint the indices and pre-extending
the store if necessary), the task notifies the
ITransactionService that it has prepared using
ITransactionService#prepared(long). The
ITransactionService will wait until all tasks have prepared. If a
task CAN NOT prepare the transaction, then it MUST throw an exception out
of its ITxCommitProtocol.prepare(long, long) method.
Once all tasks have send an ITransactionService#prepared(long)
message to the ITransactionService, it will assign a commitTime
to the transaction and permit those methods to return that commitTime to
the IDataServices. Once the task receives the assigned commit
time, it must obtain an exclusive write lock for the live journal (this
is a higher requirement than just an exclusive lock on the necessary
indices and will lock out all other write requests for the journal),
register the checkpointed indices on the commit list and then request a
commit of the journal using the specified commitTime. The task then
notifies the transaction service that it has completed its commit using
ITransactionService#committed(long) and awaits a response. If the
ITransactionService indicates that the commit was not successful,
the task rolls back the live journal to the prior commit point and throws
an exception out of ITxCommitProtocol.prepare(long, long).
A sample flow for successful a distributed transaction commit is shown
below. This example shows two IDataServices on which the client
has written. (If the client only writes on a single data service then we
use a single-phase commit protocol).
client -------+----txService----+--dataService1--+--dataService2--+... | [1] | commit(tx) -------- + [2] | | prepare(tx,rev) + | | [3] | | | prepare(tx,rev) ------------------+ | | | | | | <--prepared(tx) + | | | | | | <------------------- prepared(tx) + | | | "prepared" barrier [4] | | | | -- (commitTime) + | | -------------------- (commitTime) + | | [5] | | | | <--committed(tx)------------------+ | | [6] | | | <--committed(tx)+ | | | "committed" barrier [7] | | [8] | | ------ (success)+ | | [9] | | | (void)----------+ | | halt | | [10] | | ------------------------ (success)+ | | [11] | | | (void)----------------------------+ | [12] | halt | (commitTime)--------+ |
ITransactionService.commit(long)
request, in which it specifies the transaction identifier (tx). ITxCommitProtocol.prepare(long, long) requests to the participating
IDataServices, specifying the transaction identifier (tx) and
the revision timestamp (rev) to be used and then waits at a barrier until
it receives ITransactionService#prepared(long) messages from
those IDataServices.ITransactionService assigns a commitTime and returns that
commitTime as the return value for the prepared messages.IDataService obtains that commitTime, it
proceeds with its atomic commit using the specified commitTime and then
sends an ITransactionService#committed(long) message to the
ITransactionService.ITransactionService waits at another barrier.ITransactionService#committed(long) message from each
participating IDataService the transaction has been successfully
committed and the barrier breaks. The ITransactionService now
lets the ITransactionService#committed(long) messages return
true, indicating success.IDataServices return (void) from their
ITxCommitProtocol.prepare(long, long) message and the threads running their side
of the commit protocol halt.ITransactionService returns the commit time which
it assigned and which was used by each participating IDataService
to the client.
IDataServices and discard any local
state associated with the transaction and throw an exception out of
ITxCommitProtocol.prepare(long, long). Once the first barrier has been
satisfied, persistent side-effects MAY occur. Error handling in this
case must rollback the state of the live journal for each of the
participating IDataServices. If error handling was performed in
response to a local error, then the IDataService must throw that
error out of ITxCommitProtocol.prepare(long, long). However, if error handling
was initiated because ITransactionService#committed(long)
returned false then it should return normally (after
rolling back the journal).prepare in interface ITxCommitProtocoltx - The transaction identifier.revisionTime - The timestamp that will be written into the ITuples
when the write set of the validated transaction is merged down
onto the unisolated indices.IOException - if there is an RMI problem.ExecutionExceptionInterruptedExceptionpublic void abort(long tx)
throws IOException
ITxCommitProtocolITransactionService.abort(long) to each
IDataService on which the transaction has written. It is NOT sent
for read-only transactions since they have no local state on the
IDataServices.abort in interface ITxCommitProtocoltx - The transaction identifier.IOException - if there is an RMI problem.public static final String getIndexPartitionName(String name, int partitionId)
Another advantage of this naming scheme is that index partitions are just
named indices and all of the mechanisms for operating on named indices
and for concurrency control for named indices apply automatically. Among
other things, this means that different tasks can write concurrently on
different partitions of the same named index on a given
DataService.
public Class getServiceIface()
IDataService or IMetadataService as
appropriate.getServiceIface in interface IServicegetServiceIface in class AbstractServicepublic void registerIndex(String name, IndexMetadata metadata) throws IOException, InterruptedException, ExecutionException
IDataServiceDataService.
Note: In order to register an index partition the
partition metadata property
MUST be set. The resources
property will then be overriden when the index is actually registered so
as to reflect the IResourceMetadata description of the journal on
which the index actually resides.
registerIndex in interface IDataServicename - The name that can be used to recover the index. In order to
create a partition of an index you must form the name of the
index partition using
getIndexPartitionName(String, int) (this
operation is generally performed by the
IMetadataService which manages scale-out indices).metadata - The metadata describing the index.
The LocalPartitionMetadata.getResources() property on
the IndexMetadata.getPartitionMetadata() SHOULD NOT be
set. The correct IResourceMetadata[] will be assigned
when the index is registered on the IDataService.
IOExceptionInterruptedExceptionExecutionExceptionpublic void dropIndex(String name) throws IOException, InterruptedException, ExecutionException
IDataService
Note: In order to drop a partition of an index you must form the name of
the index partition using
getIndexPartitionName(String, int) (this operation is
generally performed by the IMetadataService which manages
scale-out indices).
dropIndex in interface IDataServicename - The index name.IOExceptionInterruptedExceptionExecutionExceptionpublic IndexMetadata getIndexMetadata(String name, long timestamp) throws IOException, InterruptedException, ExecutionException
IDataServicegetIndexMetadata in interface IDataServicename - The index name.timestamp - A transaction identifier, ITx.UNISOLATED for the
unisolated index view, ITx.READ_COMMITTED, or
timestamp for a historical view no later than
the specified timestamp.IOExceptionInterruptedExceptionExecutionExceptionpublic <T> Future<T> submit(long tx, String name, IIndexProcedure<T> proc)
ITx.READ_COMMITTED if the the index has
ITx.UNISOLATED isolation and the IIndexProcedure is an
read-only operation. This provides better concurrency on the
DataService by moving read-only operations off of the
WriteExecutorService.
Note: When the DataService is accessed via RMI the Future
MUST be a proxy. This gets handled by the concrete server implementation.
submit in interface IDataServicetx - The transaction identifier, ITx.UNISOLATED for an ACID
operation NOT isolated by a transaction,
ITx.READ_COMMITTED for a read-committed operation not
protected by a transaction (no global read lock), or any valid
commit time for a read-historical operation not protected by a
transaction (no global read lock).name - The name of the index partition.proc - The procedure to be executed.Future from which the outcome of the procedure may be
obtained.public Future<? extends Object> submit(Callable<? extends Object> task)
DataService is accessed via RMI the Future
MUST be a proxy. This gets handled by the concrete server implementation.submit in interface IDataServicesubmit in interface IRemoteExecutorFuture for that task.AbstractDistributedFederation.getProxy(Future)public ResultSet rangeIterator(long tx, String name, byte[] fromKey, byte[] toKey, int capacity, int flags, IFilter filter) throws InterruptedException, ExecutionException
IDataServiceStreaming traversal of keys and/or values in a key range.
Note: In order to visit all keys in a range, clients are expected to
issue repeated calls in which the fromKey is incremented to the
successor of the last key visited until either an empty ResultSet
is returned or the ResultSet#isLast() flag is set, indicating
that all keys up to (but not including) the startKey have been
visited. See ClientIndexView (scale-out indices) and
DataServiceTupleIterator (unpartitioned indices), both of which
encapsulate this method.
Note: If the iterator can be determined to be read-only and it is
submitted as ITx.UNISOLATED then it will be run as
ITx.READ_COMMITTED to improve concurrency.
rangeIterator in interface IDataServicetx - The transaction identifier -or- ITx.UNISOLATED IFF the
operation is NOT isolated by a transaction -or-
- tx to read from the most recent commit point
not later than the absolute value of tx (a fully
isolated read-only transaction using a historical start time).name - The index name (required).fromKey - The starting key for the scan (or null iff
there is no lower bound).toKey - The first key that will not be visited (or null
iff there is no upper bound).capacity - When non-zero, this is the maximum #of entries to process.flags - One or more flags formed by bitwise OR of zero or more of the
constants defined by IRangeQuery.filter - An optional object that may be used to layer additional
semantics onto the iterator. The filter will be constructed on
the server and in the execution context for the iterator, so
it will execute directly against the index for the maximum
efficiency.InterruptedException - if the operation was interrupted.ExecutionException - If the operation caused an error. See
Throwable.getCause() for the underlying
error.public IBlock readBlock(IResourceMetadata resource, long addr)
IDataServiceIRawStore described by
the IResourceMetadata.readBlock in interface IDataServiceresource - The description of the resource containing that block.addr - The address of the block in that resource.public void forceOverflow(boolean immediate,
boolean compactingMerge)
throws IOException,
InterruptedException,
ExecutionException
IDataService
Normally there is no reason to invoke this method directly. Overflow
processing is triggered automatically on a bottom-up basis when the
extent of the live journal nears the Options.MAXIMUM_EXTENT.
forceOverflow in interface IDataServiceimmediate - The purpose of this argument is to permit the caller to
trigger an overflow event even though there are no writes
being made against the data service. When true
the method will write a token record on the live journal in
order to provoke a group commit. In this case synchronous
overflow processing will have occurred by the time the method
returns. When false a flag is set and overflow
processing will occur on the next commit.compactingMerge - The purpose of this flag is to permit the caller to indicate
that a compacting merge should be performed for all indices on
the data service (at least, all indices whose data are not
simply copied onto the new journal) during the next
synchronous overflow. Note that compacting merges of indices
are performed automatically from time to time so this flag
exists mainly for people who want to force a compacting merge
for some reason.IOExceptionInterruptedException - may be thrown if immediate is true.ExecutionException - may be thrown if immediate is true.public boolean purgeOldResources(long timeout,
boolean truncateJournal)
throws InterruptedException
IDataServiceITx.UNISOLATED
writes and then purges any resources that are no longer required based on
the StoreManager.Options#MIN_RELEASE_AGE.
Note: Resources are normally purged during synchronous overflow handling. However, asynchronous overflow handling can cause resources to no longer be needed as new index partition views are defined. This method MAY be used to trigger a release before the next overflow event.
purgeOldResources in interface IDataServicetimeout - The timeout (in milliseconds) that the method will await the
pause of the write service.truncateJournal - When true, the live journal will be truncated
to its minimum extent (all writes will be preserved but there
will be no free space left in the journal). This may be used
to force the DataService to its minimum possible
footprint for the configured history retention policy.true if successful and false if the
write service could not be paused after the specified timeout.InterruptedExceptionpublic long getAsynchronousOverflowCounter()
throws IOException
IDataServicegetAsynchronousOverflowCounter in interface IDataServiceIOExceptionpublic boolean isOverflowActive()
throws IOException
IDataServicetrue iff the data service is currently engaged in
overflow processing.isOverflowActive in interface IDataServiceIOExceptionCopyright © 2006–2019 SYSTAP, LLC DBA Blazegraph. All rights reserved.