public abstract class DataService extends AbstractService implements IDataService, IServiceShutdown, ISession
IDataService
. The service is
started using the DataServer
class. Operations are submitted using an
IConcurrencyManager.submit(AbstractTask)
and will run with the
appropriate concurrency controls as imposed by that method.DataServer, which is used to start this service.
Modifier and Type | Class and Description |
---|---|
static class |
DataService.DataServiceFederationDelegate
Delegate handles custom counters for the
ResourceManager , local
AbstractTransactionService and the ConcurrencyManager , dynamic
re-attachment of counters, etc. |
class |
DataService.DataServiceTransactionManager
Concrete implementation manages the local state of transactions executing
on a
DataService . |
static class |
DataService.GetIndexMetadataTask
Retrieves the
IndexMetadata for the named index as of the
specified timestamp. |
static interface |
DataService.IDataServiceCounters
Interface defines and documents the counters and counter namespaces
reported by the
DataService and the various services which it
uses. |
static interface |
DataService.Options
Options understood by the
DataService . |
protected static class |
DataService.RangeIteratorTask
Task for running a rangeIterator operation.
|
protected static class |
DataService.ReadBlockCounters |
Modifier and Type | Field and Description |
---|---|
protected static org.apache.log4j.Logger |
log |
Modifier | Constructor and Description |
---|---|
protected |
DataService(Properties properties)
Core constructor - you MUST
start() the DataService
before it can be used. |
Modifier and Type | Method and Description |
---|---|
void |
abort(long tx)
Request abort of the transaction by the data service.
|
void |
destroy()
Destroy the service.
|
void |
dropIndex(String name)
Drops the named index.
|
void |
forceOverflow(boolean immediate,
boolean compactingMerge)
Method sets a flag that will force overflow processing during the next
group commit and optionally forces a group commit (Note: This
method exists primarily for unit tests and benchmarking activities and
SHOULD NOT be used on a deployed federation as the overhead associated
with a compacting merge of each index partition can be significant).
|
long |
getAsynchronousOverflowCounter()
The #of asynchronous overflows that have taken place on this data service
(the counter is not restart safe).
|
ConcurrencyManager |
getConcurrencyManager()
The object used to control access to the local resources.
|
protected File |
getHTTPDURLFile()
The file on which the URL of the embedded httpd service is written.
|
IndexMetadata |
getIndexMetadata(String name,
long timestamp)
Return the metadata for the named index.
|
static String |
getIndexPartitionName(String name,
int partitionId)
Forms the name of the index corresponding to a partition of a named
scale-out index as name#partitionId.
|
ILocalTransactionManager |
getLocalTransactionManager()
The object used to coordinate transactions executing against local
resources.
|
Properties |
getProperties()
An object wrapping the properties specified to the ctor.
|
IQueryPeer |
getQueryEngine()
The object used to support distributed query against an
IBigdataFederation . |
ResourceManager |
getResourceManager()
The object used to manage the local resources.
|
Class |
getServiceIface()
Returns either
IDataService or IMetadataService as
appropriate. |
Session |
getSession()
A transient and dynamic property set (aka session).
|
boolean |
isOpen()
Note: "open" is judged by the
ConcurrencyManager.isOpen() but the
DataService is not usable until StoreManager.isStarting()
returns false (there is asynchronous processing involved
in reading the existing store files or creating the first store file and
you can not use the DataService until that processing has been
completed). |
boolean |
isOverflowActive()
Return
true iff the data service is currently engaged in
overflow processing. |
protected IResourceManager |
newResourceManager(Properties properties)
Returns the
IResourceManager . |
void |
prepare(long tx,
long revisionTime)
Request that the
IDataService participate in a 3-phase commit. |
boolean |
purgeOldResources(long timeout,
boolean truncateJournal)
This attempts to pause the service accepting
ITx.UNISOLATED
writes and then purges any resources that are no longer required based on
the StoreManager.Options#MIN_RELEASE_AGE . |
ResultSet |
rangeIterator(long tx,
String name,
byte[] fromKey,
byte[] toKey,
int capacity,
int flags,
IFilter filter)
Streaming traversal of keys and/or values in a key range.
|
IBlock |
readBlock(IResourceMetadata resource,
long addr)
Read a low-level record from the described
IRawStore described by
the IResourceMetadata . |
void |
registerIndex(String name,
IndexMetadata metadata)
Register a named mutable index on the
DataService . |
void |
setReleaseTime(long releaseTime)
Notify a data service that it MAY release data required to support views
for up to the specified releaseTime .
|
void |
shutdown()
Polite shutdown does not accept new requests and will shutdown once the
existing requests have been processed.
|
void |
shutdownNow()
Shutdown attempts to abort in-progress requests and shutdown as soon as
possible.
|
long |
singlePhaseCommit(long tx)
Note: This is basically identical to the standalone journal case.
|
DataService |
start()
Starts the
DataService . |
Future<? extends Object> |
submit(Callable<? extends Object> task)
Note: When the
DataService is accessed via RMI the Future
MUST be a proxy. |
<T> Future<T> |
submit(long tx,
String name,
IIndexProcedure<T> proc)
Note: This chooses
ITx.READ_COMMITTED if the the index has
ITx.UNISOLATED isolation and the IIndexProcedure is an
read-only operation. |
clearLoggingContext, getFederation, getHostname, getServiceName, getServiceUUID, setServiceUUID, setupLoggingContext
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getHostname, getServiceName, getServiceUUID
protected DataService(Properties properties)
start()
the DataService
before it can be used.properties
- The configuration properties.DataService.Options
,
start()
public ResourceManager getResourceManager()
public IQueryPeer getQueryEngine()
IBigdataFederation
.getQueryEngine
in interface IDataService
public ConcurrencyManager getConcurrencyManager()
public ILocalTransactionManager getLocalTransactionManager()
protected IResourceManager newResourceManager(Properties properties)
IResourceManager
.properties
- Properties to configure that object.IResourceManager
.public Properties getProperties()
public Session getSession()
ISession
getSession
in interface ISession
public boolean isOpen()
ConcurrencyManager.isOpen()
but the
DataService
is not usable until StoreManager.isStarting()
returns false
(there is asynchronous processing involved
in reading the existing store files or creating the first store file and
you can not use the DataService
until that processing has been
completed). The ConcurrencyManager
will block for a while waiting
for the StoreManager
startup to complete and will reject tasks if
startup processing does not complete within a timeout.isOpen
in interface IServiceShutdown
public DataService start()
DataService
.start
in class AbstractService
public void shutdown()
shutdown
in interface IServiceShutdown
shutdown
in class AbstractService
public void shutdownNow()
shutdownNow
in interface IServiceShutdown
shutdownNow
in class AbstractService
public void destroy()
IService
DestroyAdmin#destroy()
.destroy
in interface IService
destroy
in class AbstractService
protected File getHTTPDURLFile()
public void setReleaseTime(long releaseTime)
ITxCommitProtocol
setReleaseTime
in interface ITxCommitProtocol
releaseTime
- The new release time (strictly advanced by the transaction
manager).public long singlePhaseCommit(long tx) throws ExecutionException, InterruptedException, IOException
singlePhaseCommit
in interface ITxCommitProtocol
tx
- The transaction identifier.ExecutionException
- This will wrap a ValidationError
if validation fails.InterruptedException
- if interrupted.IOException
- if there is an RMI problem.JournalTransactionService#commitImpl(long)}.
public void prepare(long tx, long revisionTime) throws ExecutionException, InterruptedException, IOException
ITxCommitProtocol
IDataService
participate in a 3-phase commit.
When the IDataService
is sent the ITxCommitProtocol.prepare(long, long)
message it executes a task which will handle commit processing for the
transaction. That task MUST hold exclusive locks for the unisolated
indices to which the transaction write sets will be applied. While
holding those locks, the task must first validate the transaction's write
set and then merge down the write set onto the corresponding unisolated
indices using the specified revisionTime and checkpoint the
indices in order to reduce all possible sources of latency. Note that
each IDataService
is able to independently prepare exactly those
parts of the transaction's write set which are mapped onto index
partitions hosted by a given IDataService
.
Once validation is complete and all possible steps have been taken to
reduce sources of latency (e.g., checkpoint the indices and pre-extending
the store if necessary), the task notifies the
ITransactionService
that it has prepared using
ITransactionService#prepared(long)
. The
ITransactionService
will wait until all tasks have prepared. If a
task CAN NOT prepare the transaction, then it MUST throw an exception out
of its ITxCommitProtocol.prepare(long, long)
method.
Once all tasks have send an ITransactionService#prepared(long)
message to the ITransactionService
, it will assign a commitTime
to the transaction and permit those methods to return that commitTime to
the IDataService
s. Once the task receives the assigned commit
time, it must obtain an exclusive write lock for the live journal (this
is a higher requirement than just an exclusive lock on the necessary
indices and will lock out all other write requests for the journal),
register the checkpointed indices on the commit list and then request a
commit of the journal using the specified commitTime. The task then
notifies the transaction service that it has completed its commit using
ITransactionService#committed(long)
and awaits a response. If the
ITransactionService
indicates that the commit was not successful,
the task rolls back the live journal to the prior commit point and throws
an exception out of ITxCommitProtocol.prepare(long, long)
.
A sample flow for successful a distributed transaction commit is shown
below. This example shows two IDataService
s on which the client
has written. (If the client only writes on a single data service then we
use a single-phase commit protocol).
client -------+----txService----+--dataService1--+--dataService2--+... | [1] | commit(tx) -------- + [2] | | prepare(tx,rev) + | | [3] | | | prepare(tx,rev) ------------------+ | | | | | | <--prepared(tx) + | | | | | | <------------------- prepared(tx) + | | | "prepared" barrier [4] | | | | -- (commitTime) + | | -------------------- (commitTime) + | | [5] | | | | <--committed(tx)------------------+ | | [6] | | | <--committed(tx)+ | | | "committed" barrier [7] | | [8] | | ------ (success)+ | | [9] | | | (void)----------+ | | halt | | [10] | | ------------------------ (success)+ | | [11] | | | (void)----------------------------+ | [12] | halt | (commitTime)--------+ |
ITransactionService.commit(long)
request, in which it specifies the transaction identifier (tx). ITxCommitProtocol.prepare(long, long)
requests to the participating
IDataService
s, specifying the transaction identifier (tx) and
the revision timestamp (rev) to be used and then waits at a barrier until
it receives ITransactionService#prepared(long)
messages from
those IDataService
s.ITransactionService
assigns a commitTime and returns that
commitTime as the return value for the prepared messages.IDataService
obtains that commitTime, it
proceeds with its atomic commit using the specified commitTime and then
sends an ITransactionService#committed(long)
message to the
ITransactionService
.ITransactionService
waits at another barrier.ITransactionService#committed(long)
message from each
participating IDataService
the transaction has been successfully
committed and the barrier breaks. The ITransactionService
now
lets the ITransactionService#committed(long)
messages return
true
, indicating success.IDataService
s return (void) from their
ITxCommitProtocol.prepare(long, long)
message and the threads running their side
of the commit protocol halt.ITransactionService
returns the commit time which
it assigned and which was used by each participating IDataService
to the client.
IDataService
s and discard any local
state associated with the transaction and throw an exception out of
ITxCommitProtocol.prepare(long, long)
. Once the first barrier has been
satisfied, persistent side-effects MAY occur. Error handling in this
case must rollback the state of the live journal for each of the
participating IDataService
s. If error handling was performed in
response to a local error, then the IDataService
must throw that
error out of ITxCommitProtocol.prepare(long, long)
. However, if error handling
was initiated because ITransactionService#committed(long)
returned false
then it should return normally (after
rolling back the journal).prepare
in interface ITxCommitProtocol
tx
- The transaction identifier.revisionTime
- The timestamp that will be written into the ITuple
s
when the write set of the validated transaction is merged down
onto the unisolated indices.IOException
- if there is an RMI problem.ExecutionException
InterruptedException
public void abort(long tx) throws IOException
ITxCommitProtocol
ITransactionService.abort(long)
to each
IDataService
on which the transaction has written. It is NOT sent
for read-only transactions since they have no local state on the
IDataService
s.abort
in interface ITxCommitProtocol
tx
- The transaction identifier.IOException
- if there is an RMI problem.public static final String getIndexPartitionName(String name, int partitionId)
Another advantage of this naming scheme is that index partitions are just
named indices and all of the mechanisms for operating on named indices
and for concurrency control for named indices apply automatically. Among
other things, this means that different tasks can write concurrently on
different partitions of the same named index on a given
DataService
.
public Class getServiceIface()
IDataService
or IMetadataService
as
appropriate.getServiceIface
in interface IService
getServiceIface
in class AbstractService
public void registerIndex(String name, IndexMetadata metadata) throws IOException, InterruptedException, ExecutionException
IDataService
DataService
.
Note: In order to register an index partition the
partition metadata
property
MUST be set. The resources
property will then be overriden when the index is actually registered so
as to reflect the IResourceMetadata
description of the journal on
which the index actually resides.
registerIndex
in interface IDataService
name
- The name that can be used to recover the index. In order to
create a partition of an index you must form the name of the
index partition using
getIndexPartitionName(String, int)
(this
operation is generally performed by the
IMetadataService
which manages scale-out indices).metadata
- The metadata describing the index.
The LocalPartitionMetadata.getResources()
property on
the IndexMetadata.getPartitionMetadata()
SHOULD NOT be
set. The correct IResourceMetadata
[] will be assigned
when the index is registered on the IDataService
.
IOException
InterruptedException
ExecutionException
public void dropIndex(String name) throws IOException, InterruptedException, ExecutionException
IDataService
Note: In order to drop a partition of an index you must form the name of
the index partition using
getIndexPartitionName(String, int)
(this operation is
generally performed by the IMetadataService
which manages
scale-out indices).
dropIndex
in interface IDataService
name
- The index name.IOException
InterruptedException
ExecutionException
public IndexMetadata getIndexMetadata(String name, long timestamp) throws IOException, InterruptedException, ExecutionException
IDataService
getIndexMetadata
in interface IDataService
name
- The index name.timestamp
- A transaction identifier, ITx.UNISOLATED
for the
unisolated index view, ITx.READ_COMMITTED
, or
timestamp
for a historical view no later than
the specified timestamp.IOException
InterruptedException
ExecutionException
public <T> Future<T> submit(long tx, String name, IIndexProcedure<T> proc)
ITx.READ_COMMITTED
if the the index has
ITx.UNISOLATED
isolation and the IIndexProcedure
is an
read-only operation. This provides better concurrency on the
DataService
by moving read-only operations off of the
WriteExecutorService
.
Note: When the DataService
is accessed via RMI the Future
MUST be a proxy. This gets handled by the concrete server implementation.
submit
in interface IDataService
tx
- The transaction identifier, ITx.UNISOLATED
for an ACID
operation NOT isolated by a transaction,
ITx.READ_COMMITTED
for a read-committed operation not
protected by a transaction (no global read lock), or any valid
commit time for a read-historical operation not protected by a
transaction (no global read lock).name
- The name of the index partition.proc
- The procedure to be executed.Future
from which the outcome of the procedure may be
obtained.public Future<? extends Object> submit(Callable<? extends Object> task)
DataService
is accessed via RMI the Future
MUST be a proxy. This gets handled by the concrete server implementation.submit
in interface IDataService
submit
in interface IRemoteExecutor
Future
for that task.AbstractDistributedFederation.getProxy(Future)
public ResultSet rangeIterator(long tx, String name, byte[] fromKey, byte[] toKey, int capacity, int flags, IFilter filter) throws InterruptedException, ExecutionException
IDataService
Streaming traversal of keys and/or values in a key range.
Note: In order to visit all keys in a range, clients are expected to
issue repeated calls in which the fromKey is incremented to the
successor of the last key visited until either an empty ResultSet
is returned or the ResultSet#isLast()
flag is set, indicating
that all keys up to (but not including) the startKey have been
visited. See ClientIndexView
(scale-out indices) and
DataServiceTupleIterator
(unpartitioned indices), both of which
encapsulate this method.
Note: If the iterator can be determined to be read-only and it is
submitted as ITx.UNISOLATED
then it will be run as
ITx.READ_COMMITTED
to improve concurrency.
rangeIterator
in interface IDataService
tx
- The transaction identifier -or- ITx.UNISOLATED
IFF the
operation is NOT isolated by a transaction -or-
- tx
to read from the most recent commit point
not later than the absolute value of tx (a fully
isolated read-only transaction using a historical start time).name
- The index name (required).fromKey
- The starting key for the scan (or null
iff
there is no lower bound).toKey
- The first key that will not be visited (or null
iff there is no upper bound).capacity
- When non-zero, this is the maximum #of entries to process.flags
- One or more flags formed by bitwise OR of zero or more of the
constants defined by IRangeQuery
.filter
- An optional object that may be used to layer additional
semantics onto the iterator. The filter will be constructed on
the server and in the execution context for the iterator, so
it will execute directly against the index for the maximum
efficiency.InterruptedException
- if the operation was interrupted.ExecutionException
- If the operation caused an error. See
Throwable.getCause()
for the underlying
error.public IBlock readBlock(IResourceMetadata resource, long addr)
IDataService
IRawStore
described by
the IResourceMetadata
.readBlock
in interface IDataService
resource
- The description of the resource containing that block.addr
- The address of the block in that resource.public void forceOverflow(boolean immediate, boolean compactingMerge) throws IOException, InterruptedException, ExecutionException
IDataService
Normally there is no reason to invoke this method directly. Overflow
processing is triggered automatically on a bottom-up basis when the
extent of the live journal nears the Options.MAXIMUM_EXTENT
.
forceOverflow
in interface IDataService
immediate
- The purpose of this argument is to permit the caller to
trigger an overflow event even though there are no writes
being made against the data service. When true
the method will write a token record on the live journal in
order to provoke a group commit. In this case synchronous
overflow processing will have occurred by the time the method
returns. When false
a flag is set and overflow
processing will occur on the next commit.compactingMerge
- The purpose of this flag is to permit the caller to indicate
that a compacting merge should be performed for all indices on
the data service (at least, all indices whose data are not
simply copied onto the new journal) during the next
synchronous overflow. Note that compacting merges of indices
are performed automatically from time to time so this flag
exists mainly for people who want to force a compacting merge
for some reason.IOException
InterruptedException
- may be thrown if immediate is true
.ExecutionException
- may be thrown if immediate is true
.public boolean purgeOldResources(long timeout, boolean truncateJournal) throws InterruptedException
IDataService
ITx.UNISOLATED
writes and then purges any resources that are no longer required based on
the StoreManager.Options#MIN_RELEASE_AGE
.
Note: Resources are normally purged during synchronous overflow handling. However, asynchronous overflow handling can cause resources to no longer be needed as new index partition views are defined. This method MAY be used to trigger a release before the next overflow event.
purgeOldResources
in interface IDataService
timeout
- The timeout (in milliseconds) that the method will await the
pause of the write service.truncateJournal
- When true
, the live journal will be truncated
to its minimum extent (all writes will be preserved but there
will be no free space left in the journal). This may be used
to force the DataService
to its minimum possible
footprint for the configured history retention policy.true
if successful and false
if the
write service could not be paused after the specified timeout.InterruptedException
public long getAsynchronousOverflowCounter() throws IOException
IDataService
getAsynchronousOverflowCounter
in interface IDataService
IOException
public boolean isOverflowActive() throws IOException
IDataService
true
iff the data service is currently engaged in
overflow processing.isOverflowActive
in interface IDataService
IOException
Copyright © 2006–2019 SYSTAP, LLC DBA Blazegraph. All rights reserved.