com.bigdata.journal

Class ConcurrencyManager

java.lang.Object

com.bigdata.journal.ConcurrencyManager

All Implemented Interfaces:

ICounterSetAccess, IConcurrencyManager, IServiceShutdown

public class ConcurrencyManager
extends Object
implements IConcurrencyManager

Supports concurrent operations against named indices. Historical read and read-committed tasks run with full concurrency. For unisolated tasks, the ConcurrencyManager uses a NonBlockingLockManager to identify a schedule of operations such that access to an unisolated named index is always single threaded while access to distinct unisolated named indices MAY be concurrent.

There are several thread pools that facilitate concurrency. They are:

readService

Concurrent historical and read-committed tasks are run against a historical view of a named index using this service. No locking is imposed. Concurrency is limited by the size of the thread pool.

writeService

Concurrent unisolated writers running against the current view of (or more more) named index(s) (the "live" or "mutable" index(s)). The underlying BTree is NOT thread-safe for writers. Therefore writers MUST predeclare their locks, which allows us to avoid deadlocks altogether. This is also used to schedule the commit phrase of transactions (transaction commits are in fact unisolated tasks).

txWriteService

This is used for the "active" phrase of transaction. Transactions read from historical states of named indices during their active phase and buffer the results on isolated indices backed by a per-transaction TemporaryStore. Since transactions never write on the unisolated indices during their "active" phase, distinct transactions may be run with arbitrary concurrency. However, concurrent tasks for the same transaction must obtain an exclusive lock on the isolated index(s) that are used to buffer their writes.

A transaction that requests a commit using the ITransactionManagerService results in a unisolated task being submitted to the writeService. Transactions are selected to commit once they have acquired a lock on the corresponding unisolated indices, thereby enforcing serialization of their write sets both among other transactions and among unisolated writers. The commit itself consists of the standard validation and merge phrases.

Author:

Bryan Thompson

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	ConcurrencyManager.IConcurrencyManagerCounters Interface defines and documents the counters and counter namespaces for the ConcurrencyManager.
static interface	ConcurrencyManager.Options Options for the ConcurrentManager.

Field Summary

Fields

Modifier and Type	Field and Description
protected TaskCounters	countersHR Counters for the readService.
protected TaskCounters	countersTX Counters for the txWriteService.
protected WriteTaskCounters	countersUN Counters for writeService.

Constructor Summary

Constructors

Constructor and Description
ConcurrencyManager(Properties properties, ILocalTransactionManager transactionManager, IResourceManager resourceManager) (Re-)open a journal supporting concurrent operations.

Method Summary

Methods

Modifier and Type	Method and Description
CounterSet	getCounters() Return the CounterSet.
double	getJournalOverextended() Return the overextension multiplier for the journal.
IResourceManager	getResourceManager() The object used to manage local resources.
ILocalTransactionManager	getTransactionManager() The client side of the transaction manager.
WriteExecutorService	getWriteService() The service on which read-write tasks are executed.
<T> List<Future<T>>	invokeAll(Collection<? extends AbstractTask<T>> tasks) Executes the given tasks, returning a list of Futures holding their status and results when all complete.
<T> List<Future<T>>	invokeAll(Collection<? extends AbstractTask<T>> tasks, long timeout, TimeUnit unit) Executes the given tasks, returning a list of Futures holding their status and results when all complete or the timeout expires, whichever happens first.
boolean	isOpen() Return true iff the service is running.
void	shutdown() Shutdown the thread pools (running tasks will run to completion, but no new tasks will start).
void	shutdownNow() Immediate shutdown (running tasks are canceled rather than being permitted to complete).
<T> FutureTask<T>	submit(AbstractTask<T> task) Submit a task (asynchronous).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

countersUN

protected final WriteTaskCounters countersUN

Counters for writeService.

countersTX

protected final TaskCounters countersTX

Counters for the txWriteService.

countersHR

protected final TaskCounters countersHR

Counters for the readService.

Constructor Detail

ConcurrencyManager

public ConcurrencyManager(Properties properties,
                  ILocalTransactionManager transactionManager,
                  IResourceManager resourceManager)

(Re-)open a journal supporting concurrent operations.

Parameters:

properties - See ConcurrencyManager.Options.

transactionManager - The object managing the local transactions.

resourceManager - The object managing the resources on which the indices are stored.

Method Detail

getWriteService

public WriteExecutorService getWriteService()

Description copied from interface: IConcurrencyManager

The service on which read-write tasks are executed.

Specified by:

getWriteService in interface IConcurrencyManager

getTransactionManager

public ILocalTransactionManager getTransactionManager()

Description copied from interface: IConcurrencyManager

The client side of the transaction manager.

Specified by:

getTransactionManager in interface IConcurrencyManager

getResourceManager

public IResourceManager getResourceManager()

Description copied from interface: IConcurrencyManager

The object used to manage local resources.

Specified by:

getResourceManager in interface IConcurrencyManager

isOpen

public boolean isOpen()

Description copied from interface: IServiceShutdown

Return true iff the service is running.

Specified by:

isOpen in interface IServiceShutdown

shutdown

public void shutdown()

Shutdown the thread pools (running tasks will run to completion, but no new tasks will start).

Specified by:

shutdown in interface IConcurrencyManager

Specified by:

shutdown in interface IServiceShutdown

See Also:

IConcurrencyManager.shutdownNow()

shutdownNow

public void shutdownNow()

Immediate shutdown (running tasks are canceled rather than being permitted to complete).

Specified by:

shutdownNow in interface IConcurrencyManager

Specified by:

shutdownNow in interface IServiceShutdown

See Also:

shutdown()

getCounters

public CounterSet getCounters()

Return the CounterSet.

Specified by:

getCounters in interface ICounterSetAccess

submit

public <T> FutureTask<T> submit(AbstractTask<T> task)

Submit a task (asynchronous). Tasks will execute asynchronously in the appropriate thread pool with as much concurrency as possible.

Note: Unisolated write tasks will NOT return before the next group commit (exceptions may be thrown if the task fails or the commit fails). The purpose of group commits is to provide higher throughput for writes on the store by only syncing the data to disk periodically rather than after every write. Group commits are scheduled by the #commitService. The trigger conditions for group commits may be configured using ConcurrencyManager.Options. If you are using the store in a single threaded context then you may set ConcurrencyManager.Options.WRITE_SERVICE_CORE_POOL_SIZE to ONE (1) which has the effect of triggering commit immediately after each unisolated write. However, note that you can not sync a disk more than ~ 30-40 times per second so your throughput in write operations per second will never exceed that for a single-threaded application writing on a hard disk. (Your mileage can vary if you are writing on a transient store or using a durable medium other than disk).

Note: The isolated indices used by a read-write transaction are NOT thread-safe. Therefore a partial order is imposed over concurrent tasks for the same transaction that seek to read or write on the same index(s). Full concurrency is allowed when different transactions access the same index(s), but write-write conflicts MAY be detected during commit processing.

Note: The following exceptions MAY be wrapped by Future.get() for tasks submitted via this method:

ValidationError

An unisolated write task was attempting to commit the write set for a transaction but validation failed. You may retry the entire transaction.

InterruptedException

A task was interrupted during execution and before the task had completed normally. You MAY retry the task, but note that this exception is also generated when tasks are cancelled when the journal is being shutdown() after the timeout has expired or shutdownNow(). In either of these cases the task will not be accepted by the journal.

Specified by:

submit in interface IConcurrencyManager

Parameters:

task - The task.

Returns:

The Future that may be used to resolve the outcome of the task.

Throws:

RejectedExecutionException - if task cannot be scheduled for execution (typically the queue has a limited capacity and is full)

NullPointerException - if task null

getJournalOverextended

public double getJournalOverextended()

Return the overextension multiplier for the journal. This is the ratio of the bytes written on the journal against its nominal maximum extent. For example, an overextension of two means that the journal has reached twice is nominal maximum extent. This is zero unless we are running in a distributed federation.

Returns:

The overextension multipler.

invokeAll

public <T> List<Future<T>> invokeAll(Collection<? extends AbstractTask<T>> tasks)
                          throws InterruptedException

Executes the given tasks, returning a list of Futures holding their status and results when all complete. Note that a completed task could have terminated either normally or by throwing an exception. The results of this method are undefined if the given collection is modified while this operation is in progress.

Note: Contract is per ExecutorService.invokeAll(Collection)

Specified by:

invokeAll in interface IConcurrencyManager

Parameters:

tasks - The tasks.

Returns:

Their Futures.

Throws:

InterruptedException - if interrupted while waiting, in which case unfinished tasks are canceled.

NullPointerException - if tasks or any of its elements are null

RejectedExecutionException - if any task cannot be scheduled for execution

invokeAll

public <T> List<Future<T>> invokeAll(Collection<? extends AbstractTask<T>> tasks,
                            long timeout,
                            TimeUnit unit)
                          throws InterruptedException

Executes the given tasks, returning a list of Futures holding their status and results when all complete or the timeout expires, whichever happens first. Note that a completed task could have terminated either normally or by throwing an exception. The results of this method are undefined if the given collection is modified while this operation is in progress.

Note: Contract is based on ExecutorService.invokeAll(Collection, long, TimeUnit) but only the Futures of the submitted tasks are returned.

Specified by:

invokeAll in interface IConcurrencyManager

Parameters:

tasks - The tasks.

Returns:

The Futures of all tasks that were submitted prior to the expiration of the timeout.

Throws:

InterruptedException - if interrupted while waiting, in which case unfinished tasks are canceled.

NullPointerException - if tasks or any of its elements are null

RejectedExecutionException - if any task cannot be scheduled for execution