public abstract class AbstractScaleOutClientIndexView extends Object implements IScaleOutClientIndex
IScaleOutClientIndex implementation(s).| Modifier and Type | Field and Description |
|---|---|
protected boolean |
batchOnly
This may be used to disable the non-batch API, which is quite convenient
for locating code that needs to be re-written to use
IIndexProcedures. |
protected static String |
ERR_ABORT_TX
Error message used if we were unable to abort a transaction that we
started in order to provide read-consistent semantics for an
ITx.READ_COMMITTED view or for a read-only operation on an
ITx.UNISOLATED view. |
protected static String |
ERR_NEW_TX
Error message used if we were unable to start a new transaction in order
to provide read-consistent semantics for an
ITx.READ_COMMITTED
view or for a read-only operation on an ITx.UNISOLATED view. |
protected AbstractScaleOutFederation |
fed |
protected static org.apache.log4j.Logger |
log
Note: Invocations of the non-batch API are logged at the WARN level since
they result in an application that can not scale-out efficiently.
|
protected String |
name
The name of the scale-out index (from the ctor).
|
protected static String |
NON_BATCH_API |
protected boolean |
readConsistent |
protected long |
taskTimeout
The timeout in milliseconds for tasks run on an
IDataService. |
protected long |
timestamp
The timestamp from the ctor.
|
protected boolean |
WARN
True iff the
log level is WARN or less. |
| Constructor and Description |
|---|
AbstractScaleOutClientIndexView(AbstractScaleOutFederation fed,
String name,
long timestamp,
IMetadataIndex metadataIndex)
Create a view on a scale-out index.
|
| Modifier and Type | Method and Description |
|---|---|
boolean |
contains(byte[] key)
Return
true iff there is a (non-deleted) index entry for
the key. |
boolean |
contains(Object key)
Return true iff there is an entry for the key.
|
ICounter |
getCounter()
Counters are local to a specific index partition and are only available
to unisolated procedures running inside of an
IConcurrencyManager
(which includes procedures run on an IDataService). |
CounterSet |
getCounters()
Return a new
CounterSet backed by the ScaleOutIndexCounters
for this scale-out index. |
IDataService |
getDataService(PartitionLocator pmd)
Resolve the data service to which the index partition is mapped.
|
AbstractScaleOutFederation |
getFederation()
Return the object used to access the services in the connected
federation.
|
IndexMetadata |
getIndexMetadata()
The metadata for the managed scale-out index.
|
protected IMetadataIndex |
getMetadataIndex()
Return a view of the metadata index for the scale-out index as of the
timestamp associated with this index view.
|
MetadataIndex.MetadataIndexMetadata |
getMetadataIndexMetadata()
Metadata for the
MetadataIndex that manages the scale-out index
(cached). |
protected IMetadataService |
getMetadataService()
Obtain the proxy for a metadata service.
|
String |
getName()
The name of the scale-out index.
|
IResourceMetadata[] |
getResourceMetadata()
This operation is not supported - the resource description of a scale-out
index would include all "live" resources in the corresponding
MetadataIndex. |
protected ThreadPoolExecutor |
getThreadPool()
The thread pool exposed by
IBigdataFederation.getExecutorService() |
long |
getTimestamp()
Either the startTime of an active transaction,
ITx.UNISOLATED for
the current unisolated index view, ITx.READ_COMMITTED for a
read-committed view, or the timestamp for a historical
view no later than the specified timestamp. |
protected ITupleSerializer |
getTupleSerializer() |
byte[] |
insert(byte[] key,
byte[] value)
Insert or update a value under the key.
|
Object |
insert(Object key,
Object val)
Insert with auto-magic handling of keys and value objects.
|
Iterator<PartitionLocator> |
locatorScan(long ts,
byte[] fromKey,
byte[] toKey,
boolean reverseScan)
Returns an iterator that will visit the
PartitionLocators for
the specified scale-out index key range. |
byte[] |
lookup(byte[] key)
Lookup a value for a key.
|
Object |
lookup(Object key)
Lookup a value for a key.
|
<T extends IKeyArrayIndexProcedure,O,R,A> |
newWriteBuffer(IResultHandler<R,A> resultHandler,
IDuplicateRemover<O> duplicateRemover,
AbstractKeyArrayIndexProcedureConstructor<T> ctor)
Asynchronous write API (streaming writes).
|
byte[] |
putIfAbsent(byte[] key,
byte[] value)
Insert or update a value under the key iff there is no entry for that key
in the index.
|
long |
rangeCount()
Return the #of tuples in the index.
|
long |
rangeCount(byte[] fromKey,
byte[] toKey)
Returns the sum of the range count for each index partition spanned by
the key range.
|
long |
rangeCountExact(byte[] fromKey,
byte[] toKey)
The exact range count is obtained by mapping a key-range scan over the
index partitions.
|
long |
rangeCountExactWithDeleted(byte[] fromKey,
byte[] toKey)
The exact range count of deleted and undeleted tuples is obtained by
mapping a key-range scan over the index partitions.
|
ITupleIterator |
rangeIterator()
Visits all tuples in key order.
|
ITupleIterator |
rangeIterator(byte[] fromKey,
byte[] toKey)
An
ITupleIterator that kinds the use of a series of
ResultSets to cover all index partitions spanned by the key
range. |
ITupleIterator |
rangeIterator(byte[] fromKey,
byte[] toKey,
int capacity,
int flags,
IFilter filter)
Identifies the index partition(s) that are spanned by the key range query
and maps an iterator across each index partition.
|
byte[] |
remove(byte[] key)
Remove the key and its associated value.
|
Object |
remove(Object key)
Remove the key and its associated value.
|
LinkedList<Split> |
splitKeys(long ts,
int fromIndex,
int toIndex,
byte[][] keys)
Utility method to split a set of ordered keys into partitions based the
index partitions defined for a scale-out index.
|
LinkedList<Split> |
splitKeys(long ts,
int fromIndex,
int toIndex,
KVO[] a)
|
void |
staleLocator(long ts,
PartitionLocator locator,
StaleLocatorException cause)
Notifies the client that a
StaleLocatorException was received. |
void |
submit(byte[] fromKey,
byte[] toKey,
IKeyRangeIndexProcedure proc,
IResultHandler resultHandler)
Maps an
IIndexProcedure across a key range by breaking it down
into one task per index partition spanned by that key range. |
Object |
submit(byte[] key,
ISimpleIndexProcedure proc)
Submits an index procedure that operations on a single key to the
appropriate index partition returning the result of that procedure.
|
void |
submit(int fromIndex,
int toIndex,
byte[][] keys,
byte[][] vals,
AbstractKeyArrayIndexProcedureConstructor ctor,
IResultHandler aggregator)
The procedure will be transparently broken down and executed against each
index partitions spanned by its keys.
|
protected abstract void |
submit(long ts,
byte[] fromKey,
byte[] toKey,
IKeyRangeIndexProcedure proc,
IResultHandler resultHandler)
Variant uses the caller's timestamp.
|
protected abstract Object |
submit(long ts,
byte[] key,
ISimpleIndexProcedure proc)
Variant uses the caller's timestamp.
|
protected abstract void |
submit(long ts,
int fromIndex,
int toIndex,
byte[][] keys,
byte[][] vals,
AbstractKeyArrayIndexProcedureConstructor ctor,
IResultHandler aggregator)
Variant uses the caller's timestamp.
|
String |
toString() |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, waitgetRecursionDepthprotected static final transient org.apache.log4j.Logger log
protected final boolean WARN
log level is WARN or less.protected static final transient String ERR_NEW_TX
ITx.READ_COMMITTED
view or for a read-only operation on an ITx.UNISOLATED view.protected static final transient String ERR_ABORT_TX
ITx.READ_COMMITTED view or for a read-only operation on an
ITx.UNISOLATED view.protected final AbstractScaleOutFederation fed
protected final long taskTimeout
IDataService.protected static final String NON_BATCH_API
protected final boolean batchOnly
IIndexProcedures.protected final long timestamp
protected final String name
protected final boolean readConsistent
IBigdataClient.isReadConsistent()public AbstractScaleOutClientIndexView(AbstractScaleOutFederation fed, String name, long timestamp, IMetadataIndex metadataIndex)
fed - The federation containing the index.name - The index name.timestamp - A transaction identifier, ITx.UNISOLATED for the
unisolated index view, ITx.READ_COMMITTED, or
timestamp for a historical view no later than
the specified timestamp.metadataIndex - The IMetadataIndex for the named scale-out index as of
that timestamp. Note that the IndexMetadata on this
object contains the template IndexMetadata for the
scale-out index partitions.public AbstractScaleOutFederation getFederation()
IScaleOutClientIndexgetFederation in interface IScaleOutClientIndexprotected ThreadPoolExecutor getThreadPool()
IBigdataFederation.getExecutorService()public final long getTimestamp()
IClientIndexITx.UNISOLATED for
the current unisolated index view, ITx.READ_COMMITTED for a
read-committed view, or the timestamp for a historical
view no later than the specified timestamp.getTimestamp in interface IClientIndexpublic final String getName()
IClientIndexgetName in interface IClientIndexprotected final IMetadataService getMetadataService()
protected final IMetadataIndex getMetadataIndex()
public MetadataIndex.MetadataIndexMetadata getMetadataIndexMetadata()
MetadataIndex that manages the scale-out index
(cached).public IndexMetadata getIndexMetadata()
IKeyArrayIndexProcedures when we serialize a procedure to be
sent to a remote IDataService.getIndexMetadata in interface IIndexICheckpointProtocol.getIndexMetadata()public IDataService getDataService(PartitionLocator pmd)
IScaleOutClientIndexgetDataService in interface IScaleOutClientIndexpmd - The index partition locator.null.public Iterator<PartitionLocator> locatorScan(long ts, byte[] fromKey, byte[] toKey, boolean reverseScan)
IScaleOutClientIndexPartitionLocators for
the specified scale-out index key range.locatorScan in interface IScaleOutClientIndexts - The timestamp that will be used to visit the locators.fromKey - The scale-out index first key that will be visited
(inclusive). When null there is no lower bound.toKey - The first scale-out index key that will NOT be visited
(exclusive). When null there is no upper bound.reverseScan - true if you need to visit the index partitions
in reverse key order (this is done when the partitioned
iterator is scanning backwards).ITuple.getValue()
will be a serialized PartitionLocator object.AbstractScaleOutFederation.locatorScan(String, long, byte[], byte[],
boolean)public IResourceMetadata[] getResourceMetadata()
MetadataIndex.getResourceMetadata in interface IIndexpublic ICounter getCounter()
IClientIndexIConcurrencyManager
(which includes procedures run on an IDataService).getCounter in interface IIndexLocalCountergetCounter in interface IClientIndexprotected ITupleSerializer getTupleSerializer()
public boolean contains(Object key)
IAutoboxBTreecontains in interface IAutoboxBTreekey - The key is implicitly converted to an unsigned
byte[].public boolean contains(byte[] key)
ISimpleBTreetrue iff there is a (non-deleted) index entry for
the key. An index entry with a null value will cause this
method to return true. A deleted index entry will cause
this method to return false.contains in interface ISimpleBTreekey - The key.true if the index contains an (un-deleted) entry
for that key.public Object insert(Object key, Object val)
IAutoboxBTreeinsert in interface IAutoboxBTreekey - The key is implicitly converted to an unsigned
byte[].val - The value is implicitly converted to a byte[].null if there was
no value stored under that key.public byte[] insert(byte[] key,
byte[] value)
ISimpleBTreeinsert in interface ISimpleBTreekey - The key.value - The value (may be null).null if the
key was not found or if the previous entry for that key was
marked as deleted.public byte[] putIfAbsent(byte[] key,
byte[] value)
ISimpleBTree
if (!contains(key))
insert(key, value);
However, if the index allows null values to be stored under
a key and the application in fact stores null values for
some tuples, then caller is not able to decide using this method whether
or not the mutation was applied based on the return value. For these
cases if the caller needs to know whether or not the conditional mutation
actually took place, the caller CAN use the pattern
if(!contains()) insert(key,value); to obtain that
information.putIfAbsent in interface ISimpleBTreekey - The key.value - The value (may be null).null if the key
was not found or if the previous entry for that key was marked as
deleted. Note that the return value MAY be null even
if there was an entry under the key. This is because the index is
capable of storing a null value. In such cases the
conditional mutation WAS NOT applied.(putIfAbsent)public Object lookup(Object key)
IAutoboxBTreelookup in interface IAutoboxBTreekey - The key is implicitly converted to an unsigned
byte[].null if there is no
entry for that key.public byte[] lookup(byte[] key)
ISimpleBTreelookup in interface ISimpleBTreenull if there
is no entry for that key or if the entry under that key is marked
as deleted.public Object remove(Object key)
IAutoboxBTreeremove in interface IAutoboxBTreekey - The key is implicitly converted to an unsigned
byte[].null if the key was not found.public byte[] remove(byte[] key)
ISimpleBTreeremove in interface ISimpleBTreekey - The key.null if the key
was not found or if the previous entry under that key was marked
as deleted.public long rangeCount()
IRangeQueryNote: If the index supports deletion markers then the range count will be an upper bound and may double count tuples which have been overwritten, including the special case where the overwrite is a delete.
rangeCount in interface IRangeQueryISimpleIndexAccess.rangeCount()public long rangeCount(byte[] fromKey,
byte[] toKey)
rangeCount in interface IRangeQueryfromKey - The lowest key that will be counted (inclusive). When
null there is no lower bound.toKey - The first key that will not be counted (exclusive). When
null there is no upper bound.public final long rangeCountExact(byte[] fromKey,
byte[] toKey)
rangeCountExact in interface IRangeQueryfromKey - The lowest key that will be counted (inclusive). When
null there is no lower bound.toKey - The first key that will not be counted (exclusive). When
null there is no upper bound.public final long rangeCountExactWithDeleted(byte[] fromKey,
byte[] toKey)
rangeCountExactWithDeleted in interface IRangeQueryfromKey - The lowest key that will be counted (inclusive). When
null there is no lower bound.toKey - The first key that will not be counted (exclusive). When
null there is no upper bound.IRangeQuery.rangeCountExact(byte[], byte[])public final ITupleIterator rangeIterator()
IRangeQueryrangeIterator(null, null)
rangeIterator in interface IRangeQuerypublic ITupleIterator rangeIterator(byte[] fromKey, byte[] toKey)
ITupleIterator that kinds the use of a series of
ResultSets to cover all index partitions spanned by the key
range.rangeIterator in interface IRangeQueryfromKey - The first key that will be visited (inclusive lower bound).
When null there is no lower bound.toKey - The first key that will NOT be visited (exclusive upper
bound). When null there is no upper bound.SuccessorUtil, which may be used to compute the successor of a value
before encoding it as a component of a key.,
BytesUtil#successor(byte[]), which may be used to compute the
successor of an encoded key.,
EntryFilter, which may be used to filter the entries visited by the
iterator.public ITupleIterator rangeIterator(byte[] fromKey, byte[] toKey, int capacity, int flags, IFilter filter)
rangeIterator in interface IRangeQueryfromKey - The first key that will be visited (inclusive lower bound).
When null there is no lower bound.toKey - The first key that will NOT be visited (exclusive upper
bound). When null there is no upper bound.capacity - The #of entries to buffer at a time. This is a hint and MAY be
zero (0) to use an implementation specific default
capacity. A non-zero value may be used if you know that you
want at most N results or if you want to override the default
#of results to be buffered before sending them across a
network interface. (Note that you can control the default
value using
IBigdataClient.Options#DEFAULT_CLIENT_RANGE_QUERY_CAPACITY).flags - A bitwise OR of IRangeQuery.KEYS, IRangeQuery.VALS, etc.filter - An optional object used to construct a stacked iterator. When
IRangeQuery.CURSOR is specified in flags, the base
iterator will implement ITupleCursor and the first
filter in the stack can safely cast the source iterator to an
ITupleCursor. If the outermost filter in the stack
does not implement ITupleIterator, then it will be
wrapped an ITupleIterator.SuccessorUtil, which may be used to compute the successor of a value
before encoding it as a component of a key.,
BytesUtil#successor(byte[]), which may be used to compute the
successor of an encoded key.,
IFilterConstructor, which may be used to construct an iterator stack
performing filtering or other operations.public LinkedList<Split> splitKeys(long ts, int fromIndex, int toIndex, byte[][] keys)
Find the partition for the first key. Check the last key, if it is in the same partition then then this is the simplest case and we can just send the data along.
Otherwise, perform a binary search on the remaining keys looking for the index of the first key GTE the right separator key for that partition. The batch for this partition is formed from all keys from the first key for that partition up to but excluding the index position identified by the binary search (if there is a match; if there is a miss, then the binary search result needs to be converted into a key index and that will be the last key for the current partition).
Examine the next key and repeat the process until all keys have been allocated to index partitions.
Note: Split points MUST respect the "row" identity for a sparse row store, but we get that constraint by maintaining the index partition boundaries in agreement with the split point constraints for the index.
splitKeys in interface ISplitterts - The timestamp for the IMetadataIndex view that will be
applied to choose the Splits.fromIndex - The index of the first key in keys to be processed
(inclusive).toIndex - The index of the last key in keys to be processed.keys - An array of keys. Each key is an interpreted as an unsigned
byte[]. All keys must be non-null. The keys must be in sorted
order.Splits that you can use to form requests based on
the identified first/last key and partition identified by this
process.Arrays.sort(Object[], int, int, java.util.Comparator),
BytesUtil.compareBytes(byte[], byte[])public LinkedList<Split> splitKeys(long ts, int fromIndex, int toIndex, KVO[] a)
ISplitterSplits for an ordered KVO[] such that there
is one Split per index partition spanned by the data.splitKeys in interface ISplitterts - The timestamp for the IMetadataIndex view that will be
applied to choose the Splits.fromIndex - The index of the first key in keys to be processed
(inclusive).toIndex - The index of the last key in keys to be processed.Splits that you can use to form requests based on the
identified first/last key and partition identified by this
process.public void staleLocator(long ts,
PartitionLocator locator,
StaleLocatorException cause)
IScaleOutClientIndexStaleLocatorException was received.
The client will use this information to refresh the
IMetadataIndex.staleLocator in interface IScaleOutClientIndexts - The timestamp of the metadata index view from which the
locator was obtained.locator - The locator that was stale.cause - The reason why the locator became stale (split, join, or
move).public Object submit(byte[] key, ISimpleIndexProcedure proc)
IIndexsubmit in interface IIndexkey - The key.proc - The procedure.IIndexProcedure.apply(IIndex)public void submit(byte[] fromKey,
byte[] toKey,
IKeyRangeIndexProcedure proc,
IResultHandler resultHandler)
IIndexProcedure across a key range by breaking it down
into one task per index partition spanned by that key range.
Note: In order to avoid growing the task execution queue without bound,
an upper bound of IBigdataClient.Options.CLIENT_MAX_PARALLEL_TASKS_PER_REQUEST
tasks will be placed onto the queue at a time. More tasks will be
submitted once those tasks finish until all tasks have been executed.
When the task is not parallelizable the tasks will be submitted to the
corresponding index partitions at a time and in key order.
submit in interface IIndexfromKey - The lower bound (inclusive) -or- null if there
is no lower bound.toKey - The upper bound (exclusive) -or- null if there
is no upper bound.proc - The procedure. If the procedure implements the
IParallelizableIndexProcedure marker interface then it
MAY be executed in parallel against the relevant index
partition(s).public void submit(int fromIndex,
int toIndex,
byte[][] keys,
byte[][] vals,
AbstractKeyArrayIndexProcedureConstructor ctor,
IResultHandler aggregator)
IParallelizableIndexProcedure then the procedure
will be mapped in parallel against the relevant index partitions.
Note: Unlike mapping an index procedure across a key range, this method
is unable to introduce a truely enourmous burden on the client's task
queue since the #of tasks arising is equal to the #of splits and bounded
by n := toIndex - fromIndex.
submit in interface IIndexfromIndex - The index of the first key to be used (inclusive).toIndex - The index of the last key to be used (exclusive).keys - The keys (required).vals - The values (optional depending on the procedure).ctor - An object that can create instances of the procedure.aggregator - When defined, results from each procedure application will be
reported to this object.
TODO In order to allow parallelization within a shard, we need to modify
this method signature to pass in an IResultHandler constructor
object. That might be something which could be pushed down onto the ctor
argument. It would be used in scale-out to create a DS local result handler
so we can locally aggregate when parallelizing against each shard and then
return that aggregated result to the client which would extract the aggregate
result across the shards from the client's result handler. See BLZG-1537.(Schedule more IOs when loading data)protected abstract Object submit(long ts, byte[] key, ISimpleIndexProcedure proc)
ts - key - proc - protected abstract void submit(long ts,
byte[] fromKey,
byte[] toKey,
IKeyRangeIndexProcedure proc,
IResultHandler resultHandler)
ts - fromKey - toKey - proc - resultHandler - protected abstract void submit(long ts,
int fromIndex,
int toIndex,
byte[][] keys,
byte[][] vals,
AbstractKeyArrayIndexProcedureConstructor ctor,
IResultHandler aggregator)
ts - fromIndex - toIndex - keys - vals - ctor - aggregator - public <T extends IKeyArrayIndexProcedure,O,R,A> BlockingBuffer<KVO<O>[]> newWriteBuffer(IResultHandler<R,A> resultHandler, IDuplicateRemover<O> duplicateRemover, AbstractKeyArrayIndexProcedureConstructor<T> ctor)
IAsynchronousWriteBufferFactory
The returned buffer provides a streaming API which is highly efficient.
The caller writes ordered KVO[] chunks onto the thread-safe
BlockingBuffer. Those chunks are dynamically combined and then
split into per-index partition chunks which are written on internally
managed BlockingBuffers for each index partition which will be
touched by a write operation. The splits are slices of ordered chunks for
a specific index partition. The BlockingBuffer uses a merge sort
when it combines ordered chunks so that the combined chunks remain fully
ordered. Once a chunk is ready, it is re-shaped for the CTOR and sent to
the target data service using RMI.
Since this API is asynchronous, you will not have synchronous access to
values returned by asynchronous writes. However, patterns can be created
using KVOC and KVOLatch which provide notification when
application defined sets of results have become available. Such patterns
are created by associated the KVOLatch with the set of results
and using IResultHandler and the object reference on the
KVOC to capture the side-effect of the write.
BlockingBuffer.getFuture() may be used to obtain the
Future of the consumer. You can use Future.get() to await
the completion of the consumer, to cancel the consumer, etc. The
Future will not terminate (other than by error) until the buffer
has been closed. The Future
evaluates to an IndexAsyncWriteStats object. Those statistics are
also reported to the ILoadBalancerService via the
IBigdataFederation.
Each buffer returned by this method is independent, and writes onto
independent sinks which write through to the index partitions. This is
necessary in order for the caller to retain control over the life cycle
of their write operations. The BlockingBuffer is thread-safe so
it may be the target for concurrent producers can be can utilized to
create very high throughput designs. While the returned buffers are
independent, the performance counters for all asynchronous write buffers
for a given client and scale-out index are aggregated by a single
ScaleOutIndexCounters instance.
newWriteBuffer in interface IAsynchronousWriteBufferFactoryT - The generic type of the procedure used to write on the index.O - The generic type for unserialized value objects.R - The type of the result from applying the index procedure to a
single Split of data.A - The type of the aggregated result.resultHandler - Used to aggregate results.duplicateRemover - Used to filter out duplicates in an application specified
manner (optional).ctor - Used to create instances of the procedure that will execute a
write on an individual index partition (this implies that
insert and remove operations as well as custom index write
operations must use separate buffers).IndexMetadata.getAsynchronousIndexWriteConfiguration(),
AbstractFederation.getIndexCounters(String)public CounterSet getCounters()
CounterSet backed by the ScaleOutIndexCounters
for this scale-out index.getCounters in interface IIndexgetCounters in interface ICounterSetAccessCopyright © 2006–2019 SYSTAP, LLC DBA Blazegraph. All rights reserved.