RWStore (Blazegraph Database Platform 2.1.5 API)

java.lang.Object
- com.bigdata.rwstore.RWStore

All Implemented Interfaces:

IBackingReader, IBufferedWriter, IStreamStore, IAllocationManager, IHistoryManager, IStore
```
public class RWStore
extends Object
implements IStore, IBufferedWriter, IBackingReader
```
Storage class
Provides an interface to allocating storage within a disk file.
Essentially provides a DiskMalloc interface.
In addition to the DiskMalloc/ReAlloc mechanism, a single root address can be associated. This can be used when opening an existing storage file to retrieve some management object - such as an object manager!
The allocator also support atomic update via a simple transaction mechanism.
Updates are normally committed immediately, but by using startTransaction and commitTransaction, the previous state of the store is retained until the moment of commitment.
It would also be possible to add some journaling/version mechanism, where snapshots of the allocation maps are retained for sometime. For a store which was only added to this would not be an unreasonable overhead and would support the rolling back of the database weekly or monthly if required.
The input/output mechanism uses ByteArray Input and Output Streams.
One difference between the disk realloc and in memory realloc is that the disk realloc will always return a new address and mark the old address as ready to be freed.
The method of storing the allocation headers has been changed from always allocating at the end of the file (and moving them on file extend) to allocation of fixed areas. The meta-allocation data, containing the bitmap that controls these allocations, is itself stored in the heap, and is now structured to include both the bit data and the list of meta-storage addresses.
Sizing: 256 allocators would reference approximately 2M objects/allocations. At 1K per allocator this would require 250K of store. The meta-allocation data would therefore need a start address plus 32 bytes (or 8 ints) to represent the meta-allocation bits. An array of such data referencing sequentially allocated storage areas completes the meta-allocation requirements.
A meta-allocation address can therefore be represented as a single bit offset from which the block, providing start address, and bit offset can be directly determined.
The m_metaBits int array used to be fully used as allocation bits, but now stores both the start address plus the 8 ints used to manage that data block.
Allocation is reduced to sets of allocator objects which have a start address and a bitmap of allocated storage maps.
Searching thousands of allocation blocks to find storage is not efficient, but by utilizing roving pointers and sorting blocks with free space available this can be made most efficient.
In order to provide optimum use of bitmaps, this implementation will NOT use the BitSet class.
Using the meta-allocation bits, it is straightforward to load ALL the allocation headers. A total of (say) 100 allocation headers might provide up to 4000 allocations each -> 400 000 objects, while 1000 headers -> 4m objects and 2000 -> 8m objects.
The allocators are split into a set of FixedAllocators and then BlobAllocation. The FixedAllocators will allocate from 128 to 32K objects, with a minimum block allocation of 64K, and a minimum bit number per block of 32.
Where possible lists and roving pointers will be used to minimize searching of the potentially large structures.
Since the memory is allocated on (at least) a 128 byte boundary, there is some leeway on storing the address. Added to the address is the shift required to make to the "standard" 128 byte block, e.g. blocksize = 128 << (addr % 8)
NB Useful method on RandomAccessFile.setLength(newLength)
When session data is preserved two things must happen - the allocators must not reallocate data that has been freed in this session, or more clearly can only free data that has been allocated in this session. That should be it.
The ALLOC_SIZES table is the fibonacci sequence. We multiply by 64 bytes to get actual allocation block sizes. We then allocate bits based on 8K allocation rounding and 32 bits at a time allocation. Note that 4181 * 64 = 267,584 and 256K is 262,144
All data is checksummed, both allocated/saved data and the allocation blocks.
BLOB allocation is not handled using chained data buffers but with a blob header record. This is indicated with a BlobAllocator that provides indexed offsets to the header record (the address encodes the BlobAllocator and the offset to the address). The header record stores the number of component allocations and the address of each.
This approach makes for much more efficient freeing/re-allocation of Blob storage, in particular avoiding the need to read in the component blocks to determine chained blocks for freeing. This is particularly important for larger stores where a disk cache could be flushed through simply freeing BLOB allocations.
Deferred Free List

The previous implementation has been amended to associate a single set of deferredFree blocks with each CommitRecord. The CommitRecordIndex will then provide access to the CommitRecords to support the deferred freeing of allocations based on age/earliestTxReleaseTime.
The last release time processed is held with the MetaAllocation data

Author:

Martyn Cutcher FIXME Release checklist:
Add metabits header record checksum field and verify on read back.
Done. Checksum fixed allocators (needs to be tested on read back).
Done. Add version field to the fixed allocator.
Done. Checksum delete blocks / blob records.
PSOutputStream - remove caching logic. It is unused and makes this class much more complex. A separate per-RWStore caching class for recycling PSOutputStreams can be added later.
Modify FixedAllocator to use arrayCopy() rather than clone and declare more fields to be final. See notes on AllocBlock.
Done. Implement logic to "abort" a shadow allocation context.
Unit test to verify that we do not recycle allocations from the last commit point even when the retention time is zero such that it is always possible to re-open the store from the alternative root block even after you have allocated things against the current root block (but not yet committed).
Read-only mode.
Unit tests looking for persistent memory leaks (e.g., all allocated space can be reclaimed).

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`RWStore.AllocationStats`
`static class`	`RWStore.DeleteBlockStats` Simple class to collect up DeleteBlockStats and returned by checkDeleteBlocks, called from DumpJournal.
`static interface`	`RWStore.Options` Options understood by the `RWStore`.
`static class`	`RWStore.RWStoreState`
`static class`	`RWStore.StoreCounters<T extends RWStore.StoreCounters<T>>` Striped performance counters for `IRawStore` access, including operations that read or write through to the underlying media.

Constructor Summary

Constructors
Constructor and Description
`RWStore(FileMetadata fileMetadata, Quorum<?,?> quorum)` The ALLOC_SIZES must be initialized from either the file or the properties associated with the fileMetadataView

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`abortContext(IAllocationContext context)` The ContextAllocation object manages a freeList of associated allocators and an overall list of allocators.
`long`	`alloc(byte[] buf, int size, IAllocationContext context)` Called by PSOutputStream to make to actual allocation or directly by lower level API clients.
`int`	`alloc(int size, IAllocationContext context)` Core allocation method.
`protected int`	`allocBlock(int size)` Return the address of a contiguous region on the persistent heap.
`int`	`checkDeferredFrees(AbstractJournal journal)` This method is invoked during the commit protocol and gives the backing store an opportunity to check whether storage associated with deferred frees can now be released.
`RWStore.DeleteBlockStats`	`checkDeleteBlocks(AbstractJournal journal)` Utility to check the deleteBlocks associated with each active CommitRecord
`void`	`close()` Close the backing storage.
`void`	`commit()` Global commit on the backing store.
`void`	`computeDigest(Object snapshot, MessageDigest digest)`
`static long`	`convertAddr(int addr)` Convert an implicitly scaled int32 offset into the backing file into an int64 address into the backing file.
`int`	`convertFromAddr(long addr)` Convert an int64 address into the backing file into an int32 offset that is implicitly scaled by `ALLOCATION_SCALEUP`.
`void`	`deferFree(int rwaddr, int sze)` Adds the address for later freeing to the deferred free list.
`void`	`detachContext(IAllocationContext context)` Indicates that the allocation context will no longer be used, but that the allocations made within the context should be preserved.
`boolean`	`ensureMetabitsDemispace(boolean useDemispace)` Forces a reset of the metabits allocation on the next commit.
`void`	`establishExtent(long extent)` If the current file extent is different from the required extent then the call is made to `extendFile(int)`.
`protected void`	`finalize()` make sure resource is closed!
`void`	`flushWrites(boolean metadata)`
`void`	`free(long laddr, int sze)` Frees allocated storage (clears the bit to enable recycling after the next commit).
`void`	`free(long laddr, int sze, IAllocationContext context)` free
`int`	`getActiveTxCount()` Debug ONLY method added to permit unit tests to be written that the native transaction counter is correctly decremented to zero.
`int`	`getAllocatedBlocks()`
`long`	`getAllocatedSlots()` Computes the amount of utilised storage
`Allocator`	`getAllocator(int i)`
`int`	`getAssociatedSlotSize(int addr)` Returns the slot size associated with this address
`long`	`getBlockSequence()`
`Lock`	`getCommitLock()` Optionally return a `Lock` that must be used (when non- `null`) to make the `IStore.commit()` / `IStore.postCommit()` strategy atomic.
`CounterSet`	`getCounters()` Return interesting information about the write cache and file operations.
`long`	`getCurrentBlockSequence()`
`void`	`getData(long addr, byte[] buf)` If the buf[] size is greater than the maximum fixed allocation, then the direct read will be the blob header record.
`void`	`getData(long addr, byte[] buf, int offset, int length)`
`ByteBuffer`	`getData(long rwaddr, int sze)` Alternative method signature returning a ByteBuffer rather than receiving a byte array.
`long`	`getFileStorage()`
`int`	`getFixedAllocatorCount()`
`InputStream`	`getInputStream(long addr)` Return an input stream from which a previously written stream may be read back.
`long`	`getLastReleaseTime()` If history is retained this returns the time for which data was most recently released.
`int`	`getMaxAllocSize()` The maximum allocation size (bytes).
`int`	`getMaxBlobSize()`
`long`	`getMaxFileSize()`
`long`	`getMetaBitsAddr()` Since we need to store the absolute address and the size can be a maximum of 64K, the absolute address is limited to 48 bits, setting the maximum address as 140T, which is sufficient.
`long`	`getMetaBitsStoreAddress()`
`long`	`getMetaStartAddr()`
`long`	`getNextOffset()`
`IPSOutputStream`	`getOutputStream()` Return an output stream which can be used to write on the backing store.
`IPSOutputStream`	`getOutputStream(IAllocationContext context)`
`int`	`getSlotSize(int data_len)`
`StorageStats`	`getStorageStats()`
`RWStore.StoreCounters<?>`	`getStoreCounters()` Returns the striped performance counters for the store.
`File`	`getStoreFile()` Retrieves store file.
`StoreState`	`getStoreState()`
`long`	`getTotalAllocations()` The # of allocation requests made.
`long`	`getTotalAllocationsSize()` The # of bytes requested - as opposed to the size of the slots allocated.
`long`	`getTotalFrees()` The # of free requests made
`CounterSet`	`getWriteCacheCounters()`
`RWWriteCacheService`	`getWriteCacheService()` Return the then current `WriteCacheService` object.
`boolean`	`inWriteCache(int rwaddr)`
`boolean`	`isCommitted(int rwaddr)` Return `true` iff the allocation having that address is flagged as committed.
`boolean`	`isDirty()`
`boolean`	`isNativeAddress(long addr)` The `RWStore` always generates negative address values.
`boolean`	`isOpen()`
`boolean`	`isUsingDemiSpace()`
`void`	`lockAddress(int addr)` lockAddress adds the address passed to a lock list.
`IAllocationContext`	`newAllocationContext(boolean isolated)` Creates a context to be used to isolate updates to within the context until it is released to the parent environment.
`IRawTx`	`newTx()` A hook used to support session protection by incrementing and decrementing a transaction counter within the `IStore`.
`long`	`physicalAddress(int addr)` Return the byte offset in the file.
`void`	`postCommit()` Hook that supports synchronization with an external commit before which a rollback to "pre-commit" state is supported.
`void`	`postHACommit(IRootBlockView rbv)` Called from `AbstractJournal` commit2Phase to ensure that a downstream HA quorum member ensures it is able to read committed data that has been streamed directly to the backing store.
`protected void`	`readAllocationBlocks()`
`byte[]`	`readFromLatchedAddress(int nxtAddr)` A low level utility method that reads directly from the backing `FileChannel`.
`ByteBuffer`	`readRaw(long offset, ByteBuffer dst)` Read on the backing file.
`ByteBuffer`	`readRootBlock(boolean rootBlock0)`
`PSOutputStream`	`realloc(long oldAddr, int size)` The base realloc method that returns a stream for writing to rather than handle the reallocation immediately.
`void`	`registerExternalCache(ConcurrentWeakValueCache<Long,ICommitter> externalCache, int dataSize)` Call made from AbstractJournal to register the cache used.
`boolean`	`requiresCommit()`
`void`	`reset()` The semantics of reset are to revert unisolated writes to committed state.
`void`	`resetFromHARootBlock(IRootBlockView rootBlock)` Low level routine used when we replace the root blocks of an empty journal in HA with those from the leader.
`long`	`saveDeferrals()` Saves the current list of delete blocks, returning the address allocated.
`Future<Void>`	`sendHALogBuffer(IHALogRequest req, IHAWriteMessage msg, IBufferAccess buf)`
`Future<Void>`	`sendRawBuffer(IHARebuildRequest req, long sequence, long quorumToken, long fileExtent, long offset, int nbytes, ByteBuffer b)`
`void`	`setStoreCounters(RWStore.StoreCounters<?> storeCounters)` Replaces the `RWStore.StoreCounters` object.
`void`	`showAllocators(StringBuilder str)` Utility debug outputing the allocator array, showing index, start address and alloc type/size Collected statistics are against each Allocation Block size: total number of slots \| store size number of filled slots \| store used AllocatorSizeThe #of bytes in the allocated slots issued by this allocator. AllocatorCountThe #of fixed allocators for that slot size. SlotsInUseThe difference between the two previous columns (net slots in use for this slot size). SlotsReservedThe #of slots in this slot size which have had storage reserved for them. SlotsAllocatedCumulative allocation of slots to date in this slot size (regardless of the transaction outcome). SlotsRecycledCumulative recycled slots to date in this slot size (regardless of the transaction outcome). SlotsChurnHow frequently slots of this size are re-allocated (SlotsInUse/SlotsAllocated). %SlotsUnusedThe percentage of slots of this size which are not in use (1-(SlotsInUse/SlotsReserved)). BytesReservedThe space reserved on the backing file for those allocation slots BytesAppDataThe #of bytes in the allocated slots which are used by application data (including the record checksum). %SlotWasteHow well the application data fits in the slots (BytesAppData/(SlotsInUse*AllocatorSize)). %AppDataHow much of your data is stored by each allocator (BytesAppData/Sum(BytesAppData)). %StoreFileHow much of the backing file is reserved for each allocator (BytesReserved/Sum(BytesReserved)). %StoreWasteHow much of the total waste on the store is waste for this allocator size ((BytesReserved-BytesAppData)/(Sum(BytesReserved)-Sum(BytesAppData))).
`void`	`showWriteCacheDebug(long paddr)`
`void`	`snapshotAllocators(AbstractJournal.ISnapshotData tm)` Add the address/allocator associated with each FixedAllocator to the snapshot map
`void`	`snapshotMetabits(AbstractJournal.ISnapshotData tm)` Add the address/byte[] to the snapshot representing the metabits allocaiton data
`boolean`	`verify(long laddr)` Given a physical address (byte offset on the store), return true if that address could be managed by an allocated block.
`boolean`	`verifyAllocatedAddress(long addr)` Can be used to determine if an address is within an allocated slot.
`void`	`writeOnStream(OutputStream os, AbstractJournal.ISnapshotData snapshotData, Quorum<HAGlue,QuorumService<HAGlue>> quorum, long token)`
`void`	`writeOnStream2(OutputStream os, Set<Map.Entry<Long,byte[]>> snapshotData, Quorum<HAGlue,QuorumService<HAGlue>> quorum, long token)`
`void`	`writeRaw(long offset, ByteBuffer transfer)` Used as part of the rebuild protocol
`void`	`writeRawBuffer(IHAWriteMessage msg, IBufferAccess b)`
`void`	`writeRootBlock(IRootBlockView rootBlock, ForceEnum forceOnCommit)` This can be called as part of the HA downstream replication.

Methods inherited from class java.lang.Object
clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - RWStore
```
public RWStore(FileMetadata fileMetadata,
       Quorum<?,?> quorum)
```
    The ALLOC_SIZES must be initialized from either the file or the properties associated with the fileMetadataView
    
    Parameters:
    fileMetadataView -
    readOnly -
    quorum -
    
    Throws:
    
    InterruptedException
- Method Detail
  - getWriteCacheService
```
public RWWriteCacheService getWriteCacheService()
```
    Return the then current WriteCacheService object.
    
    See Also:
    IHABufferStrategy.getWriteCacheService()
  - isOpen
```
public boolean isOpen()
```
  - close
```
public void close()
```
    Description copied from interface: IStore
    
    Close the backing storage.
    
    Specified by:
    
    close in interface IStore
  - finalize
```
protected void finalize()
```
    make sure resource is closed!
    
    Overrides:
    
    finalize in class Object
  - readAllocationBlocks
```
protected void readAllocationBlocks()
                             throws IOException
```
    Throws:
    
    IOException
  - getMaxFileSize
```
public long getMaxFileSize()
```
  - getData
```
public ByteBuffer getData(long rwaddr,
                 int sze)
```
    Alternative method signature returning a ByteBuffer rather than receiving a byte array.
    If a blob then an extra byte array is required in which to build the data, but otherwise extra buffering could be avoided be reading directly from the WriteCacheService.
    
    Parameters:
    rwaddr -
    sze -
    
    Returns:
  - getData
```
public void getData(long addr,
           byte[] buf)
```
    If the buf[] size is greater than the maximum fixed allocation, then the direct read will be the blob header record. In this case we should hand over the streaming to a PSInputStream. FIXME: Javadoc update (was: For now we do not use the PSInputStream but instead process directly...) If it is a BlobAllocation, then the BlobAllocation address points to the address of the BlobHeader record.
    
    Specified by:
    
    getData in interface IStore
    
    Parameters:
    addr - the address of the data
    buf - the buffer of the size required!
  - getData
```
public void getData(long addr,
           byte[] buf,
           int offset,
           int length)
```
  - free
```
public void free(long laddr,
        int sze)
```
    Description copied from interface: IStore
    
    Frees allocated storage (clears the bit to enable recycling after the next commit).
    
    Specified by:
    
    free in interface IStore
    
    Parameters:
    laddr - the storage address to be freed
  - free
```
public void free(long laddr,
        int sze,
        IAllocationContext context)
```
    free
    If the address is greater than zero than it is interpreted as a physical address and the allocators are searched to find the allocations. Otherwise the address directly encodes the allocator index and bit offset, allowing direct access to clear the allocation.
    A blob allocator contains the allocator index and offset, so an allocator contains up to 245 blob references.
    
    Parameters:
    laddr -
    sze -
    context -
  - alloc
```
public int alloc(int size,
        IAllocationContext context)
```
    Core allocation method.
  - realloc
```
public PSOutputStream realloc(long oldAddr,
                     int size)
```
    The base realloc method that returns a stream for writing to rather than handle the reallocation immediately.
  - alloc
```
public long alloc(byte[] buf,
         int size,
         IAllocationContext context)
```
    Called by PSOutputStream to make to actual allocation or directly by lower level API clients.
    If the allocation is for greater than MAX_FIXED_ALLOC, then a PSOutputStream is used to manage the chained buffers. TODO: Instead of using PSOutputStream, manage allocations written to the WriteCacheService, building BlobHeader as you go.
    
    Specified by:
    
    alloc in interface IStore
    
    Returns:
    the allocated address
  - reset
```
public void reset()
```
    The semantics of reset are to revert unisolated writes to committed state.
    Unisolated writes must also be removed from the write cache.
    The AllocBlocks of the FixedAllocators maintain the state to determine the correct reset behavior.
    If the store is using DirectFixedAllocators then an IllegalStateException is thrown.
    If there is an active m_commitStateRef, then this indicates a failure after the commit() had "succeeded".
  - isDirty
```
public boolean isDirty()
```
    Returns:
  - commit
```
public void commit()
```
    Description copied from interface: IStore
    
    Global commit on the backing store. Previously committed data which has been marked as IStore.free(long, int) is now available for recycling. However, recycling can not occur if session protection is active.
    
    Specified by:
    
    commit in interface IStore
  - getCommitLock
```
public Lock getCommitLock()
```
    Optionally return a Lock that must be used (when non- null) to make the IStore.commit() / IStore.postCommit() strategy atomic.
    
    Specified by:
    
    getCommitLock in interface IStore
  - postCommit
```
public void postCommit()
```
    Hook that supports synchronization with an external commit before which a rollback to "pre-commit" state is supported.
    Commits the FixedAllocator bits
    
    Specified by:
    
    postCommit in interface IStore
  - checkDeferredFrees
```
public int checkDeferredFrees(AbstractJournal journal)
```
    Description copied from interface: IHistoryManager
    
    This method is invoked during the commit protocol and gives the backing store an opportunity to check whether storage associated with deferred frees can now be released. The backing store needs to reply the deferred free blocks for the commit points that will be released, mark those addresses as free, and free the logged delete blocks.
    
    Specified by:
    
    checkDeferredFrees in interface IHistoryManager
    
    Returns:
    number of addresses freed
    See Also:
    AbstractJournal#commitNow()
  - allocBlock
```
protected int allocBlock(int size)
```
    Return the address of a contiguous region on the persistent heap.
    
    Parameters:
    size - The size of that region (this is not bytes, but something a bit more complicated).
  - convertAddr
```
public static long convertAddr(int addr)
```
    Convert an implicitly scaled int32 offset into the backing file into an int64 address into the backing file.
    
    Parameters:
    addr - An int32 offset into the backing file formed by convertFromAddr(long). The representation is a negative integer that has been left shifted by ALLOCATION_SCALEUP to reduce its bit size.
    
    Returns:
    A byte offset in the backing file.
    See Also:
    convertFromAddr(long), ALLOCATION_SCALEUP
  - convertFromAddr
```
public int convertFromAddr(long addr)
```
    Convert an int64 address into the backing file into an int32 offset that is implicitly scaled by ALLOCATION_SCALEUP.
    
    Parameters:
    addr - An int64 offset into the backing file.
    
    Returns:
    The implicitly scaled int32 offset.
    See Also:
    convertAddr(int), ALLOCATION_SCALEUP
  - showAllocators
```
public void showAllocators(StringBuilder str)
```
    Utility debug outputing the allocator array, showing index, start address and alloc type/size Collected statistics are against each Allocation Block size: total number of slots | store size number of filled slots | store used
    
    AllocatorSize
    The #of bytes in the allocated slots issued by this allocator.
    
    AllocatorCount
    The #of fixed allocators for that slot size.
    
    SlotsInUse
    The difference between the two previous columns (net slots in use for this slot size).
    
    SlotsReserved
    The #of slots in this slot size which have had storage reserved for them.
    
    SlotsAllocated
    Cumulative allocation of slots to date in this slot size (regardless of the transaction outcome).
    
    SlotsRecycled
    Cumulative recycled slots to date in this slot size (regardless of the transaction outcome).
    
    SlotsChurn
    How frequently slots of this size are re-allocated (SlotsInUse/SlotsAllocated).
    
    %SlotsUnused
    The percentage of slots of this size which are not in use (1-(SlotsInUse/SlotsReserved)).
    
    BytesReserved
    The space reserved on the backing file for those allocation slots
    
    BytesAppData
    The #of bytes in the allocated slots which are used by application data (including the record checksum).
    
    %SlotWaste
    How well the application data fits in the slots (BytesAppData/(SlotsInUse*AllocatorSize)).
    
    %AppData
    How much of your data is stored by each allocator (BytesAppData/Sum(BytesAppData)).
    
    %StoreFile
    How much of the backing file is reserved for each allocator (BytesReserved/Sum(BytesReserved)).
    
    %StoreWaste
    How much of the total waste on the store is waste for this allocator size ((BytesReserved-BytesAppData)/(Sum(BytesReserved)-Sum(BytesAppData))).
    
    See Also:
    StorageStats.showStats(StringBuilder)
  - verify
```
public boolean verify(long laddr)
```
    Given a physical address (byte offset on the store), return true if that address could be managed by an allocated block.
    
    Parameters:
    a - the storage address to be tested.
  - physicalAddress
```
public final long physicalAddress(int addr)
```
    Return the byte offset in the file.
    
    Parameters:
    addr - A latched address.
    
    Returns:
    The byte offset.
  - isNativeAddress
```
public boolean isNativeAddress(long addr)
```
    The RWStore always generates negative address values.
    
    Returns:
    whether the address given is a native IStore address
  - getStoreFile
```
public File getStoreFile()
```
    Description copied from interface: IStore
    
    Retrieves store file. Can be used to delete the store after the IStore has been released
    
    Specified by:
    
    getStoreFile in interface IStore
    
    Returns:
    the File object
  - requiresCommit
```
public boolean requiresCommit()
```
  - getMetaBitsAddr
```
public long getMetaBitsAddr()
```
    Since we need to store the absolute address and the size can be a maximum of 64K, the absolute address is limited to 48 bits, setting the maximum address as 140T, which is sufficient.
    
    Returns:
    long representation of metaBitsAddr PLUS the size
  - getMetaBitsStoreAddress
```
public long getMetaBitsStoreAddress()
```
    Returns:
    the address of the metaBits
  - getMetaStartAddr
```
public long getMetaStartAddr()
```
    Returns:
    long representation of metaStartAddr PLUS the size where addr + size is fileSize (not necessarily physical size)
  - getNextOffset
```
public long getNextOffset()
```
    Returns:
    the nextAllocation from the file Heap to be provided to an Allocation Block
  - flushWrites
```
public void flushWrites(boolean metadata)
                 throws IOException
```
    Throws:
    
    IOException
  - getTotalAllocations
```
public long getTotalAllocations()
```
    The # of allocation requests made.
  - getTotalFrees
```
public long getTotalFrees()
```
    The # of free requests made
  - getTotalAllocationsSize
```
public long getTotalAllocationsSize()
```
    The # of bytes requested - as opposed to the size of the slots allocated.
  - getAllocator
```
public Allocator getAllocator(int i)
```
  - establishExtent
```
public void establishExtent(long extent)
```
    If the current file extent is different from the required extent then the call is made to extendFile(int).
    
    Parameters:
    extent - The new file extent.
  - getFixedAllocatorCount
```
public int getFixedAllocatorCount()
```
    Returns:
    number of FixedAllocators
  - getAllocatedBlocks
```
public int getAllocatedBlocks()
```
    Returns:
    the number of heap allocations made to the FixedAllocators.
  - getFileStorage
```
public long getFileStorage()
```
    Returns:
    the amount of heap storage assigned to the FixedAllocators.
  - getAllocatedSlots
```
public long getAllocatedSlots()
```
    Computes the amount of utilised storage
    
    Returns:
    the amount of storage to alloted slots in the allocation blocks
  - deferFree
```
public void deferFree(int rwaddr,
             int sze)
```
    Adds the address for later freeing to the deferred free list.
    If the allocation is for a BLOB then the sze is also stored
    The deferred list is checked on AllocBlock and prior to commit.
    DeferredFrees are written to the deferred PSOutputStream
  - saveDeferrals
```
public long saveDeferrals()
```
    Description copied from interface: IHistoryManager
    
    Saves the current list of delete blocks, returning the address allocated. This can be used later to retrieve the addresses of allocations to be freed.
    Writes the content of currentTxnFreeList to the store.
    These are the current buffered frees that have yet been saved into a block referenced from the deferredFreeList
    
    Specified by:
    
    saveDeferrals in interface IHistoryManager
    
    Returns:
    the address of the deferred addresses saved on the store, or zero if none.
    See Also:
    DeleteBlockCommitter
  - detachContext
```
public void detachContext(IAllocationContext context)
```
    Indicates that the allocation context will no longer be used, but that the allocations made within the context should be preserved. The allocations associated with the context are propagated to the parent allocation context. The IStore is the top-level parent of allocation contexts. The allocators associated with the allocation context are return to the global list of available allocators.
    The ContextAllocation object manages a freeList of associated allocators and an overall list of allocators. When the context is detached, all allocators must be released and any that has available capacity will be assigned to the global free lists. See #releaseSession
    
    Specified by:
    
    detachContext in interface IAllocationManager
    
    Parameters:
    context - The context to be released from all FixedAllocators.
  - abortContext
```
public void abortContext(IAllocationContext context)
```
    The ContextAllocation object manages a freeList of associated allocators and an overall list of allocators. When the context is aborted then allocations made by that context should be released. See #abortShadow
    
    Specified by:
    
    abortContext in interface IAllocationManager
    
    Parameters:
    context - The context to be released from all FixedAllocators.
  - getSlotSize
```
public int getSlotSize(int data_len)
```
    Specified by:
    
    getSlotSize in interface IBufferedWriter
  - getMaxAllocSize
```
public int getMaxAllocSize()
```
    The maximum allocation size (bytes).
  - writeRootBlock
```
public void writeRootBlock(IRootBlockView rootBlock,
                  ForceEnum forceOnCommit)
```
    This can be called as part of the HA downstream replication. FIXME: If part of downstream replication then the current metabits held by the RWStore will not be in sync with that stored on disk. This will only be a problem if the RWStore needs to take over as leader and be able to allocate and write to the store. Note that the metabits are not needed in order to determine the physical address mapping of an rw-native address.
    
    Parameters:
    rootBlock -
    forceOnCommit -
  - readRootBlock
```
public ByteBuffer readRootBlock(boolean rootBlock0)
```
  - getStoreCounters
```
public RWStore.StoreCounters<?> getStoreCounters()
```
    Returns the striped performance counters for the store.
    
    Specified by:
    
    getStoreCounters in interface IBufferedWriter
  - setStoreCounters
```
public void setStoreCounters(RWStore.StoreCounters<?> storeCounters)
```
    Replaces the RWStore.StoreCounters object.
    
    Parameters:
    storeCounters - The new BTree.Counters.
    
    Throws:
    
    IllegalArgumentException - if the argument is null.
  - getCounters
```
public CounterSet getCounters()
```
    Return interesting information about the write cache and file operations.
  - writeRawBuffer
```
public void writeRawBuffer(IHAWriteMessage msg,
                  IBufferAccess b)
                    throws IOException,
                           InterruptedException
```
    Throws:
    
    IOException
    
    InterruptedException
  - sendHALogBuffer
```
public Future<Void> sendHALogBuffer(IHALogRequest req,
                           IHAWriteMessage msg,
                           IBufferAccess buf)
                             throws IOException,
                                    InterruptedException
```
    Throws:
    
    IOException
    
    InterruptedException
  - sendRawBuffer
```
public Future<Void> sendRawBuffer(IHARebuildRequest req,
                         long sequence,
                         long quorumToken,
                         long fileExtent,
                         long offset,
                         int nbytes,
                         ByteBuffer b)
                           throws IOException,
                                  InterruptedException
```
    Throws:
    
    IOException
    
    InterruptedException
    See Also:
    IHABufferStrategy.sendRawBuffer(IHARebuildRequest, long, long, long, long, int, ByteBuffer)
  - writeOnStream
```
public void writeOnStream(OutputStream os,
                 AbstractJournal.ISnapshotData snapshotData,
                 Quorum<HAGlue,QuorumService<HAGlue>> quorum,
                 long token)
                   throws IOException,
                          QuorumException,
                          InterruptedException
```
    Throws:
    
    IOException
    
    QuorumException
    
    InterruptedException
  - writeOnStream2
```
public void writeOnStream2(OutputStream os,
                  Set<Map.Entry<Long,byte[]>> snapshotData,
                  Quorum<HAGlue,QuorumService<HAGlue>> quorum,
                  long token)
                    throws IOException,
                           QuorumException
```
    Throws:
    
    IOException
    
    QuorumException
  - readRaw
```
public ByteBuffer readRaw(long offset,
                 ByteBuffer dst)
```
    Read on the backing file. Buffer.remaining() bytes will be read into the caller's buffer, starting at the specified offset in the backing file.
    
    Specified by:
    
    readRaw in interface IBackingReader
    
    Parameters:
    offset - The offset of the first byte (relative to the start of the data region).
    dst - Where to put the data. Bytes will be written at position until limit.
    
    Returns:
    The caller's buffer, prepared for reading.
  - getMaxBlobSize
```
public int getMaxBlobSize()
```
  - getStorageStats
```
public StorageStats getStorageStats()
```
  - newTx
```
public IRawTx newTx()
```
    Description copied from interface: IHistoryManager
    
    A hook used to support session protection by incrementing and decrementing a transaction counter within the IStore. As long as a transaction is active we can not release data which is currently marked as freed but was committed at the point the session started.
    
    Specified by:
    
    newTx in interface IHistoryManager
  - getActiveTxCount
```
public int getActiveTxCount()
```
    Debug ONLY method added to permit unit tests to be written that the native transaction counter is correctly decremented to zero. The returned value is ONLY valid while holding the m_allocationLock. Therefore this method MAY NOT be used reliably outside of code that can guarantee that there are no concurrent committers on the RWStore.
    
    See Also:
    Journal file growth reported with 1.3.3
  - getAssociatedSlotSize
```
public int getAssociatedSlotSize(int addr)
```
    Returns the slot size associated with this address
    
    Specified by:
    
    getAssociatedSlotSize in interface IStore
    
    Parameters:
    addr - - the address
    
    Returns:
    the size of the slot associated
  - lockAddress
```
public void lockAddress(int addr)
```
    lockAddress adds the address passed to a lock list. This is for debug only and is not intended to be used generally for the live system.
    
    Parameters:
    addr - - address to be locked
  - showWriteCacheDebug
```
public void showWriteCacheDebug(long paddr)
```
  - getWriteCacheCounters
```
public CounterSet getWriteCacheCounters()
```
  - getLastReleaseTime
```
public long getLastReleaseTime()
```
    Description copied from interface: IHistoryManager
    
    If history is retained this returns the time for which data was most recently released. No request can be made for data earlier than this.
    
    Specified by:
    
    getLastReleaseTime in interface IHistoryManager
    
    Returns:
    latest data release time
  - registerExternalCache
```
public void registerExternalCache(ConcurrentWeakValueCache<Long,ICommitter> externalCache,
                         int dataSize)
```
    Description copied from interface: IHistoryManager
    
    Call made from AbstractJournal to register the cache used. This can then be accessed to clear entries when storage is made available for re-cycling.
    Note: It is not safe to clear at the point of the delete request since the data could still be loaded if the data is retained for a period due to a non-zero retention period or session protection.
    
    Specified by:
    
    registerExternalCache in interface IHistoryManager
  - isCommitted
```
public boolean isCommitted(int rwaddr)
```
    Return true iff the allocation having that address is flagged as committed. The caller must be holding the allocation lock in order for the result to remain valid outside of the method call.
    
    Parameters:
    addr - The address.
    
    Returns:
    true iff the address is currently committed.
  - inWriteCache
```
public boolean inWriteCache(int rwaddr)
```
  - getInputStream
```
public InputStream getInputStream(long addr)
```
    Description copied from interface: IStreamStore
    
    Return an input stream from which a previously written stream may be read back.
    
    Specified by:
    
    getInputStream in interface IStreamStore
    
    Parameters:
    addr - The address at which the stream was written.
    
    Returns:
    an input stream for the data for provided address
  - getOutputStream
```
public IPSOutputStream getOutputStream()
```
    Description copied from interface: IStreamStore
    
    Return an output stream which can be used to write on the backing store. You can recover the address used to read back the data from the IPSOutputStream.
    
    Specified by:
    
    getOutputStream in interface IStreamStore
    
    Returns:
    The output stream.
  - getOutputStream
```
public IPSOutputStream getOutputStream(IAllocationContext context)
```
  - resetFromHARootBlock
```
public void resetFromHARootBlock(IRootBlockView rootBlock)
```
    Low level routine used when we replace the root blocks of an empty journal in HA with those from the leader.
    Note: This method is only invoked in contexts where there should not be concurrent access to the journal. This we should not need to worry about concurrent readers during resetFromHARootBlock(IRootBlockView).
    
    See Also:
    postHACommit(IRootBlockView)
  - postHACommit
```
public void postHACommit(IRootBlockView rbv)
```
    Called from AbstractJournal commit2Phase to ensure that a downstream HA quorum member ensures it is able to read committed data that has been streamed directly to the backing store.
    The data stream will have included metabits and modified FixedAllocators so these must be reset using the metabitsAddr data in the root block.
    Note: Reads on the RWStore MUST block during this method since some allocators may be replaced as part of the post-commit protocol.
    Ticket #778 was for a problem when a follower takes over as leader and was not correctly synchronised. This was traced, eventually, to a problem in calculating the diskAddr metabit for the modified Allocator. The problem was demonstrated by a temporary method to reserve metaAllocations by extending and setting the m_transient bits. But that has to be done within the commit() method before it attempts to save all the dirty allocators. If we need to contrive a similar scenario in the future a better approach would be a special debug property on the RWStore that indicates a "TRANSIENT_RESERVE" or something similar.
    
    Parameters:
    rbv - The new IRootBlockView.
  - checkDeleteBlocks
```
public RWStore.DeleteBlockStats checkDeleteBlocks(AbstractJournal journal)
```
    Utility to check the deleteBlocks associated with each active CommitRecord
  - readFromLatchedAddress
```
public final byte[] readFromLatchedAddress(int nxtAddr)
                                    throws IOException
```
    A low level utility method that reads directly from the backing FileChannel.
    Note: The latched address does not encode the actual length of the data. Therefore, all data in the slot addressed by the latched address will be returned.
    
    Parameters:
    nxtAddr - The latched address.
    
    Returns:
    The byte[] in the addressed slot.
    
    Throws:
    
    IOException
  - getBlockSequence
```
public long getBlockSequence()
```
    See Also:
    IHABufferStrategy.getBlockSequence()
  - getCurrentBlockSequence
```
public long getCurrentBlockSequence()
```
    See Also:
    IHABufferStrategy.getCurrentBlockSequence()
  - computeDigest
```
public void computeDigest(Object snapshot,
                 MessageDigest digest)
                   throws DigestException,
                          IOException
```
    Throws:
    
    DigestException
    
    IOException
    See Also:
    IHABufferStrategy.computeDigest(Object, MessageDigest)
  - writeRaw
```
public void writeRaw(long offset,
            ByteBuffer transfer)
              throws IOException
```
    Used as part of the rebuild protocol
    
    Throws:
    
    IOException
  - verifyAllocatedAddress
```
public boolean verifyAllocatedAddress(long addr)
```
    Can be used to determine if an address is within an allocated slot.
    
    Parameters:
    addr -
    
    Returns:
    whether addr is within slot allocated area
  - getStoreState
```
public StoreState getStoreState()
```
  - ensureMetabitsDemispace
```
public boolean ensureMetabitsDemispace(boolean useDemispace)
```
    Forces a reset of the metabits allocation on the next commit.
    Note that a side-effect of this is that there will be a memory leak of either a FixedAllocation slot or an existing demi-space.
    
    Parameters:
    useDemispace -
    
    Returns:
    whether the storage has been modified.
  - isUsingDemiSpace
```
public boolean isUsingDemiSpace()
```
  - snapshotMetabits
```
public void snapshotMetabits(AbstractJournal.ISnapshotData tm)
                      throws IOException
```
    Add the address/byte[] to the snapshot representing the metabits allocaiton data
    
    Throws:
    
    IOException
  - snapshotAllocators
```
public void snapshotAllocators(AbstractJournal.ISnapshotData tm)
```
    Add the address/allocator associated with each FixedAllocator to the snapshot map
  - newAllocationContext
```
public IAllocationContext newAllocationContext(boolean isolated)
```
    Description copied from interface: IAllocationManager
    
    Creates a context to be used to isolate updates to within the context until it is released to the parent environment.
    
    Specified by:
    
    newAllocationContext in interface IAllocationManager
    
    Returns:
    a new IAlocationContext

Class RWStore

Deferred Free List

Nested Class Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

RWStore

Method Detail

getWriteCacheService

isOpen

close

finalize

readAllocationBlocks

getMaxFileSize

getData

getData

getData

free

free

alloc

realloc

alloc

reset

isDirty

commit

getCommitLock

postCommit

checkDeferredFrees

allocBlock

convertAddr

convertFromAddr

showAllocators

verify

physicalAddress

isNativeAddress

getStoreFile

requiresCommit

getMetaBitsAddr

getMetaBitsStoreAddress

getMetaStartAddr

getNextOffset

flushWrites

getTotalAllocations

getTotalFrees

getTotalAllocationsSize

getAllocator

establishExtent

getFixedAllocatorCount

getAllocatedBlocks

getFileStorage

getAllocatedSlots

deferFree

saveDeferrals

detachContext

abortContext

getSlotSize

getMaxAllocSize

writeRootBlock

readRootBlock

getStoreCounters

setStoreCounters

getCounters

writeRawBuffer

sendHALogBuffer

sendRawBuffer

writeOnStream

writeOnStream2

readRaw

getMaxBlobSize

getStorageStats

newTx

getActiveTxCount

getAssociatedSlotSize

lockAddress

showWriteCacheDebug

getWriteCacheCounters

getLastReleaseTime

registerExternalCache

isCommitted