com.bigdata.rwstore

Interface RWStore.Options

Enclosing class:

RWStore

public static interface RWStore.Options

Options understood by the RWStore.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ALLOCATION_SIZES Option defines the Allocation block sizes for the RWStore.
static String	DEFAULT_ALLOCATION_SIZES Note: The default allocation sizes SHOULD NOT provide for allocation slots larger than an 8k page.
static String	DEFAULT_DOUBLE_BUFFER_WRITES
static String	DEFAULT_FREE_BITS_THRESHOLD
static String	DEFAULT_META_BITS_DEMI_SPACE
static String	DEFAULT_META_BITS_SIZE Deprecated.
static String	DEFAULT_READ_BLOBS_ASYNC Note: Windows does not handle async IO channel reopens in the same fashion as Linux, leading to "overlapping file exceptions" and other weirdness.
static String	DEFAULT_SMALL_SLOT_HIGH_WASTE
static String	DEFAULT_SMALL_SLOT_THRESHOLD
static String	DEFAULT_SMALL_SLOT_TYPE Enable the small slot optimization by default.
static String	DEFAULT_SMALL_SLOT_WASTE_CHECK_ALLOCATORS
static String	DOUBLE_BUFFER_WRITES When true, scattered writes which are strictly ascending will be coalesced within a buffer and written out as a single IO (default "true").
static String	FREE_BITS_THRESHOLD Defines the number of bits that must be free in a FixedAllocator for it to be added to the free list.
static String	META_BITS_DEMI_SPACE Defines whether the metabits should be allocated an explicit demispace (default) or if not, then to use a standard Allocation (which limits the metabits size to the maximum FixedAllocator slot size).
static String	META_BITS_SIZE Deprecated.
static String	READ_BLOBS_ASYNC Defines whether blobs, which are stored in multiple slot locations, are read concurrently using Async NIO.
static String	SMALL_SLOT_HIGH_WASTE Once there are at least SMALL_SLOT_WASTE_CHECK_ALLOCATORS for a given slot size, then the SMALL_SLOT_HIGH_WASTE specifies the maximum percentage of waste that will be allowed for that slot size.
static String	SMALL_SLOT_THRESHOLD The #of free bits required to be free in a "small slot" allocator before it is automatically returned to the free list.
static String	SMALL_SLOT_TYPE Defines the size of a slot that defines it as a small slot.
static String	SMALL_SLOT_WASTE_CHECK_ALLOCATORS We have introduced extra parameters to adjust allocator usage if we notice that a significant amount of storage is wasted.

Field Detail

ALLOCATION_SIZES

static final String ALLOCATION_SIZES

Option defines the Allocation block sizes for the RWStore. The values defined are multiplied by 64 to provide the actual allocations. The list of allocations should be ',' delimited and in increasing order. This array is written into the store so changing the values does not break older stores. For example,

 "1,2,4,8,116,32,64"
 

defines allocations from 64 to 4K in size. It is a good to define block sizes on 4K boundaries as soon as possible to optimize IO. This is particularly relevant for SSDs. A 1K boundary is expressed as 16 in the allocation sizes, so a 4K boundary is expressed as 64 and an 8k boundary as 128.

The default allocations are "1, 2, 3, 5, 8, 12, 16, 32, 48, 64, 128".

See Also:

DEFAULT_ALLOCATION_SIZES

DEFAULT_ALLOCATION_SIZES

static final String DEFAULT_ALLOCATION_SIZES

Note: The default allocation sizes SHOULD NOT provide for allocation slots larger than an 8k page. This can lead to large allocation slots when a B+Tree index is sparsely populated (less efficient prefix compression) followed by a gradual reduction in the average page size with the net effect that large allocators become unused and turn into wasted and unrecoverable space on the backing file. Keeping to an 8k maximum allocation slot size means that we have to do a few more IOs if the page exceeds the 8k boundary, but we never wind up with those large and (mostly) unused allocators. The B+Tree branching factors should be tuned to target perhaps 80% of an 8k page in order to have only a small number of pages that spill over into blobs. TODO: We should consider a more adaptable BLOB approach where we specify the maximum "slop" in an allocation as the means to determine a blob boundary. So, for example, a 5.5K allocation, with maximum slop of 1K, would be allocated as a blob of 4K + 2K and not an 8K slot.

See Also:

ALLOCATION_SIZES, Constant Field Values

META_BITS_SIZE

@Deprecated
static final String META_BITS_SIZE

Deprecated.

Option defines the initial size of the meta bits region and effects how rapidly this region will grow (default "9").

Note: A value of 9 may be used to stress the logic which is responsible for the growth in the meta bits region.

This has now been deprecated since it adds complexity with no significant benefit

DEFAULT_META_BITS_SIZE

@Deprecated
static final String DEFAULT_META_BITS_SIZE

Deprecated.

See Also:

Constant Field Values

META_BITS_DEMI_SPACE

static final String META_BITS_DEMI_SPACE

Defines whether the metabits should be allocated an explicit demispace (default) or if not, then to use a standard Allocation (which limits the metabits size to the maximum FixedAllocator slot size).

The value should be either "true" or "false"

DEFAULT_META_BITS_DEMI_SPACE

static final String DEFAULT_META_BITS_DEMI_SPACE

See Also:

Constant Field Values

READ_BLOBS_ASYNC

static final String READ_BLOBS_ASYNC

Defines whether blobs, which are stored in multiple slot locations, are read concurrently using Async NIO. This was introduced specifically to reduce commit latency in scenarios where large transactions can lead to very large deferred free lists (>> 10 million addresses), stored as blobs.

BLZG-1884 indicated a possible problem with this approach. The root causes of that problem (poor handling of exceptions) have been dealt with. This option was also introduced so the async IO support can now be disabled if a problem does materialize.

DEFAULT_READ_BLOBS_ASYNC

static final String DEFAULT_READ_BLOBS_ASYNC

Note: Windows does not handle async IO channel reopens in the same fashion as Linux, leading to "overlapping file exceptions" and other weirdness. Therefore this option is explicitly disabled by default on Windows.

See Also:

(Blazegraph 2.1 version does not work on Windows (async IO causes file lock errors))

FREE_BITS_THRESHOLD

static final String FREE_BITS_THRESHOLD

Defines the number of bits that must be free in a FixedAllocator for it to be added to the free list. This is used to ensure a level of locality when making large numbers of allocations within a single commit.

The value should be >= 1 and <= 5000

DEFAULT_FREE_BITS_THRESHOLD

static final String DEFAULT_FREE_BITS_THRESHOLD

See Also:

Constant Field Values

SMALL_SLOT_TYPE

static final String SMALL_SLOT_TYPE

Defines the size of a slot that defines it as a small slot.

Any slot equal to or less than this is considered a small slot and its availability for allocation is restricted to ensure a high chance that contiguous allocations can be made.

This is arranged by only returning small slot allocators to the free list if they have greater than 50% available slots, and then only allocating slots from sparse regions with >= 50% free/committed bits.

Small slot processing can be disabled by setting the smallSlotType to zero.

DEFAULT_SMALL_SLOT_TYPE

static final String DEFAULT_SMALL_SLOT_TYPE

Enable the small slot optimization by default.

See Also:

(Enable small slot optimization by default), Constant Field Values

SMALL_SLOT_THRESHOLD

static final String SMALL_SLOT_THRESHOLD

The #of free bits required to be free in a "small slot" allocator before it is automatically returned to the free list. Once the small slot waste threshold comes into play, the small slot allocator for a given slot size having the maximum free bits will be automatically returned to the free list if the percentage of waste in that slot size exceeds a threshold.

See Also:

(Implement maximum waste policy for small slot allocators)

DEFAULT_SMALL_SLOT_THRESHOLD

static final String DEFAULT_SMALL_SLOT_THRESHOLD

See Also:

Constant Field Values

SMALL_SLOT_WASTE_CHECK_ALLOCATORS

static final String SMALL_SLOT_WASTE_CHECK_ALLOCATORS

We have introduced extra parameters to adjust allocator usage if we notice that a significant amount of storage is wasted.

First we check how many allocators of a given slot size have been created. If above then we look a little closer.

We retrieve the allocation statistics and determine if the waste threshold is exceeded, as determined by SMALL_SLOT_HIGH_WASTE.

If so, then we attempt to find an available allocator with more free bits as determined by SMALL_SLOT_THRESHOLD_HIGH_WASTE.

See Also:

(Implement maximum waste policy for small slot allocators)

DEFAULT_SMALL_SLOT_WASTE_CHECK_ALLOCATORS

static final String DEFAULT_SMALL_SLOT_WASTE_CHECK_ALLOCATORS

See Also:

Constant Field Values

SMALL_SLOT_HIGH_WASTE

static final String SMALL_SLOT_HIGH_WASTE

Once there are at least SMALL_SLOT_WASTE_CHECK_ALLOCATORS for a given slot size, then the SMALL_SLOT_HIGH_WASTE specifies the maximum percentage of waste that will be allowed for that slot size. This prevents the amount of waste for small slot allocators from growing significantly as the size of the backing store increases.

The dynamic policy for small slots can be thought of as follows.

A normal allocator will be dropped onto the free list once it has FREE_BITS_THRESHOLD bits free (default 300 bits out of 8192 = 3.6%).

For a new store, a small slot allocator will be dropped onto the free list once it has SMALL_SLOT_THRESHOLD bits free (default 4096 bits out of 8192 = 50%).

Once the #of small slots allocators for a given sized allocator exceeds the DEFAULT_SMALL_SLOT_WASTE_CHECK_ALLOCATORS, a small slot allocator will be dropped onto the free list once it is SMALL_SLOT_HIGH_WASTE percent free (this amounts to 1638 bits out of 8192).

Thus, the small slot allocators initially are created freely because they need to be highly sparse before they can be on the free list. Once we have "enough" small slot allocators, we create them less freely - this is achieved by changing the sparsity threshold to a value that still requires the small slot allocator to be significantly more sparse than a general purpose allocator.

See Also:

(Implement maximum waste policy for small slot allocators)

DEFAULT_SMALL_SLOT_HIGH_WASTE

static final String DEFAULT_SMALL_SLOT_HIGH_WASTE

See Also:

Constant Field Values

DOUBLE_BUFFER_WRITES

static final String DOUBLE_BUFFER_WRITES

When true, scattered writes which are strictly ascending will be coalesced within a buffer and written out as a single IO (default "true"). This improves write performance for SATA, SAS, and even SSD.

DEFAULT_DOUBLE_BUFFER_WRITES

static final String DEFAULT_DOUBLE_BUFFER_WRITES

See Also:

Constant Field Values