Package com.bigdata.service

This package provides implementations of bigdata services (metadata service, data service, transaction manager service.

See: Description

Interface Summary

Interface	Description
AbstractScaleOutClient.Options	Options understood by the AbstractScaleOutClient.
AbstractTransactionService.Options	Options understood by this service.
ClientService.Options	Configuration options.
DataService.IDataServiceCounters	Interface defines and documents the counters and counter namespaces reported by the DataService and the various services which it uses.
DataService.Options	Options understood by the DataService.
DistributedTransactionService.Options	Options understood by this service.
EmbeddedClient.Options	Options for the embedded (in process) federation.
IBigdataClient<T>	Interface for clients of a IBigdataFederation.
IBigdataClient.Options	Configuration options for IBigdataClients.
IBigdataFederation<T>	The client-facing interface to a bigdata federation.
IClientService	A service for distributing client Callables across the resources of an IBigdataFederation.
IDataService	The data service interface provides remote access to named indices, provides for both unisolated and isolated operations on those indices, and exposes the ITxCommitProtocol interface to the ITransactionManagerService service for the coordination of distributed transactions.
IDataServiceCallable	Interface for procedures that require access to the IDataService and or the federation.
IEventReceivingService	Remote interface for a service which can receive Events.
IEventReportingService	Extension of the common service interface to support event reporting.
IFederationCallable	Interface for Callables which require access to the IBigdataFederation when running on an IRemoteExecutor.
IFederationDelegate<T>	Interface allowing services to take over handling of events normally handled by the AbstractFederation.
ILoadBalancerService	Interface for collecting, reporting, and decision-making based on node and service utilization statistics.
IMetadataService	A metadata service for a named index.
IRemoteExecutor	Interface for running procedures on a remote service.
IService	Common service interface.
IServiceInit<T>	Interface for post-constructor initialization.
IServiceLoadHelper	Interface for decision making about the load imposed on services.
IServiceShutdown	Local API for service shutdown.
IServiceShutdown.Options	Options for IServiceShutdown implementations.
ISession	Non-remote interface exposing a transient property set associated with a service.
ITxCommitProtocol	Remote interface by which the ITransactionService manages the state of transactions on the distributed IDataServices.
ITxState	Interface for the state associated with a transaction in an ITransactionService.
ITxState0	Immutable state for a transaction.
LoadBalancerService.Options	Options understood by the LoadBalancerService.
MetadataService.Options	Options for the MetadataService.
Params	An interface designed to expose select fields for Event reporting.
StressTestConcurrent.TestOptions	Additional properties understood by this test.

Class Summary

Class	Description
AbstractClient<T>	Abstract base class for IBigdataClient implementations.
AbstractDistributedFederation<T>	Abstract base class for IBigdataFederation implementations where the services are distributed using RMI and are running, at least in principle, across more than one host/JVM.
AbstractEmbeddedDataService	A local (in process) data service.
AbstractEmbeddedFederationTestCase	An abstract test harness that sets up (and tears down) the metadata and data services required for a bigdata federation using in-process services rather than service discovery (which means that there is no network IO).
AbstractEmbeddedLoadBalancerService	Embedded LoadBalancerService.
AbstractEmbeddedTransactionService	Concrete implementation for an EmbeddedFederation.
AbstractFederation<T>	Abstract base class for IBigdataFederation implementations.
AbstractFederation.ReportTask	Periodically report performance counter data to the ILoadBalancerService.
AbstractHATransactionService	Adds local methods to support an HA aware ITransactionService.
AbstractIndexCache<T extends IRangeQuery>	Abstract base class providing caching for IIndex like objects.
AbstractRoundRobinServiceLoadHelper	A round robin implementation that may be used when there are no scores available.
AbstractScaleOutClient<T>	Client class for AbstractScaleOutFederations.
AbstractScaleOutFederation<T>	Abstract base class for federation implementations using the scale-out index architecture (federations that support key-range partitioned indices).
AbstractScaleOutFederation.ForceOverflowTask	Task forces immediate overflow of the specified data service, returning once both synchronous AND asynchronous overflow are complete.
AbstractScaleOutFederation.PurgeResourcesTask	Task directs a DataService to purge any unused resources and to optionally truncate the extent of the live journal.
AbstractService	Abstract base class defines protocols for setting the service UUID, etc.
AbstractServiceLoadHelper	Base class for abstract implementations with integration points for the LoadBalancerService.
AbstractServiceLoadHelperWithoutScores	Implementation that may be used when service scores are not yet available.
AbstractServiceLoadHelperWithScores	The default implementation used when scores are available.
AbstractTransactionService	Centralized transaction manager service.
CacheOnceMetadataIndex	Implementation caches all locators but does not allow stale locators.
CachingMetadataIndex	Implementation caches all locators and then updates them on demand as stale locators are discovered.
ClientService	A service for distributing application Callables across an IBigdataFederation.
ClientService.ClientServiceFederationDelegate	Extended to attach the various performance counters reported by the DistributedTransactionService.
CommitTimeIndex	BTree whose keys are commit times.
CommitTimeIndex.TupleSerializer	Encapsulates key and value formation.
DataService	An implementation of a network-capable IDataService.
DataService.DataServiceFederationDelegate	Delegate handles custom counters for the ResourceManager, local AbstractTransactionService and the ConcurrencyManager, dynamic re-attachment of counters, etc.
DataService.GetIndexMetadataTask	Retrieves the IndexMetadata for the named index as of the specified timestamp.
DataService.RangeIteratorTask	Task for running a rangeIterator operation.
DataService.ReadBlockCounters
DataServiceCallable<T>	Base class for IDataServiceCallable.
DefaultClientDelegate<T>	Default IFederationDelegate implementation used by a standard client.
DefaultServiceFederationDelegate<T extends AbstractService>	Basic delegate for services that need to override the service UUID and service interface reported to the ILoadBalancerService.
DistributedTransactionService	Implementation for an IBigdataFederation supporting both single-phase commits (for transactions that execute on a single IDataService) and distributed commits.
DistributedTransactionService.SnapshotHelper	A helper class for reading and writing snapshots of the commit time index.
EmbeddedClient<T>	A client for an embedded federation (the client and the data services all run in the same process).
EmbeddedFederation<T>	An implementation that uses an embedded database rather than a distributed database.
EmbeddedMetadataService	A local (in process) metadata service.
Event	An event.
EventReceiver	Class capable of receiving Events from remote services.
EventReceiver.EventBTree	A BTree whose keys are event start times and whose values are the serialized Events.
EventReceiver.EventBTree.EventBTreeTupleSerializer	Encapsulates key and value formation for the EventReceiver.EventBTree.
EventResource	Semi-structured representation of the data service on which the event occurred, the name of the index, and the index partition identifier.
FederationCallable<T>	Base class for IFederationCallable.
HostScore	Per-host metadata and a score for that host which gets updated periodically by LoadBalancerService.UpdateTask.
IndexCache	Concrete implementation for IClientIndex views.
ListIndicesTask	Task returns an array of the named indices on the DataService to which it is submitted.
LoadBalancerService	The LoadBalancerService collects a variety of performance counters from hosts and services, identifies over- and under- utilized hosts and services based on the collected data and reports those to DataService s so that they can auto-balance, and acts as a clearing house for WARN and URGENT alerts for hosts and services.
ManagedResourceService	This class manages a pool of direct ByteBuffers.
MetadataIndexCache	Concrete implementation for IMetadataIndex views.
MetadataService	Implementation of a metadata service for a named scale-out index.
MetadataService.DropScaleOutIndexTask	Drops a scale-out index.
MetadataService.JoinIndexPartitionTask	Updates the MetadataIndex to reflect the join of 2 or more index partitions.
MetadataService.MoveIndexPartitionTask	Updates the MetadataIndex to reflect the move of an index partition.
MetadataService.NextPartitionIdTask	Task assigns the next partition identifier for a registered scale-out index in a restart-safe manner.
MetadataService.RegisterScaleOutIndexTask	Registers a metadata index for a named scale-out index and statically partition the index using the given separator keys and data services.
MetadataService.SplitIndexPartitionTask	Atomic operation removes the pre-existing entry for specified index partition and replaces it with N new entries giving the locators for the N new index partitions created when that index partition was split.
NoCacheMetadataIndexView	An implementation that performs NO caching.
ResourceService	A service which permits resources (managed files or buffers) identified by a UUID to be read by a remote service.
ResourceService.Counters	Performance counters for the ResourceService.
ResourceService.FetchResourceTask<S,T>	Client for a BufferService reads a single resource from the specified service, writing it into the local file system.
ResourceService.ReadBufferTask	Class sends a request for a remote ByteBuffer and then receives the data into a local ByteBuffer.
ResourceService.ReadResourceTask	Task sends a request for a file's data and then receives the data onto a local file.
ScaleOutClientFactory	Convenience class to break out bigdata-jini dependency.
ServiceScore	Per-service metadata and a score for that service which gets updated periodically by the LoadBalancerService.UpdateTask.
Session	A (transient) property set associated with some kinds of services.
Split	Describes a "split" of keys for a batch operation.
StressTestConcurrent	Test suite for concurrent operations on a DataService.
StressTestConcurrent.GenerateExperiment	Experiment generation utility class.
StressTestConcurrent.Task	Run an unisolated operation.
TestAll	Test suite for embedded services.
TestAll_DynamicSharding	Test suite for dynamic sharding.
TestAll_ResourceService	Test suite for ResourceService and friends.
TestBasicIndexStuff	Test of basic index operations.
TestCase3	Deprecated This is currently duplicating a class by the same name in com.bigdata.io in the JOURNAL_HA_BRANCH.
TestClientException	Class written to verify the stack trace printing behavior of ClientException.
TestDistributedTransactionService	Unit tests of local (all writes are on a single data service) and distributed abort and commit protocols for an IBigdataFederation using the DistributedTransactionService.
TestDistributedTransactionServiceRestart	Unit tests of the setReleaseTime, snapshot and restart aspects of the DistributedTransactionService (all having to do with the maintenance of the commit time index, including across restart).
TestDistributedTransactionServiceRestart.MockDistributedTransactionService
TestEDS	Delegate for ProxyTestCases for services running against an EmbeddedFederation.
TestEmbeddedClient	Test suite for the EmbeddedClient.
TestEventParser	Unit tests for parsing Events.
TestEventReceiver	Unit tests for the EventReceiver.
TestJournal	Delegate for ProxyTestCases for services running against a Journal.
TestLoadBalancerRoundRobin	Unit tests for the round robin behaviors of the load balancer in isolation (not an integration test).
TestMetadataIndex	Some unit tests for the MetadataIndex as accessed via the IMetadataService.
TestMove	Some unit tests for moving an index partition.
TestOverflow	Simple test verifies that a scale-out index is preserved across both synchronous and asynchronous overflow events.
TestOverflowGRS	Test drives inserts on the GRS index partition until the data service is forced to go through an overflow such that an index build is performed for the GRS index (rather than copying the index into the new live journal).
TestRangeQuery	Test suite for the IRangeQuery API.
TestReceiveBuffer	Test verifies the ability to transmit a file using the ResourceService.
TestReceiveFile	Test verifies the ability to transmit a file using the ResourceService.
TestRestartSafe	Test suite for the ability to re-open an EmbeddedFederation.
TestScatterSplit	Some unit tests for moving an index partition.
TestSnapshotHelper	Unit tests for DistributedTransactionService.SnapshotHelper.
TestSplitJoin	Test suite verifies that inserts eventually split an index and that deletes eventually cause the index partitions to be joined.
TxId2CommitTimeIndex	BTree whose keys are the absolute value of the txIds and whose values are ITxState0 tuples for the transaction.
TxId2CommitTimeIndex.TupleSerializer	Encapsulates key and value formation.

Enum Summary

Enum	Description
AbstractScaleOutClient.MetadataIndexCachePolicy	Policy options for caching PartitionLocators for an IMetadataIndex.
EventType	Type safe enum for Events.
ResourceService.ResourceTypeEnum	Type safe enumeration of the kinds of resources which can be served.
ResourceService.StatusEnum	Known status codes.
TxServiceRunState	Run states for the AbstractTransactionService.

Exception Summary

Exception	Description
NoSuchService	Exception thrown when a service was requested but has not been discovered or is otherwise not available.

Package com.bigdata.service Description

This package provides implementations of bigdata services (metadata service, data service, transaction manager service.

A bigdata federation is comprised of the following essential services:

metadata service

The metadata service manages scale-out indices and is used to locate index partitions.

data service

The data service supports reads and writes on index partitions.

Clients are responsible for discovering the metadata service. Clients then use the metadata service to manage scale-out indices and to locate and cache leases for data service instances having data for key range partitions that the client will read or write. If a client will use transactions (vs ACID batch operations), then the client must also locate the transaction manager service. Service location is performed using JINI, but other service locator protocols are possible.

bigdata provides the following optional services. These are not considered essential since (a) they are not required by clients that make direct use of bigdata as a distributed database; and (b) they are themselves applications of the services briefly described above.

transaction manager service

Clients use the transaction manager service to start new transactions and to coordinate the distributed commit protocol. The transaction manager also provides a centralized timestamp factory.

map and reduce service

Manages the distributed execution of a functional program.

deployment

In order to deploy a bigdata federation, the following preconditions must be met (a standalone deployment can be realized by meeting these preconditions on a single host). This will go more smoothly if JINI is running before you start the various bigdata services since they will then be able to register themselves immediately on startup. Service discovery by default uses the local network and the "public" group. If you want to run more than one bigdata federation on the same network, then you MUST edit the configuration file.

Hosts:

• java 5 or better is installed. The use of a server VM is highly recommended. The following options are recommended for some well known VMs:

  Sun

  -server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=256M

  JRockit

  -Xms1g -Xmx1g