QuorumPipelineImpl (Blazegraph Database Platform 2.1.5 API)

java.lang.Object
- com.bigdata.ha.QuorumPipelineImpl<S>

Type Parameters:
S -

All Implemented Interfaces:

QuorumPipeline<S>, QuorumStateChangeListener
```
public abstract class QuorumPipelineImpl<S extends HAPipelineGlue>
extends Object
implements QuorumPipeline<S>, QuorumStateChangeListener
```
QuorumPipeline implementation.
The QuorumMember must pass along the "pipeline" messages, including:
When a quorum is met, the leader is always first in the write pipeline since it is the node which receives writes from clients. When a service joins the write pipeline, it always does so at the end of the chain. Services may enter the write pipeline before joining a quorum in order to synchronize with the quorum. If a service in the middle of the chain leaves the pipeline, then the upstream node will reconfigure and retransmit the current cache block to its new downstream node. This prevent nodes which are "bouncing" during synchronization from causing write sets to be discarded. However, if the leader leaves the write pipeline, then the quorum is broken and the write set will be discarded.
Since the write pipeline is used to synchronize services trying to join the quorum as well as the replicate writes for services joined with the quorum, HAReceiveService may be live for a met quorum even though the QuorumMember on whose behalf this class is acting is not joined with the met quorum.
Pipeline maintenance
There are three broad categories which have to be handled: (1) leader leaves; (2) pipeline leader election; and (3) follower leaves. A leader leave causes the quorum to break, which will cause service leaves and pipeline leaves for all joined services. However, services must add themselves to the pipeline before they join the quorum and the pipeline will be reorganized if necessary when the quorum leader is elected. This will result in a pipelineElectedLeader() event. A follower leave only causes the follower to leave the pipeline and results in a pipelineChange(UUID, UUID) event.
There are two cases for a follower leave: (A) when the follower did not did not have a downstream node; and (B) when there is downstream node. For (B), the upstream node from the left follower should reconfigure for the new downstream node and retransmit the current cache block and the event should be otherwise unnoticed.
Handling a follower join requires us to synchronize the follower first which requires some more infrastructure and should be done as part of the HA synchronization test suite.
What follows is an example of how events will arrive for a quorum of three services: A, B, and C.
```
 A.getActor().pipelineAdd() => A.pipelineAdd()
 B.getActor().pipelineAdd() => B.pipelineAdd(); A.pipelineChange(null,B); 
 C.getActor().pipelineAdd() => C.pipelineAdd(); B.pipelineChange(null,C);
 
```
At this point the pipeline order is [A,B,C]. Notice that the HASendService for A is not established until the A.pipelineChange(null,B) sets B as the new downstream service for A. Likewise, B will not relay to C until it handles the B.pipelineChange(null,C) event.
Given the pipeline order [A,B,C], if B were to leave, then the events would be:
```
 B.getActor().pipelineRemove() => B.pipelineRemove(); A.pipelineChange(B,C);
 
```
and when this class handles the A.pipelineChange(B,C) event, it must update the HAReceiveService such that it now relays data to C.
On the other hand, given the pipeline order [A,B,C], if C were to leave the events would be:
```
 C.getActor().pipelineRemove() => C.pipelineRemove(); B.pipelineChange(C,null);
 
```
and when this class handles the B.pipelineChange(C,null) event, it must update the C's HAReceiveService such that it continues to receive data, but no longer relays data to a downstream service.
Author:

Bryan Thompson

See Also:
HAJournalServer deadlock: pipelineRemove() and getLeaderId()

Constructor Summary

Constructors
Constructor and Description

QuorumPipelineImpl(QuorumMember<S> member)

Constructors
Constructor and Description
`QuorumPipelineImpl(QuorumMember<S> member)`

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`consensus(long lastCommitTime)` Invoked when a consensus has been achieved among `(k+1)/2` services concerning a shared lastCommitTime (really, this is not a consensus but a simple majority).
`protected void`	`finalize()` Extended to invoke `#tearDown()` in order to guarantee the eventual release of the `receiveBuffer` and the shutdown of the `sendService` or `receiveService`.
`protected long`	`getRetrySendTimeoutNanos()` Once this timeout is elapsed, retrySend() will fail.
`protected abstract void`	`handleReplicatedWrite(IHASyncRequest req, IHAWriteMessage msg, ByteBuffer data)` Core implementation handles the message and payload when received on a service.
`protected abstract void`	`incReceive(IHASyncRequest req, IHAWriteMessage msg, int nreads, int rdlen, int rem)` Notify that some payload bytes have been incrementally received for an `IHAMessage`.
`void`	`lostConsensus()` Invoked when the consensus is lost.
`void`	`memberAdd()` Invoked when this service is added as a quorum member.
`void`	`memberRemove()` Invoked when this service is removed as a quorum member.
`void`	`pipelineAdd()` Invoked when this service is added to the write pipeline.
`void`	`pipelineChange(UUID oldDownStreamId, UUID newDownStreamId)` Invoked for this service when the downstream service in the write pipeline has changed.
`void`	`pipelineElectedLeader()` Invoked for this service when the service is already in the pipeline and this service becomes the first service in the write pipeline because all previous services in the pipeline order have been removed from the pipeline (failover into the leader position).
`void`	`pipelineRemove()` Invoked when this service is removed from the write pipeline.
`void`	`pipelineUpstreamChange()` Invoked for this service when the upstream service in the write pipeline has been removed.
`void`	`processEvents()` Called from ErrorTask in HAJournalServer to ensure that events are processed before entering SeekConsensus.
`void`	`quorumBreak()` Invoked when a quorum breaks.
`void`	`quorumMeet(long token, UUID leaderId)` Invoked when a quorum meets.
`Future<Void>`	`receiveAndReplicate(IHASyncRequest req, IHASendState snd, IHAWriteMessage msg)` Return a `Future` for a task which will replicate an NIO buffer along the write pipeline.
`Future<Void>`	`replicate(IHASyncRequest req, IHAWriteMessage msg, ByteBuffer b)` Return a `Future` for a task which will replicate an NIO buffer along the write pipeline.
`Future<IHAPipelineResetResponse>`	`resetPipeline(IHAPipelineResetRequest req)` Reset the pipeline (blocking).
`void`	`serviceJoin()` Invoked when this service joins the quorum.
`void`	`serviceLeave()` Invoked when this service leaves the quorum.

Methods inherited from class java.lang.Object
clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.bigdata.ha.QuorumPipeline
getLastCommitCounter, getLastCommitTime, getStoreUUID, logRootBlock, logWriteCacheBlock, purgeHALogs

- Constructor Detail
  - QuorumPipelineImpl
```
public QuorumPipelineImpl(QuorumMember<S> member)
```
- Method Detail
  - getRetrySendTimeoutNanos
```
protected long getRetrySendTimeoutNanos()
```
    Once this timeout is elapsed, retrySend() will fail.
    Note: This gets overridden in the ZK aware version and set to a constant greater than the then-current negotiated timeout for the client.
  - finalize
```
protected void finalize()
                 throws Throwable
```
    Extended to invoke #tearDown() in order to guarantee the eventual release of the receiveBuffer and the shutdown of the sendService or receiveService.
    
    Overrides:
    
    finalize in class Object
    
    Throws:
    
    Throwable
  - pipelineAdd
```
public void pipelineAdd()
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked when this service is added to the write pipeline. The service always enters at the of the pipeline.
    
    Specified by:
    
    pipelineAdd in interface QuorumStateChangeListener
  - pipelineElectedLeader
```
public void pipelineElectedLeader()
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked for this service when the service is already in the pipeline and this service becomes the first service in the write pipeline because all previous services in the pipeline order have been removed from the pipeline (failover into the leader position).
    
    Specified by:
    
    pipelineElectedLeader in interface QuorumStateChangeListener
  - pipelineRemove
```
public void pipelineRemove()
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked when this service is removed from the write pipeline.
    
    Specified by:
    
    pipelineRemove in interface QuorumStateChangeListener
  - pipelineChange
```
public void pipelineChange(UUID oldDownStreamId,
                  UUID newDownStreamId)
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked for this service when the downstream service in the write pipeline has changed. Services always enter at the end of the write pipeline, but may be removed at any position in the write pipeline.
    
    Specified by:
    
    pipelineChange in interface QuorumStateChangeListener
  - pipelineUpstreamChange
```
public void pipelineUpstreamChange()
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked for this service when the upstream service in the write pipeline has been removed. This hook provides an opportunity for this service to close out its connection with the old upstream service and to prepare to establish a new connection with the new downstream service.
    
    Specified by:
    
    pipelineUpstreamChange in interface QuorumStateChangeListener
  - memberAdd
```
public void memberAdd()
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked when this service is added as a quorum member.
    
    Specified by:
    
    memberAdd in interface QuorumStateChangeListener
  - memberRemove
```
public void memberRemove()
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked when this service is removed as a quorum member.
    
    Specified by:
    
    memberRemove in interface QuorumStateChangeListener
  - consensus
```
public void consensus(long lastCommitTime)
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked when a consensus has been achieved among (k+1)/2 services concerning a shared lastCommitTime (really, this is not a consensus but a simple majority). This message is sent to each member service regardless of whether or not they participated in the consensus.
    Once a consensus has been reached, each QuorumMember which agrees on that lastCommitTime MUST do a QuorumStateChangeListener.serviceJoin() before the quorum will meet. The first quorum member to do a service join will be elected the leader. The remaining services to do a service join will be elected followers.
    
    Specified by:
    
    consensus in interface QuorumStateChangeListener
    
    Parameters:
    lastCommitTime - The last commit time around which a consensus was established.
    See Also:
    QuorumStateChangeListener.serviceJoin(), #electedLeader(long), #electedFollower(long), QuorumStateChangeListener.lostConsensus()
  - lostConsensus
```
public void lostConsensus()
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked when the consensus is lost. Services do not withdraw their cast votes until a quorum breaks and a new consensus needs to be established. This message is sent to each member service regardless of whether or not they participated in the consensus.
    
    Specified by:
    
    lostConsensus in interface QuorumStateChangeListener
    
    See Also:
    QuorumStateChangeListener.consensus(long)
  - serviceJoin
```
public void serviceJoin()
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked when this service joins the quorum.
    
    Specified by:
    
    serviceJoin in interface QuorumStateChangeListener
  - serviceLeave
```
public void serviceLeave()
```
    Description copied from interface: QuorumStateChangeListener
    
    Invoked when this service leaves the quorum.
    
    Specified by:
    
    serviceLeave in interface QuorumStateChangeListener
  - quorumMeet
```
public void quorumMeet(long token,
              UUID leaderId)
```
    Description copied from interface: QuorumStateChangeListener
    Invoked when a quorum meets. The state of the met quorum can be queried using the token. Quorum members can use this to decide whether they are the leader (using #isLeader(long), joined as a follower (using #isFollower(long)), or do not participate in the quorum (this message is sent to all quorum members, so this service might not be part of the met qourum).
    The following pre-conditions will be satisfied before this message is sent to the QuorumMember:
    - There will be at least (k+1)/2 services which have voted for the same lastCommitTime.
    - There will be at (k+1)/2 services joined with the quorum. The join order will be the same as the Quorum.getVotes() for the services which voted for the lastCommitTime
    When control returns from this method, the following post-conditions should be true:
    
    The service should be prepared to accept reads.
    
    If the service was elected as the quorum leader, then it should be prepared to accept writes.
    
    If the QuorumMember is joined with the quorum but it can not satisfy these post-conditions, then it must leave the Quorum.
    Specified by:
    
    quorumMeet in interface QuorumStateChangeListener
    
    Parameters:
    token - The newly assigned quorum token.
    leaderId - The UUID of the service which was elected to be the quorum leader. This information is only valid for the scope of the accompanying quorum token. (The leaderId may be obtained from #getLeader(long) at any time for a met quorum.)
  quorumBreak
  
  public void quorumBreak()
  
  Description copied from interface: QuorumStateChangeListener
  
  Invoked when a quorum breaks. The service MUST handle handle this event by (a) doing an abort() which will any buffered writes and reload their current root block; and (b) casting a vote for their current commit time. Once a consensus is reached on the current commit time, services will be joined in the vote order, a new leader will be elected, and the quorum will meet again. This message is sent to all member services, regardless of whether they were joined with the met quorum.
  
  Specified by:
  
  quorumBreak in interface QuorumStateChangeListener
  
  resetPipeline
  
  public Future<IHAPipelineResetResponse> resetPipeline(IHAPipelineResetRequest req) throws IOException
  
  Description copied from interface: QuorumPipeline
  
  Reset the pipeline (blocking). This message is used to handle an error in pipeline replication. If replication fails, the socket connections both upstream and downstream of the point of failure can be left in an indeterminate state with partially buffered data. In order to bring the pipeline back into a known state (without forcing a quorum break) we message each service in the pipeline to reset its HAReceiveService (including the inner HASendService). The next message and payload relayed from the leader will cause new socket connections to be established.
  
  Specified by:
  
  resetPipeline in interface QuorumPipeline<S extends HAPipelineGlue>
  
  Returns:
  The Future for the operation on the local service.
  
  Throws:
  
  IOException
  
  replicate
  
  public Future<Void> replicate(IHASyncRequest req, IHAWriteMessage msg, ByteBuffer b) throws IOException
  
  Description copied from interface: QuorumPipeline
  
  Return a Future for a task which will replicate an NIO buffer along the write pipeline. This method is only invoked by the quorum leader. The payload is replicated to the first follower in the write pipeline. That follower will accept the payload (and replicate it if necessary) using #receiveAndReplicate(IHAWriteMessage).
  Note: The implementation of this method should be robust to changes in the write pipeline. Specifically, if a follower leaves the write pipeline, it should attempt to retransmit the message and the payload while allowing time for the write pipeline to be reconfigured in response to the related QuorumMember events.
  
  Specified by:
  
  replicate in interface QuorumPipeline<S extends HAPipelineGlue>
  
  Parameters:
  req - A synchronization request (optional). This is only non-null when historical WriteCache blocks are being replayed down the write pipeline in order to synchronize a service.
  msg - The RMI metadata about the payload.
  b - The payload. The bytes from the position to the limit will be transmitted (note that the #of bytes remaining in the buffer MUST agree with IHAWriteMessageBase.getSize()).
  
  Throws:
  
  IOException
  
  receiveAndReplicate
  
  public Future<Void> receiveAndReplicate(IHASyncRequest req, IHASendState snd, IHAWriteMessage msg) throws IOException
  
  Description copied from interface: QuorumPipeline
  
  Return a Future for a task which will replicate an NIO buffer along the write pipeline. This method is invoked for any node except the master, including the last node in the failover chain.
  
  Specified by:
  
  receiveAndReplicate in interface QuorumPipeline<S extends HAPipelineGlue>
  
  Parameters:
  req - A synchronization request (optional). This is only non-null when historical WriteCache blocks are being replayed down the write pipeline in order to synchronize a service.
  snd - Metadata about the state of the sender and potentially the routing of the payload along the write replication pipeline.
  msg - The RMI metadata about the payload.
  
  Throws:
  
  IOException
  
  handleReplicatedWrite
  
  protected abstract void handleReplicatedWrite(IHASyncRequest req, IHAWriteMessage msg, ByteBuffer data) throws Exception
  
  Core implementation handles the message and payload when received on a service.
  Note: Replication of the message and payload is handled by the caller. The implementation of this method is NOT responsible for replication.
  
  Parameters:
  req - The synchronization request (optional). When non- null the msg and payload are historical data. When null they are live data.
  msg - Metadata about a buffer containing data replicated to this node.
  data - The buffer containing the data.
  
  Throws:
  
  Exception
  
  incReceive
  
  protected abstract void incReceive(IHASyncRequest req, IHAWriteMessage msg, int nreads, int rdlen, int rem) throws Exception
  
  Notify that some payload bytes have been incrementally received for an IHAMessage.
  
  Parameters:
  msg - The message.
  nreads - The number of reads performed against the upstream socket for this message.
  rdlen - The number of bytes read from the socket in this read.
  rem - The number of bytes remaining before the payload has been fully read.
  
  Throws:
  
  Exception
  
  processEvents
  
  public void processEvents()
  
  Called from ErrorTask in HAJournalServer to ensure that events are processed before entering SeekConsensus.
  
  See Also:
  HAJournalServer reports "follower" but is in SeekConsensus and is not participating in commits

Class QuorumPipelineImpl<S extends HAPipelineGlue>

Pipeline maintenance

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface com.bigdata.ha.QuorumPipeline

Constructor Detail

QuorumPipelineImpl

Method Detail

getRetrySendTimeoutNanos

finalize

pipelineAdd

pipelineElectedLeader

pipelineRemove

pipelineChange

pipelineUpstreamChange

memberAdd

memberRemove

consensus

lostConsensus

serviceJoin

serviceLeave

quorumMeet

quorumBreak

resetPipeline

replicate

receiveAndReplicate

handleReplicatedWrite

incReceive

processEvents