com.bigdata.io.writecache

Class TestWORMWriteCacheService

java.lang.Object

junit.framework.Assert

junit.framework.TestCase

junit.framework.TestCase2

com.bigdata.io.TestCase3

com.bigdata.io.writecache.TestWORMWriteCacheService

All Implemented Interfaces:

junit.framework.Test

public class TestWORMWriteCacheService
extends TestCase3

Test suite for the WriteCacheService using pure append writes.

Version:

$Id: TestWriteCacheService.java 2866 2010-05-18 18:36:35Z thompsonbry $

Author:

Bryan Thompson

Nested Class Summary

Nested classes/interfaces inherited from class junit.framework.TestCase2

TestCase2.MyProperties, TestCase2.RandomType

Field Summary

Fields

Modifier and Type	Field and Description
protected Random	r A random number generated - the seed is NOT fixed.

Fields inherited from class junit.framework.TestCase2

_randomType

Constructor Summary

Constructors

Constructor and Description
TestWORMWriteCacheService()
TestWORMWriteCacheService(String name)

Method Summary

Methods

Modifier and Type	Method and Description
protected long	doStressTest(int nbuffers, int nrecs, int maxreclen, double largeRecordRate, boolean useChecksums, boolean isHighlyAvailable, StoreTypeEnum storeType, Quorum quorum) A stress test for the WriteCacheService.
ByteBuffer	getRandomData() Returns random data that will fit in N bytes.
ByteBuffer	getRandomData(int nbytes) Returns random data that will fit in nbytes.
void	test_STRESSwriteCacheService_HA_WORM_1record_1buffer_k3_size3_reorganizePipeline() Martyn wrote:
void	test_writeCacheService_HA_RW_1buffer_k3_size2() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer and the default #of records written and the default percentage of large records.
void	test_writeCacheService_HA_RW_1record_1buffer_k3_size2() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer, and one record written.
void	test_writeCacheService_HA_RW_2buffer_k3_size2() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, two buffers and the default #of records written and the default percentage of large records.
void	test_writeCacheService_HA_RW_2buffer_k3_size3() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 3 running services, six buffers and the default #of records written and the default percentage of large records.
void	test_writeCacheService_HA_RW_7buffer_k3_size2() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, two buffers and the default #of records written and the default percentage of large records.
void	test_writeCacheService_HA_WORM_1buffer_k3_size2() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer and the default #of records written and the default percentage of large records.
void	test_writeCacheService_HA_WORM_1record_1buffer_k3_size2_reorganizePipeline() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer, and one record written where the leader must reorganize the write pipeline when it is elected.
void	test_writeCacheService_HA_WORM_1record_1buffer_k3_size2() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer, and one record written.
void	test_writeCacheService_HA_WORM_1record_1buffer_k3_size3_reorganizePipeline() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 3 running services, one buffer, and one record written where the leader must reorganize the write pipeline when it is elected.
void	test_writeCacheService_HA_WORM_2buffer_k3_size2() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, two buffers and the default #of records written and the default percentage of large records.
void	test_writeCacheService_HA_WORM_2buffer_k3_size3() A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 3 running services, six buffers and the default #of records written and the default percentage of large records.
void	test_writeCacheService_RW_1buffer() A test which looks for deadlock conditions (one buffer).
void	test_writeCacheService_RW_2buffers() A test which looks for starvation conditions (2 buffers).
void	test_writeCacheService_RW_6buffers_recordChecksums_wholeBufferChecksums() A high throughput configuration with record level checksums and whole buffer checksums.
void	test_writeCacheService_RW_6buffers_recordChecksums() A high throughput configuration with record level checksums.
void	test_writeCacheService_WORM_1buffer() A test which looks for deadlock conditions (one buffer).
void	test_writeCacheService_WORM_2buffers() A test which looks for starvation conditions (2 buffers).
void	test_writeCacheService_WORM_6buffers_recordChecksums_wholeBufferChecksums() A high throughput configuration with record level checksums and whole buffer checksums.
void	test_writeCacheService_WORM_6buffers_recordChecksums() A high throughput configuration with record level checksums.

Methods inherited from class com.bigdata.io.TestCase3

assertCondition, assertCondition, assertEquals, assertEquals, getBytes, getPort, getRandomData, setUp, tearDown

Methods inherited from class junit.framework.TestCase2

assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEqualsWithinUlps, assertSameArray, assertSameArray, assertSameBigDecimal, assertSameBigDecimal, assertSameBigInteger, assertSameBigInteger, assertSameIterator, assertSameIterator, assertSameIteratorAnyOrder, assertSameIteratorAnyOrder, assertSameValue, assertSameValue, assertZeroUlps, assertZeroUlps, fail, getInnerCause, getNormalInt, getProjectBuildPath, getProperties, getRandomObject, getRandomObject, getRandomOrder, getRandomString, getTestInputStream, getTestResource, getTestResource, getUlps, getUlps, isDEBUG, isDEBUG, isINFO, isINFO, isInnerCause, logProperties

Methods inherited from class junit.framework.TestCase

assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertFalse, assertFalse, assertNotNull, assertNotNull, assertNotSame, assertNotSame, assertNull, assertNull, assertSame, assertSame, assertTrue, assertTrue, countTestCases, createResult, fail, fail, failNotEquals, failNotSame, failSame, format, getName, run, run, runBare, runTest, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

r

protected final Random r

A random number generated - the seed is NOT fixed.

Constructor Detail

TestWORMWriteCacheService

public TestWORMWriteCacheService()

TestWORMWriteCacheService

public TestWORMWriteCacheService(String name)

Parameters:

name -

Method Detail

test_writeCacheService_WORM_1buffer

public void test_writeCacheService_WORM_1buffer()
                                         throws InterruptedException,
                                                IOException

A test which looks for deadlock conditions (one buffer).

Throws:

InterruptedException

IOException

test_writeCacheService_RW_1buffer

public void test_writeCacheService_RW_1buffer()
                                       throws InterruptedException,
                                              IOException

A test which looks for deadlock conditions (one buffer).

Throws:

InterruptedException

IOException

test_writeCacheService_WORM_2buffers

public void test_writeCacheService_WORM_2buffers()
                                          throws InterruptedException,
                                                 IOException

A test which looks for starvation conditions (2 buffers).

Throws:

InterruptedException

IOException

test_writeCacheService_RW_2buffers

public void test_writeCacheService_RW_2buffers()
                                        throws InterruptedException,
                                               IOException

A test which looks for starvation conditions (2 buffers).

Throws:

InterruptedException

IOException

test_writeCacheService_WORM_6buffers_recordChecksums

public void test_writeCacheService_WORM_6buffers_recordChecksums()
                                                          throws InterruptedException,
                                                                 IOException

A high throughput configuration with record level checksums.

Throws:

InterruptedException

IOException

test_writeCacheService_RW_6buffers_recordChecksums

public void test_writeCacheService_RW_6buffers_recordChecksums()
                                                        throws InterruptedException,
                                                               IOException

A high throughput configuration with record level checksums.

Throws:

InterruptedException

IOException

test_writeCacheService_WORM_6buffers_recordChecksums_wholeBufferChecksums

public void test_writeCacheService_WORM_6buffers_recordChecksums_wholeBufferChecksums()
                                                                               throws InterruptedException,
                                                                                      IOException

A high throughput configuration with record level checksums and whole buffer checksums.

Throws:

InterruptedException

IOException

test_writeCacheService_RW_6buffers_recordChecksums_wholeBufferChecksums

public void test_writeCacheService_RW_6buffers_recordChecksums_wholeBufferChecksums()
                                                                             throws InterruptedException,
                                                                                    IOException

A high throughput configuration with record level checksums and whole buffer checksums.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_WORM_1record_1buffer_k3_size2

public void test_writeCacheService_HA_WORM_1record_1buffer_k3_size2()
                                                             throws InterruptedException,
                                                                    IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer, and one record written.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_WORM_1record_1buffer_k3_size2_reorganizePipeline

public void test_writeCacheService_HA_WORM_1record_1buffer_k3_size2_reorganizePipeline()
                                                                                throws InterruptedException,
                                                                                       IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer, and one record written where the leader must reorganize the write pipeline when it is elected.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_WORM_1record_1buffer_k3_size3_reorganizePipeline

public void test_writeCacheService_HA_WORM_1record_1buffer_k3_size3_reorganizePipeline()
                                                                                throws InterruptedException,
                                                                                       IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 3 running services, one buffer, and one record written where the leader must reorganize the write pipeline when it is elected.

Throws:

InterruptedException

IOException

test_STRESSwriteCacheService_HA_WORM_1record_1buffer_k3_size3_reorganizePipeline

public void test_STRESSwriteCacheService_HA_WORM_1record_1buffer_k3_size3_reorganizePipeline()
                                                                                      throws InterruptedException,
                                                                                             IOException

Martyn wrote:

 The simplest way to recreate the deadlock issue is to add a stress test
 to TestWORMWriteCacheService [this method].
 
 ...
 
 This will create a deadlock after a failure in serviceJoin spewed from
 the pipeline reorganization:
 
 com.bigdata.quorum.QuorumException: Not a pipeline member : 3b680846-30dc-4013-bc15-47fd1c4b20a5
      at com.bigdata.quorum.AbstractQuorum$QuorumActorBase.serviceJoin(AbstractQuorum.java:1251)
      at com.bigdata.quorum.AbstractQuorum$QuorumWatcherBase$3.run(AbstractQuorum.java:2249)
      at com.bigdata.quorum.AbstractQuorum$QuorumWatcherBase$1.run(AbstractQuorum.java:1927)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:680)
 
 As you summarized it appears to be due to an awaited condition which
 allows an interleaving of the lock such that a concurrent request to
 serviceJoin gets processed after the service had been removed from the
 pipeline prior to adding at the end.  Naively increasing lock protection
 results in immediate deadlock.
 
 I have on occasion seen other failures relating to malformed quorums,
 but I suspect this is more of the same.
 

Throws:

InterruptedException

IOException

test_writeCacheService_HA_RW_1record_1buffer_k3_size2

public void test_writeCacheService_HA_RW_1record_1buffer_k3_size2()
                                                           throws InterruptedException,
                                                                  IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer, and one record written.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_WORM_1buffer_k3_size2

public void test_writeCacheService_HA_WORM_1buffer_k3_size2()
                                                     throws InterruptedException,
                                                            IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer and the default #of records written and the default percentage of large records.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_RW_1buffer_k3_size2

public void test_writeCacheService_HA_RW_1buffer_k3_size2()
                                                   throws InterruptedException,
                                                          IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, one buffer and the default #of records written and the default percentage of large records.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_WORM_2buffer_k3_size2

public void test_writeCacheService_HA_WORM_2buffer_k3_size2()
                                                     throws InterruptedException,
                                                            IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, two buffers and the default #of records written and the default percentage of large records.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_RW_2buffer_k3_size2

public void test_writeCacheService_HA_RW_2buffer_k3_size2()
                                                   throws InterruptedException,
                                                          IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, two buffers and the default #of records written and the default percentage of large records.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_RW_7buffer_k3_size2

public void test_writeCacheService_HA_RW_7buffer_k3_size2()
                                                   throws InterruptedException,
                                                          IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 2 running services, two buffers and the default #of records written and the default percentage of large records.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_WORM_2buffer_k3_size3

public void test_writeCacheService_HA_WORM_2buffer_k3_size3()
                                                     throws InterruptedException,
                                                            IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 3 running services, six buffers and the default #of records written and the default percentage of large records.

Throws:

InterruptedException

IOException

test_writeCacheService_HA_RW_2buffer_k3_size3

public void test_writeCacheService_HA_RW_2buffer_k3_size3()
                                                   throws InterruptedException,
                                                          IOException

A test of the write pipeline driving from the WriteCacheService of the leader using a quorum with k := 3, 3 running services, six buffers and the default #of records written and the default percentage of large records.

Throws:

InterruptedException

IOException

doStressTest

protected long doStressTest(int nbuffers,
                int nrecs,
                int maxreclen,
                double largeRecordRate,
                boolean useChecksums,
                boolean isHighlyAvailable,
                StoreTypeEnum storeType,
                Quorum quorum)
                     throws InterruptedException,
                            IOException

A stress test for the WriteCacheService.

Parameters:

nbuffers - The #of WriteCache buffers.

nrecs - The #of records to write.

maxreclen - The maximum length of a record.

largeRecordRate - The rate in [0:1) of records which will be larger than the WriteCache buffer size.

useChecksums - When true, use record-level checksums.

isHighlyAvailable - When true, compute the running checksums of the WriteCache as a whole.

Returns:

The #of write cache blocks send by the quorum leader to the first downstream follower.

Throws:

InterruptedException

IOException - FIXME Test with service leave/joins when the quorum is highly available. To do this we will have to pass in the quorum[]. A leader leave causes the quorum to break and is not really testable here since we do not have any infrastructure to handle this, but we can test when a follower leaves the quorum and when it joins. [However, handling a follower join requires us to synchronize the follower first and we can't really test that here either, so all we can really test would be a follower leave, either when the follower did not did not have a downstream node. If there is downstream node, then the upstream node from the left follower should reconfigure for the new downstream node and retransmit the current cache block and the event should be otherwise unnoticed.]

getRandomData

public ByteBuffer getRandomData()

Returns random data that will fit in N bytes. N is chosen randomly in 1:256.

Returns:

A new ByteBuffer wrapping a new byte[] of random length and having random contents.

getRandomData

public ByteBuffer getRandomData(int nbytes)

Returns random data that will fit in nbytes.

Overrides:

getRandomData in class TestCase3

Returns:

A new ByteBuffer wrapping a new byte[] having random contents.