Recent changes to data-generator

IntegerGenerator incorrectly calculates the values available

Michael — Wed, 16 Nov 2011 05:30:09 -0000

In the validateRanges() method in IntegerGenerator, part of the method checks to see if there are enough values it is able to produce given the range, interval and whether looping is allowed. I.e. for a range of [-2, 2] and interval of 1, there are 5 unique values it is able to produce (-2, -1, 0, 1, 2) which, if looping is not allowed, means the most values(rows) it can produce is 5.
The method incorrectly adds 1 to the range (i.e. endRange-startRange) which when given an interval of 1, it incorrectly reports the number of values it can produce to be greater (by 1) than what it should

To test this I modified a the test case testIntegerSeqNoLoopBackwards() in IntegerGeneratorTest and set the values to produce when instantiating the IntegerGenerator to be 4 (it was set to 5 because it is using a range of [-2, 2]). This should have caused the test to fail when running ant clean test but instead it still passed

IntegerGeneratorTest overflows values passed to methods

Michael — Tue, 25 Oct 2011 03:11:30 -0000

In some of the test cases, methods are called that will take Integers as arguments but will be passed the value Integer.MAX_VALUE * 2 which is an integer overflow. Since these methods aren't being tested for overflow handling, it is unnecessary.

i.e. in test case testAllDefault()
gen.advance(0, Integer.MAX_VALUE * 2);

DG breaks the primaray key constraint for random generation

Michael — Sun, 16 Oct 2011 18:56:39 -0000

Tested in IntegerGenerator.java
When you specify DG to generate random values for a field that is also a primary key, DG will not produce unique values to be inserted and causes the commit batch to fail. It is not documented whether random generation on primary key fields should be allowed or not. If it shouldn't be allowed, DG should print an error message when parsing and shut down gracefully.

Example table:
<table>
<attr name="name" value="inter.test"/>
<attr name="row_count" value="100"/>

Output:
2076 [main] INFO com.ibm.dg.DataGenerator - Generating data for 'inter.test' table.
This table will have 100 row(s).
Threads: 4, with around 25 row(s) per thread.
Exception in thread "Thread-2" java.lang.RuntimeException: Thread failed to execute batch, stopping: SQLException 0
Code: -4229
SqlState: null
Error Message: [jcc][t4][102][10040][3.62.56] Batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null
SQLException 1
Code: -803
SqlState: 23505
Error Message: Error for batch element #10: DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505, SQLERRMC=1;INTER.TEST, DRIVER=3.62.56
SQLException 2
Code: -803
SqlState: 23505
Error Message: Error for batch element #17: DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505, SQLERRMC=1;INTER.TEST, DRIVER=3.62.56

at com.ibm.dg.threads.TableThread.commitBatch(TableThread.java:331)
at com.ibm.dg.threads.TableThread.run(TableThread.java:251)
at java.lang.Thread.run(Thread.java:636)
Exception in thread "Thread-3" java.lang.RuntimeException: Thread failed to execute batch, stopping: SQLException 0
Code: -4229
SqlState: null
Error Message: [jcc][t4][102][10040][3.62.56] Batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null
SQLException 1
Code: -803
SqlState: 23505
Error Message: Error for batch element #1: DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505, SQLERRMC=1;INTER.TEST, DRIVER=3.62.56

at com.ibm.dg.threads.TableThread.commitBatch(TableThread.java:331)
at com.ibm.dg.threads.TableThread.run(TableThread.java:251)
at java.lang.Thread.run(Thread.java:636)
Comm. (stmts) Comm. (size) Commit Rate (stmts/s) Commit Rate (size/s)
======
1 4.0 B -1 null
1 4.0 B -1 null
1 4.0 B -1 null
1 4.0 B -1 null
1 4.0 B -1 null

[junit] Testcase: testIntegerSeqLoopTwo took 0.008 sec

Nikita — Sun, 09 Oct 2011 00:55:59 -0000

One of the test cases is failing for the Integer data generator -- the specific interval is [-2..5] where the proper sequence does not get generated. Integer generator logic needs some patching:

[junit] Testcase: testIntegerSeqLoopTwo took 0.008 sec
[junit] FAILED
[junit] Sequential generator returned wrong value. expected:<-[4]> but was:<-[5]>
[junit] junit.framework.ComparisonFailure: Sequential generator returned wrong value. expected:<-[4]> but was:<-[5]>
[junit] at com.ibm.dg.generators.IntegerGeneratorTest.testIntegerSeqLoopTwo(IntegerGeneratorTest.java:197)

Statistics do not print properly when run w/ multple threads

Michael — Sat, 08 Oct 2011 15:44:22 -0000

Tested with datasource_threads and generator_threads configured with 1 and 4 threads in config.xml and received similar output:
6605 [main] INFO com.ibm.dg.DataGenerator - Generating data for 'Movie.Address' table.
This table will have 10000 row(s).
Threads: 4, with around 2500 row(s) per thread.
Comm. (stmts) Comm. (size) Commit Rate (stmts/s) Commit Rate (size/s)
======
10,000 450.3 KB 5,000 225.2 KB
10,000 450.3 KB 5,000 225.2 KB
16916 [main] INFO com.ibm.dg.DataGenerator - Generating data for 'Movie.Address' table completed in: 00:00:10

ran with -q switch
db2inst1@ubuntu-school:~/DG-test/DG/dist$ java -jar DG.jar -q 1 -d ./data/schemas/movie-merged.xml

6802 [main] INFO com.ibm.dg.DataGenerator - Generating data for 'Movie.Address' table.
This table will have 10000 row(s).
Threads: 1, with around 10000 row(s) per thread.
Comm. (stmts) Comm. (size) Commit Rate (stmts/s) Commit Rate (size/s)
======
400 18.0 KB 80 3.6 KB
809 36.4 KB 80 3.6 KB
1,212 54.6 KB 80 3.6 KB
1,611 72.5 KB 80 3.6 KB
2,004 90.2 KB 80 3.6 KB
2,394 107.7 KB 79 3.6 KB
2,797 125.9 KB 79 3.6 KB
3,205 144.3 KB 80 3.6 KB
3,610 162.5 KB 80 3.6 KB
4,016 180.8 KB 80 3.6 KB

Comm. (stmts) Comm. (size) Commit Rate (stmts/s) Commit Rate (size/s)

4,424 199.2 KB 80 3.6 KB
4,830 217.5 KB 80 3.6 KB
5,235 235.8 KB 80 3.6 KB
5,646 254.2 KB 80 3.6 KB
6,051 272.4 KB 80 3.6 KB
6,455 290.7 KB 80 3.6 KB
6,863 309.0 KB 80 3.6 KB
7,271 327.4 KB 80 3.6 KB
7,631 343.6 KB 80 3.6 KB
8,039 361.9 KB 80 3.6 KB

Comm. (stmts) Comm. (size) Commit Rate (stmts/s) Commit Rate (size/s)

8,449 380.4 KB 80 3.6 KB
8,857 398.7 KB 80 3.6 KB
9,265 417.2 KB 80 3.6 KB
9,665 435.2 KB 80 3.6 KB
10,000 450.3 KB 80 3.6 KB
10,000 450.3 KB 80 3.6 KB
137059 [main] INFO com.ibm.dg.DataGenerator - Generating data for 'Movie.Address' table completed in: 00:02:10

DG goes into infinite loop. Sometimes throwing an exception

Michael — Sat, 08 Oct 2011 13:37:40 -0000

When DG is running on the attached schema XML file, when generating the movie.movie_info table it will go into an infinite loop. Sometimes it throws an exception but I have been unable to reproduce it again. Most of the time it doesn't display any errors at all.

If I get it to throw an exception I will post that as well.

121338 [main] INFO com.ibm.dg.DataGenerator - Generating data for 'movie.movie_info' table.
This table will have 1000 row(s).
Threads: 4, with around 250 row(s) per thread.
Comm. (stmts) Comm. (size) Commit Rate (stmts/s) Commit Rate (size/s)
======
4 531.0 B -1 null
4 531.0 B -1 null
4 531.0 B -1 null
4 531.0 B -1 null
4 531.0 B -1 null
4 531.0 B -1 null
4 531.0 B -1 null
4 531.0 B -1 null
^C166354 [Thread-0] WARN com.ibm.dg.DataGenerator - Shutting down.

While this was running I checked how many records were actually in the table and it was only 4, no matter how long DG was running for.

DG goes into an infinite loop when a commit batch fails

Michael — Sat, 08 Oct 2011 10:35:51 -0000

The batch of commit statements failed because:
AN INSERTED OR UPDATED VALUE IS INVALID BECAUSE THE INDEX IN INDEX SPACE indexspace-name CONSTRAINS COLUMNS OF THE TABLE SO NO TWO ROWS CAN CONTAIN DUPLICATE VALUES IN THOSE COLUMNS. RID OF EXISTING ROW IS X record-id

DG continued to run and looked like it was still committing rows. Difficult to catch since if a user walks away from their computer while DG is running, they won't see the error message printed.

63996 [main] INFO com.ibm.dg.DataGenerator - Generating data for 'Movie.people_relationship' table.
This table will have 1000 row(s).
Threads: 4, with around 250 row(s) per thread.
Exception in thread "Thread-24" java.lang.RuntimeException: Thread failed to execute batch, stopping: SQLException 0
Code: -4229
SqlState: null
Error Message: [jcc][t4][102][10040][3.62.56] Batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null
SQLException 1
Code: -803
SqlState: 23505
Error Message: Error for batch element #1: DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505, SQLERRMC=1;MOVIE.PEOPLE_RELATIONSHIP, DRIVER=3.62.56

at com.ibm.dg.threads.TableThread.commitBatch(TableThread.java:331)
at com.ibm.dg.threads.TableThread.run(TableThread.java:251)
at java.lang.Thread.run(Thread.java:636)
Comm. (stmts) Comm. (size) Commit Rate (stmts/s) Commit Rate (size/s)
======
250 2.9 KB -1 null
250 2.9 KB -1 null
250 2.9 KB -1 null
250 2.9 KB -1 null
250 2.9 KB -1 null
250 2.9 KB -1 null
250 2.9 KB -1 null

Exception when end_range is specified but start_range is not

Michael — Sat, 08 Oct 2011 02:59:02 -0000

When specifying the attribute end_range for an integer generator, if start_range is missing then DG throws a NullPointerException. If this is expected behaviour (i.e. start_range and end_range must be specified together) then it should be noted in the documentation since it lists both of these fields as optional. Although the documentation says that if the start_range isn't specified it will be set as default to 1, so start_range shouldn't be required in order to use end_range based on this.

This will cause DG to throw the exception if it's in the xml:
<field>
<attr name="name" value="director_id"/>
<attr name="type" value="INTEGER"/>
<attr name="constraint" value="NOT NULL"/>
<data>
<attr name="type" value="integer"/>
<attr name="seq" value="false"/>
<attr name="end_range" value="7000"/>
</data>

Set custom table dependancies

Michael — Fri, 07 Oct 2011 23:40:44 -0000

It would be nice to have an option for tables to be able to specify dependencies of other tables to have better control of the order of creation of the tables. In the case where you use a SQL generator, if the SQL is pulling values from a table that DG is being created then that table will need to be created first
something like <attr name="depends" value="x,y,x"> under the <options> element

Bad example of an SQL generator in the manual

Michael — Fri, 07 Oct 2011 23:27:09 -0000

This example is given in the documentation as an example of a slow generator. This particular SQL will continuously pull the same name to use as a value to insert, while it should probably pull a random value from the table instead

<field name= "name" type= "VARCHAR(128)" >
<data type= "SQL" >
<attr name= "sql" >
SELECT name FROM db2inst1.names FETCH FIRST 1 ROW ONLY
</attr>
</data>