Recent changes to feature-requests

Cannot determine type of file '.gkpStore:untrim'. Tried

wangyugui — Wed, 06 Jul 2016 06:49:57 -0000

/usr/hpc-bio/celera-8.3-svn/Linux-amd64/bin/meryl -B -C -m 31 -threads 4 -memory 737280 -s /biowrk/celera/juglans.regian/PRJNA291087.DNA/juglans.gkpStore:untrim -o /biowrk/celera/juglans.regian/PRJNA291087.DNA/9-terminator/mercy/juglans-ms31-frgFull
seqFactory::registerFile()-- Cannot determine type of file '/biowrk/celera/juglans.regian/PRJNA291087.DNA/juglans.gkpStore:untrim'. Tried:
seqFactory::registerFile()-- 'FastA'
seqFactory::registerFile()-- 'FastAstream'
seqFactory::registerFile()-- 'Fastq'
seqFactory::registerFile()-- 'FastQstream'
seqFactory::registerFile()-- 'seqStore'
seqFactory::registerFile()-- 'gkpStore'
seqFactory::registerFile()-- 'gkpStoreChain'

this error will not stop the runCA command .

should we remove this command or fix it?
I searched the source, there is only one place about "gkpStore:untrim",

#141 sort with multiple thread support to improve performance

wangyugui — Wed, 06 Jul 2016 06:46:25 -0000

the dirty fix seems work. please close this ticket.

#141 sort with multiple thread support to improve performance

wangyugui — Tue, 05 Jul 2016 11:03:48 -0000

my friend give a dirty fix. i will test it.

Index: bin/caqc.pl
--- bin/caqc.pl (revision 676)
+++ bin/caqc.pl (working copy)
@@ -1751,8 +1751,9 @@

 # sort on UID of UTG, or CCO, then begin position, then end
 # UTGs should only be unplaced surrogates

- open( BADMATES, "	sort -k2.5,2 -k4,4n -k5,5n > $badMateFile" )
+ #open( BADMATES, "	sort -k2.5,2 -k4,4n -k5,5n > $badMateFile" )
+ #		die "Couldn't write to $badMateFile.";
+ open( BADMATES, ">","$badMateFile.tmp" );
local $\ = "\n";
local $, = "\t";
while ( my ( $badId, $coord ) = each %bad_mates ) {
@@ -1764,6 +1765,8 @@
}
}
close BADMATES;
+
+ system("sort --parallel=32 -k2.5,2 -k4,4n -k5,5n $badMateFile.tmp > $badMateFile");

 exit(0);

}

#141 sort with multiple thread support to improve performance

wangyugui — Tue, 05 Jul 2016 09:25:30 -0000

and the sort in terminator will use 1279m or more time, so we need to improve the performance of it.

sort with multiple thread support to improve performance

wangyugui — Tue, 05 Jul 2016 09:22:29 -0000

sort command is used in terminator, but it need multiple thread support to improve performance.

sort command is used by 'XXXX|sort -k2.5,2 -k4,4n -k5,5n' in terminator.

'sort -k2.5,2 -k4,4n -k5,5n ref.fa >x.fa' will have the default max 8 thread support.
'sort --parallel=32 -k2.5,2 -k4,4n -k5,5n ref.fa >x.fa' will have the default 32 thread support.

but when used a pipe input, there will be no mutiple thread support in sort event with --parallel=32.
cat ref.fa |sort -k2.5,2 -k4,4n -k5,5n >x.fa ==>single thread
cat ref.fa |sort --parallel=32 -k2.5,2 -k4,4n -k5,5n >x.fa ==>single thread

it will need a lit fix only, but I can't fix perl source.

#139 runCA SLURM/DRMAA support

Sergey Koren — Mon, 16 Nov 2015 20:04:34 -0000

Several users have had success with the SLURM patch on github so I would recommend this route for now. The only change I've seen users have to make was:
Changing
setGlobal("gridEngineNameToJobIDCommand", "squeue -h -o\%F_* -n \"WAIT_TAG\" | uniq");
to
setGlobal("gridEngineNameToJobIDCommand", "squeue -h -o\%F -n \"WAIT_TAG\" | uniq");

That is, removing the * from the option.

#139 runCA SLURM/DRMAA support

Brett Whitty — Tue, 18 Aug 2015 23:38:49 -0000

Hi Sergey et al.,

Noticed a tweet about this, and was first time for me hearing about SLURM, so thought would be fun exercise to look into modifying 'runCA.pl' to support (based on your suggestions).

I've put together a patch script that I've checked into github here:

https://github.com/brettwhitty/bw-ca-tools/blob/master/runCA-slurm-patch/do_runCA_slurm_patch.sh

that adds the following SLURM-supporting variables:

if (($var eq "gridEngine") && ($val eq "SLURM")) {
+ setGlobal("gridEngineSubmitCommand", "sbatch");
+ setGlobal("gridEngineHoldOption", "--depend=afterany:\"WAIT_TAG\"");
+ setGlobal("gridEngineHoldOptionNoArray", "--depend=afterany:\"WAIT_TAG\"");
+ setGlobal("gridEngineSyncOption", ""); ## TODO: SLURM may not support w/out wrapper; See LSF bsub manpage to compare
+ setGlobal("gridEngineNameOption", "-D pwd -J");
+ setGlobal("gridEngineArrayOption", "-a ARRAY_JOBS");
+ setGlobal("gridEngineArrayName", "ARRAY_NAME[ARRAY_JOBS]");
+ setGlobal("gridEngineOutputOption", "-o"); ## NB: SLURM default joins STDERR & STDOUT if no -e specified
+ setGlobal("gridEnginePropagateCommand", "scontrol update job=\"WAIT_TAG\""); ## TODO: manually verify this in all cases
+ setGlobal("gridEngineNameToJobIDCommand", "squeue -h -o\%F_* -n \"WAIT_TAG\" | uniq"); ## TODO: manually verify this in all cases
+ setGlobal("gridEngineNameToJobIDCommandNoArray", "squeue -h -o\%i -n \"WAIT_TAG\""); ## TODO: manually verify this in all cases
+ setGlobal("gridEngineTaskID", "SLURM_ARRAY_TASK_ID");
+ setGlobal("gridEngineArraySubmitID", "%A_%a");
+ setGlobal("gridEngineJobID", "SLURM_JOB_ID");
+ }

I initially worked from the SLURM man pages; n1/s/oge is old hat for me, so I started from the SGE (and PBS/torque) examples to map over the behavior the code seemed to be expecting. Then I set up a small SLURM virtual cluster and did a few test runs.

Was away last few days and haven't gotten back to testing yet this week, but as far as I can tell it works OK.

Array submissions seem to work, holds on array jobs seem to work, output file naming seems OK.

Only thing that was a bit sketchy to me in the code is the nested variable replacement that happens with 'WAIT_TAG', especially as it relates to 'gridEnginePropagateCommand'; the thinking wasn't clear to me with the naming of that variable, but following through the code I think everything does seem to work as it should; but I haven't tested enough to understand the special cases that may be buried in those deeply nested code blocks.

Would be happy to polish this off should anyone have any feedback or errors from their own testing, but otherwise after I do a couple more tests and am satisfied I will consider it a completed exercise. Hope this is useful to a few people.

Regards,

Brett

#139 runCA SLURM/DRMAA support

lexnederbragt — Wed, 12 Aug 2015 11:22:29 -0000

I have an opportunity to apply for resources (in the form of a competent person's time) to do work towards this. Should I go ahead or has somebody already started?

#139 runCA SLURM/DRMAA support

miquelgralo — Thu, 02 Jul 2015 04:51:43 -0000

I am also interested in runCA with SLURM. Any news?
Thanks

#140 extreme disk usage issue

Sergey Koren — Mon, 18 May 2015 01:37:03 -0000

First, you can definitely erase 0-overlaptrim/*Store folders to free up space since you're done with trimming. There are potential changes you can make to decrease the space used but you would have to re-start the assembly. You can see how close to completing the overlap store building you are by checking the asm.ovlStore.err file.

The most likely cause of a large overlap store are repeats in the sequences. There was a recent question on the user group about a large overlap store for Illumina data. I'm paraphrasing most of the response below.

You can drop shorter reads. The historical minimum is 64 bases, but you can set it higher depending on your sequence lengths.

In addition to throwing out short reads, definitely increase the minimum overlap size (ovlMinLen) to whatever the length of the shortest read is, -1 (or 2 or ...).

What kmer threshold did it pick (0-mercounts, one of the *err files)? Can you send the histogram file? Plotting the first two columns should show a definite hump at the expected coverage, with a large tail. Any humps after that are repeats that probably should be excluded from seeding overlaps. Be sure to check way out on the X axis, with Y zoomed in, for any very common repeats.

Do you have a (cumulative) histogram of read lengths?

Any chance there is adapter present?

The most recent versions of CA have been optimized for PacBio (long) sequences which makes the data structures take more space for the short Illumina reads.