Recent changes to support-requests

#32 PBCR blasr failure

Seigfried — Tue, 01 Nov 2016 02:43:54 -0000

Hello Sergey

'This looks like an internal blasr error so restarting won’t likely fix it (the match went out of the index bounds).'
Does this mean that this particular step 1-overlapper is unable to be completed if I restart it again after it fails?

I cannot use Canu since I am using a Hybrid assembly with only around 20x pacbio reads and 150x Illumina reads.
Also I cannot use the latest version of blasr because of the recent changes and I had to downgrade to version that actually works with pbcr
./blasr --version
Note : prior to Blasr version 5.1 , use ./blasr -version (single dash) incompatible with pbcr 8.3

I understand that pbcr is not being maintained. Can you possibly advise me on how to best complete a hybrid assembly OR how I can use the latest blasr with this pbcr because that would probably solve all my problems

Thanks!

PBCR blasr failure

Seigfried — Mon, 31 Oct 2016 05:45:38 -0000

Hello

I have been trying to run PBCR on my PacBio data. The pipeline was running smoothly and made 4 sam files as per my specifications in pacbio.spec

1.sam 2.sam 3.sam 4.sam got converted into 1.ovls 2.ovls 3.ovls 4.ovls
Currently my PBCR pipeline is running and forming 5.sam 6.sam 7.sam and 8.sam

However I see that blasr for 6.sam has failed

Running partition 000002 with options -h 659897171-660029876 -r 425206338-659897170 --hashstrings 132706 --hashdatalen 1000002482 start 425206337 end 659897170 total 234690833 zero job 1 and stride 2
Running partition 000001 with options -h 659897171-660029876 -r 1-425206337 --hashstrings 132706 --hashdatalen 1000002482 start 0 end 425206337 total 425206337 zero job 0 and stride 2
Running partition 000003 with options -h 660029877-660162937 -r 1-425206337 --hashstrings 133061 --hashdatalen 1000008291 start 0 end 425206337 total 425206337 zero job 0 and stride 2
Running partition 000004 with options -h 660029877-660162937 -r 425206338-659897170 --hashstrings 133061 --hashdatalen 1000008291 start 425206337 end 659897170 total 234690833 zero job 1 and stride 2
Blasr completed.
Blasr completed.
SamToCA conversion completed.
/home/iivr/Desktop/PBCR//tempec_pacbio/1-overlapper/overlap.sh 5
Running partition 000005 with options -h 660162938-660298518 -r 1-425206337 --hashstrings 135581 --hashdatalen 1000000244 start 0 end 425206337 total 425206337 zero job 0 and stride 2
SamToCA conversion completed.
/home/iivr/Desktop/PBCR//tempec_pacbio/1-overlapper/overlap.sh 6
Running partition 000006 with options -h 660162938-660298518 -r 425206338-659897170 --hashstrings 135581 --hashdatalen 1000000244 start 425206337 end 659897170 total 234690833 zero job 1 and stride 2
Blasr completed.
Blasr failed.
[INFO] 2016-10-30T02:36:04 [blasr] started.
blasr: ../../lib/cpp/alignment/algorithms/anchoring/MapBySuffixArrayImpl.hpp:303: int MapReadToGenome(T_RefSequence&, T_SuffixArray&, T_Sequence&, unsigned int, std::vector<T_MatchPos, std::allocator<_Tp2=""> >&, AnchorParameters&) [with T_SuffixArray = SuffixArray<unsigned char,="" std::vector<int,="" std::allocator<int=""> >, DefaultCompareStrings<unsigned char="">, DNATuple>, T_RefSequence = FASTASequence, T_Sequence = SMRTSequence, T_MatchPos = ChainedMatchPos]: Assertion `sa.index[mp] + matchLength[matchIndex] <= reference.length' failed.
Command terminated by signal 6
564420.20user 36035.95system 9:15:09elapsed 1802%CPU (0avgtext+0avgdata 10329280maxresident)k
9933936inputs+158959392outputs (3637major+4045822067minor)pagefaults 0swaps
/home/iivr/Desktop/PBCR//tempec_pacbio/1-overlapper/overlap.sh 7
Running partition 000007 with options -h 660298519-660438109 -r 1-425206337 --hashstrings 139591 --hashdatalen 1000002830 start 0 end 425206337 total 425206337 zero job 0 and stride 2
Blasr completed.

Right now PBCR is still running and I don't want to interrupt it. Is there any way that I can restart my failed blasr?

How to configure on a grid system

Bright — Wed, 24 Aug 2016 15:36:17 -0000

Hi!

I had PBcR successfully with demo data. However, when running with bigger data, it is very slow and it seemed only run in one node, so I am doubt whether I am right in configuration. Thus, I look for help here, any suggestion would be grateful!

The command was "PBcR -length 500 -l pbcr -s pacbio.SGE.spec -fastq data.fastq genomeSize=48000000, and when "qstat", I can find the submit job, so it shouldn't run in local model, am I right?
The pacbio.SGE.spec was copied from demo, it has paramters begin with sge, such as sgeScript, sgeFragmentCorrection, sgeOverlapCorrection. However, in the website, they were gridOptionsScript, gridOptionsFragmentCorrection, gridOptionsOverlapCorrection now, did they the same?
Most important, I want to know how to configure these parameters. There are three node with my SGE, two have 64 cpus and one has 32 cpus. Besides, all have 512G memory.

a) I therefor set ovlMemory to 460 that less than 512, should I set it to 460*3? What about ovlStoreMemory?

b) It is said "For other jobs which run as a single multi-core job (Correction, Script) you can set them to utilize a high-memory machine available on your cluster". I think gridOptionsCorrection, gridOptionsScript, gridOptionsFragmentCorrection, gridOptionsOverlapCorrection could use only one node, so I set "orte 32" that consistent with the node with minimal cpus, and "-l mf=460g", am I right?

c) It is said "Make sure that whatever you request in gridOptions for Mhap/Overlap matches what you set for ovlMemory and ovlThreads". I keep the original value of grid (16), so I set ovlThreads to 16, and ovlMemory less than maximal memory (460G). Besides, since "-pe orte" was 16, cnsConcurrency was calculated with (64+64+32)/16, and "-l mf" was calculated with (460*3)/cnsConcurrency=138. I am a idiot here?

d) It is said "tc" could limit concurrently job. I had added tc=10 in pacbio.SGE.spec or "PBcR -tc 20", but all output error, what should I do?

Best wishes!

cat pacbio.SGE.spec 
merSize=14

# grid info
useGrid = 1
scriptOnGrid = 1
frgCorrOnGrid = 1
ovlCorrOnGrid = 1

ovlMemory = 460
ovlThreads = 16
ovlStoreMemory = 460
threads = 16
cnsConcurrency = 10

gridOptionsCorrection=-pe orte 32 -l mf=460g
gridOptionsOverlap=-pe orte 16 -l mf=138g
gridOptionsMhap=-pe orte 16 -l mf=138g
gridOptionsConsensus=-pe orte 16
gridOptionsScript=-pe orte 32 -l mf=460g
gridOptionsFragmentCorrection=-pe orte 32 -l mf=460g
gridOptionsOverlapCorrection=-pe orte 32 -l mf=460g

#30 runCorrection failed as `GOMP_4.0' not found

Bright — Sun, 07 Aug 2016 12:19:58 -0000

I had solved this problem using the parameter "-V" (sed -i 's#qsub#qsub -V' PBcR). I want to close this tick, but I don't know how to close.

#30 runCorrection failed as `GOMP_4.0' not found

Bright — Sun, 07 Aug 2016 09:44:27 -0000

This problm occur when I submit job to SGE, run in local wouldn't make trouble. It seemed when submitting to SGE, the LD_LIBRARY_PATH doesn't work?

runCorrection failed as `GOMP_4.0' not found

Bright — Sun, 07 Aug 2016 07:32:11 -0000

Hi!

When I run PBcR with demo in SGE, it failed in the step of correction. According to log, it was caused by using old gcc library. Although I had solve the same problem by re-make softlink (https://sourceforge.net/p/wgs-assembler/support-requests/29/), I can't use the same method as I don't have root priviledge. Also, I had add my library path to LD_LIBRARY_PATH, but it didn't work.

Any suggestion would be grateful!

Following is what I do:

Running PBcR
PBcR -threads 20 -length 500 -partitions 200 -l lambdaIll -s pacbio.SGE.spec -fastq pacbio.filtered_subreads.fastq genomeSize=50000 illumina.frg >log/pbcr.log 2>log/pbcr.err
Tail log in templambdaIll/runPBcR.sge.out.05
Error: found output from runCorrection. Stopping to avoid infinite loops. To try again, remove asm.layout.*
Log in templambdaIll/asm.layout.err
/tools/wgs-8.3rc2/Linux-amd64/bin/correctPacBio: /usr/lib64/libgomp.so.1: version `GOMP_4.0' not found (required by /tools/wgs-8.3rc2/Linux-amd64/bin/correctPacBio)
Check with ldd
ldd /tools/wgs-8.3rc2/Linux-amd64/bin/correctPacBio
linux-vdso.so.1 => (0x00007fff663ff000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000332f200000)
libz.so.1 => /tools/zlib/destDir/lib/libz.so.1 (0x00007ffe6f8c3000)
libstdc++.so.6 => /tools/gcc/destDir/lib64/libstdc++.so.6 (0x00007ffe6f535000)
libm.so.6 => /lib64/libm.so.6 (0x000000332ea00000)
libgomp.so.1 => /tools/gcc/destDir/lib64/libgomp.so.1 (0x00007ffe6f313000)
libgcc_s.so.1 => /tools/gcc/destDir/lib64/libgcc_s.so.1 (0x00007ffe6f0fd000)
libc.so.6 => /lib64/libc.so.6 (0x000000332e600000)
/lib64/ld-linux-x86-64.so.2 (0x000000332e200000)
librt.so.1 => /lib64/librt.so.1 (0x000000332fa00000)
libdl.so.2 => /lib64/libdl.so.2 (0x000000332ee00000)
ldd /usr/lib64/libgomp.so.1
linux-vdso.so.1 => (0x00007fff25fff000)
librt.so.1 => /lib64/librt.so.1 (0x000000332fa00000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000332f200000)
libc.so.6 => /lib64/libc.so.6 (0x000000332e600000)
/lib64/ld-linux-x86-64.so.2 (0x000000332e200000)
I had add the library path to LD_LIBRARY_PATH, but it didn't work
echo $LD_LIBRARY_PATH
/tools/boost/destDir/stage/lib:/tools/zlib/destDir/lib:/tools/gcc/destDir/lib64:/tools/gcc/gmp/destDir/lib:/tools/gcc/mpfr/destDir/lib:/tools/gcc/mpc/destDir/lib:/opt/gridengine/lib/lx26-amd64:/opt/openmpi/lib:/lib

SamToCA conversion failed

Bright — Sun, 07 Aug 2016 06:02:42 -0000

Hi, I had install PBcR with souce (wgs-8.3rc2.tar.bz2), but it failed at the step of "convertSamToCA". According to the log, it seemed the smrtanalysis is using old gcc library. I then changed to the dirctory and re-link the *so file. Although this worked for me, I wonder whether there are other methods?

Any suggestion would be grateful!

Fowllowing is what had I do:

Install gcc and exprot to PATH and LD_LIBRARY_PATH
export PATH=/tools/gcc/destDir/bin:$PATH
export LD_LIBRARY_PATH=/tools/gcc/destDir/lib64:/tools/gcc/gmp/destDir/lib:$LD_LIBRARY_PATH
Install PBcR
bzip2 -dc wgs-8.3rc2.tar.bz2 | tar -xvf -
cd wgs-8.3rc2/kmer
make install >makeInstall.log 2>&1
cd ../cd src
make >make.log 2>&1
Install SMRT-Analysis
SMRT_ROOT=/data7/lcy/zhongxm/tools/smrtanalysis
SMRT_USER=lcy01
SMRT_GROUP=mobile
mkdir $SMRT_ROOT
bash smrtanalysis_2.3.0.140936.run -p smrtanalysis-patch_2.3.0.140936.p4.run --rootdir $SMRT_ROOT
Running PBcR
PBcR -threads 20 -length 500 -partitions 200 -l lambdaIll -s pacbio.SGE.spec -fastq pacbio.filtered_subreads.fastq genomeSize=50000 illumina.frg >log/pbcr.log 2>log/pbcr.err
The log
SamToCA conversion failed
/tools/wgs-8.3rc2/Linux-amd64/bin/convertSamToCA: /tools/smrtanalysis/current/analysis/lib/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /tools/wgs-8.3rc2/Linux-amd64/bin/convertSamToCA)

#28 use Celera Assembler with corrected PacBio reads

medhat — Mon, 18 Apr 2016 13:55:26 -0000

Thanks a lot for answering,
According to what you said;

First I need to do:

java -jar convertFastaAndQualToFastq.jar corrected_pacbio.fasta > corrected_pacbio.fastq

to have my corrected pacbio reads in fastq from fasta

second I need to use fastqToCA :

fastqToCA -libraryname MycorrectedPacBIo -technology pacbio-corrected -reads corrected_pacbio.fastq > ca.frg

Thanks for help

#28 use Celera Assembler with corrected PacBio reads

medhat — Sat, 16 Apr 2016 08:39:16 -0000

fastqToCA -l PacBio -s corrected_pacBio_seq.fasta > new.frg

There is no need to -q "Fasta file of quality values" cause I do not have it.

use Celera Assembler with corrected PacBio reads

medhat — Fri, 15 Apr 2016 12:44:46 -0000

Can we use Celera Assembler with precorrected pacbio reads :
runCA -d directory -p prefix corrected_pacBio.fasta