Ah, the joys of bioinformatics. I just received an email from Mox indicating that the Masurca assembly I started 11 DAYS AGO (!!) crashed.
I’m probably not going to put much effort in to trying to figure out what went wrong, but here’s some log file snippets for reference. I’ll probably drop a line to the developers and see if they have any easy ways to address whatever caused the problems, but that’s about as much effort as I’m willing to put into troubleshooting this assembly.
Additionally, since this crashed, I’m not going to bother moving any of the files off of Mox. That means they will be deleted automatically by the system around Nov. 9th, 2017.
compute_psa 6601202 2632582819 Refining alignments Joining Generating assembly input files Coverage of the mega-reads less than 5 -- using the super reads as well Coverage threshold for splitting unitigs is 138 minimum ovl 63 Running assembly /gscratch/srlab/programs/MaSuRCA-3.2.3/bin/deduplicate_unitigs.sh: line 85: 24330 Aborted (core dumped) overlapStoreBuild -o $ASM_DIR/$ASM_PREFIX.ovlStore -M 65536 -g $ASM_DIR/$ASM_PREFIX.gkpStore $ASM_DIR/overlaps_dedup.ovb.gz > $ASM_DIR/overlapStore.rebuild.err 2>&1 Assembly stopped or failed, see CA.mr.220.127.116.11.029.log [Mon Oct 30 23:19:37 PDT 2017] Assembly stopped or failed, see CA.mr.18.104.22.168.029.log
number of threads = 28 (OpenMP default) ERROR: overlapStore '/gscratch/scrubbed/samwhite/20171019_masurca_oly_assembly/CA.mr.22.214.171.124.029/genome.ovlStore' is incomplete; previous overlapStoreBuild probably crashed. ---------------------------------------- Failure message: failed to unitig
Scanning overlap files to count the number of overlaps. Found 277.972 million overlaps. Memory limit 65536MB supplied. Ill put 3246167525 IIDs (3435.97 million overlaps) into each of 1 buckets. bucketizing CA.mr.126.96.36.199.029/overlaps_dedup.ovb.gz bucketizing DONE! overlaps skipped: 0 OBT - low quality 0 DUP - non-duplicate overlap 0 DUP - different library 0 DUP - dedup not requested terminate called after throwing an instance of std::bad_alloc what(): std::bad_alloc Failed with Aborted Backtrace (mangled): overlapStoreBuild[0x40523a] /usr/lib64/libpthread.so.0(+0xf100)[0x2af83b3c0100] /usr/lib64/libc.so.6(gsignal+0x37)[0x2af83c0395f7] /usr/lib64/libc.so.6(abort+0x148)[0x2af83c03ace8] /usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x2af83b62d9d5] /usr/lib64/libstdc++.so.6(+0x5e946)[0x2af83b62b946] /usr/lib64/libstdc++.so.6(+0x5e973)[0x2af83b62b973] /usr/lib64/libstdc++.so.6(+0x5eb93)[0x2af83b62bb93] /usr/lib64/libstdc++.so.6(_Znwm+0x7d)[0x2af83b62c12d] /usr/lib64/libstdc++.so.6(_Znam+0x9)[0x2af83b62c1c9] overlapStoreBuild[0x402e10] /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2af83c025b15] overlapStoreBuild[0x403089] Backtrace (demangled):  overlapStoreBuild() [0x40523a]  /usr/lib64/libpthread.so.0::(null) + 0xf100 [0x2af83b3c0100]  /usr/lib64/libc.so.6::(null) + 0x37 [0x2af83c0395f7]  /usr/lib64/libc.so.6::(null) + 0x148 [0x2af83c03ace8]  /usr/lib64/libstdc++.so.6::__gnu_cxx::__verbose_terminate_handler() + 0x165 [0x2af83b62d9d5]  /usr/lib64/libstdc++.so.6::(null) + 0x5e946 [0x2af83b62b946]  /usr/lib64/libstdc++.so.6::(null) + 0x5e973 [0x2af83b62b973]  /usr/lib64/libstdc++.so.6::(null) + 0x5eb93 [0x2af83b62bb93]  /usr/lib64/libstdc++.so.6::operator new(unsigned long) + 0x7d [0x2af83b62c12d]  /usr/lib64/libstdc++.so.6::operator new(unsigned long) + 0x9 [0x2af83b62c1c9]  overlapStoreBuild() [0x402e10]  /usr/lib64/libc.so.6::(null) + 0xf5 [0x2af83c025b15]  overlapStoreBuild() [0x403089] GDB: