ERROR: GBK file conversion failed, tbl2asn parallel script has died

[Thatguy027]

Hello again ;_;

I ran into another issue

Thanks in advance!

Are you using the latest release?
v1.8.8

Describe the bug
A little unsure about this

output error:

ERROR: GBK file conversion failed, tbl2asn parallel script has died

Tried running the command outside of the funannotate predict workflow and the initial issue was a core dump. I requested more memory and got around the core dump issue

From predict log files:

[05/24/21 15:02:27]: Found 541 gene models to remove: 4 too short; 0 span gaps; 537 transposable elements
[05/24/21 15:02:27]: 21,702 gene models remaining
[05/24/21 15:02:27]: Predicting tRNAs
[05/24/21 15:02:28]: bedtools intersect -sorted -v -a /u/project/kruglyak/thatguy0/genomics/rnaseq/trinity_out_dir_genome_assisted/funannotate_run/fun_predict/predict_misc/trnascan.gff3.sorted.gff3 -b /u/project/kruglyak/thatguy0/genomics/rnaseq/trinity_out_dir_genome_assisted/funannotate_run/fun_predict/predict_misc/evm.cleaned.gff3.sorted.gff3 /u/project/kruglyak/thatguy0/genomics/rnaseq/trinity_out_dir_genome_assisted/funannotate_run/fun_predict/predict_misc/assembly-gaps.bed.sorted.gff3
[05/24/21 15:02:29]: 435 tRNAscan models are valid (non-overlapping)
[05/24/21 15:02:29]: Generating GenBank tbl annotation file
[05/24/21 15:02:44]: Converting to final Genbank format
[05/24/21 15:02:44]: /u/project/kruglyak/thatguy0/conda/envs/funannotate/bin/python /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/aux_scripts/tbl2asn_parallel.py -i ./predict_misc/tbl2asn/genome.tbl -f /u/project/kruglyak/thatguy0/genomics/rnaseq/trinity_out_dir_genome_assisted/funannotate_run/fun_predict/predict_misc/genome.softmasked.fa -o ./predict_misc/tbl2asn --sbt /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/config/test.sbt -d ./predict_results/caenorhabditis_XZ1516.discrepency.report.txt -s caenorhabditis -t -l paired-ends -v 1 -c 1 --isolate XZ1516
[05/24/21 15:02:48]: ERROR: GBK file conversion failed, tbl2asn parallel script has died

What command did you issue?
Same as the issue from earlier today, but with --keep_evm flag:

funannotate predict -i ../XZ1516_renamed.fasta \
	--species "caenorhabditis" \
	--isolate XZ1516 \
	--transcript_evidence ../fun_XZ1516/training/trinity.fasta.clean \
	--rna_bam ../Aligned.sortedByCoord.out.bam \
	--busco_db nematoda \
	--cpus 8 \
	--organism other \
	--busco_seed_species caenorhabditis \
	-o . \
	--keep_evm

This fails as outlined above, but I can successfully run the line causing the issue outside of the predict workflow:

/u/project/kruglyak/thatguy0/conda/envs/funannotate/bin/python /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/aux_scripts/tbl2asn_parallel.py -i ./predict_misc/tbl2asn/genome.tbl -f /u/project/kruglyak/thatguy0/genomics/rnaseq/trinity_out_dir_genome_assisted/funannotate_run/fun_predict/predict_misc/genome.softmasked.fa -o ./predict_misc/tbl2asn --sbt /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/config/test.sbt -d ./predict_results/caenorhabditis_XZ1516.discrepency.report.txt -s caenorhabditis -t -l paired-ends -v 1 -c 1 --isolate XZ1516

A couple of things about that line:

1. I'm unsure about the sbt file, this appears to be a default --sbt /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/config/test.sbt
2. the .val and .gbf files in the predict_misc/tbl2as are both empty

Logfiles
logfiles.zip

OS/Install Information

-------------------------------------------------------
Checking dependencies for 1.8.8
-------------------------------------------------------
You are running Python v 3.7.10. Now checking python packages...
biopython: 1.78
goatools: 1.0.15
matplotlib: 3.4.2
natsort: 7.1.1
numpy: 1.20.2
pandas: 1.2.4
psutil: 5.8.0
requests: 2.25.1
scikit-learn: 0.24.2
scipy: 1.6.3
seaborn: 0.11.1
All 11 python packages installed


You are running Perl v b'5.026002'. Now checking perl modules...
Bio::Perl: 1.007002
Carp: 1.38
Clone: 0.42
DBD::SQLite: 1.64
DBD::mysql: 4.046
DBI: 1.642
DB_File: 1.855
Data::Dumper: 2.173
File::Basename: 2.85
File::Which: 1.23
Getopt::Long: 2.5
Hash::Merge: 0.302
JSON: 4.02
LWP::UserAgent: 6.39
Logger::Simple: 2.0
POSIX: 1.76
Parallel::ForkManager: 2.02
Pod::Usage: 1.69
Scalar::Util::Numeric: 0.40
Storable: 3.15
Text::Soundex: 3.05
Thread::Queue: 3.13
Tie::File: 1.02
URI::Escape: 3.31
YAML: 1.30
threads: 2.21
threads::shared: 1.56
All 27 Perl modules installed


Checking Environmental Variables...
$FUNANNOTATE_DB=/u/project/kruglyak/thatguy0/funannotate_db
$PASAHOME=/u/project/kruglyak/thatguy0/conda/envs/funannotate/opt/pasa-2.4.1
$TRINITY_HOME=/u/project/kruglyak/thatguy0/conda/envs/funannotate/opt/trinity-2.8.5
$EVM_HOME=/u/project/kruglyak/thatguy0/conda/envs/funannotate/opt/evidencemodeler-1.1.1
$AUGUSTUS_CONFIG_PATH=/u/project/kruglyak/thatguy0/conda/envs/funannotate/config/
$GENEMARK_PATH=/u/project/kruglyak/thatguy0/bin/gmes_linux_64/
All 6 environmental variables are set
-------------------------------------------------------
Checking external dependencies...
	ERROR: signalp found but error running signalp
PASA: 2.4.1
CodingQuarry: 2.0
Trinity: 2.8.5
augustus: 3.3.3
bamtools: bamtools 2.5.1
bedtools: bedtools v2.30.0
blat: BLAT v36
diamond: 2.0.8
emapper.py: 2.1.2
ete3: 3.1.2
exonerate: exonerate 2.4.0
fasta: no way to determine
glimmerhmm: 3.0.4
gmap: 2017-11-15
gmes_petap.pl: 4.65_lic
hisat2: 2.2.1
hmmscan: HMMER 3.3.2 (Nov 2020)
hmmsearch: HMMER 3.3.2 (Nov 2020)
java: 11.0.1
kallisto: 0.46.1
mafft: v7.475 (2020/Nov/23)
makeblastdb: makeblastdb 2.2.31+
minimap2: 2.18-r1015
proteinortho: 6.0.30
pslCDnaFilter: no way to determine
salmon: salmon 0.14.1
samtools: samtools 1.10
snap: 2006-07-28
stringtie: 2.1.5
tRNAscan-SE: 2.0.7 (Oct 2020)
tantan: tantan 26
tbl2asn: no way to determine, likely 25.X
tblastn: tblastn 2.2.31+
trimal: trimAl v1.4.rev15 build[2013-12-17]
trimmomatic: 0.39
	ERROR: signalp not installed

[nextgenusfs]

So sounds like a memory issue then? Maybe I misunderstood, but seemed to suggest you could manually run this following command and it executed successfully?:

/u/project/kruglyak/thatguy0/conda/envs/funannotate/bin/python /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/aux_scripts/tbl2asn_parallel.py \
    -i ./predict_misc/tbl2asn/genome.tbl \
    -f /u/project/kruglyak/thatguy0/genomics/rnaseq/trinity_out_dir_genome_assisted/funannotate_run/fun_predict/predict_misc/genome.softmasked.fa \
    -o ./predict_misc/tbl2asn \
    --sbt /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/config/test.sbt \
    -d ./predict_results/caenorhabditis_XZ1516.discrepency.report.txt \
    -s caenorhabditis \
    -t -l paired-ends -v 1 \
    -c 1 --isolate XZ1516

How you have this written is it will use 1 cpu, ie -c 1, which apparently then is working? But getting perhaps a malloc error when it was run in funannotate predict -c 8. The point of this wrapper was to speed up tbl2asn, I have no idea how much memory it uses, but the wrapper is launching parallel processes by chunking the input.

You could try to wrap the command in /usr/bin/time -v to get the peak memory used to see if it is requesting too much. I guess you must not have swap enabled that it would die?

[Thatguy027]

It was a memory issue at first, but I think I got around that issue (I probably shouldn't have mentioned it, sorry for the confusion)

To clarify, funannotate predict is failing at the tbl2asn step. I mentioned the core dump because that was the first issue I ran into when trying to run the last attempted log command outside of the predict workflow.

The command you pasted technically works, in that there are no errors, but the .val and .gbf files are empty and so is the discrepancy report.

When trying to complete the pipeline, I changed the funannotate predict --cpus to 1, but didn't fix anything.

If it is useful, I think I have some issues with GLIBC in the environment because tbl2asn throws an error. Not sure how useful this is because GLIBC is included in the funannotate conda install from what i can tell

(funannotate) [thatguy0@n6459 fun_predict]$ tbl2asn
tbl2asn: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by tbl2asn)
(funannotate) [thatguy0@n6459 fun_predict]$ /u/project/kruglyak/thatguy0/conda/envs/funannotate/bin/python /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/aux_scripts/tbl2asn_parallel.py \
> -i ./predict_misc/tbl2asn/genome.tbl \
> -f /u/project/kruglyak/thatguy0/genomics/rnaseq/XZ1516_GZ_B5_A8/trinity_out_dir_genome_assisted/funannotate_run/fun_predict/predict_misc/genome.softmasked.fa \
> -o ./predict_misc/tbl2asn \
> --sbt /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/config/test.sbt \
> -d ./predict_results/caenorhabditis_XZ1516.discrepency.report.txt -s caenorhabditis -t TBL2ASN -v 1 -c 1 --isolate XZ1516
(funannotate) [thatguy0@n6459 fun_predict]$ 

[nextgenusfs]

Okay, I guess I would try to re-install tbl2asn or maybe even download the binary from NCBI and remove the one from bioconda for example.

[hyphaltip]

yes exactly - we use a system installed one not bioconda one.

[nextgenusfs]

You can "force" remove dependencies with conda with the --force flag (I think):

conda remove -n funannotate tbl2asn --force

That should just remove the tbl2asn from that environment and then add the system installed one into your PATH.

[Thatguy027]

Just tried all of your suggestions with no luck.

A couple of questions to try to get around this issue:

1. what are the expected outputs when tbl2asn is done?
2. what is the tbl2asn command that is actually being run (outside of the wrapper), maybe i can try seeing what error that throws?

Thanks again

[nextgenusfs]

In its simplest form it would be something like:

tbl2asn -t /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/config/test.sbt \
   -M n -V b -c f -a r10u -p predict_misc/tbl2asn

the tbl2asn folder should contain genome.fsa (genome) and genome.tbl (the annotation).

[Thatguy027]

Ah, then it sounds like tbl2asn is working fine because those files were generated...

[nextgenusfs]

Sorry I wasn't clear - the genome and annotation files are the input. You can essentially run the command I entered manually and it should hopefully yield an interpretable error.

Output files will be generated in that same folder, ie the genome.gbf would be the genbank flatfile. But there will be several other submission files generated.

[Thatguy027]

Annoyingly works like a charm:

(funannotate) [thatguy0@n6459 fun_predict]$ tbl2asn -t /u/project/kruglyak/thatguy0/conda/envs/funannotate/lib/python3.7/site-packages/funannotate/config/test.sbt \
>    -M n -V b -c f -a r10u -p predict_misc/tbl2asn
[tbl2asn] Flatfile genome

[tbl2asn] Validating genome
(funannotate) [thatguy0@n6459 fun_predict]$ 

So I think the issue is GLIBC because I needed to load the module from my cluster's install to make tbl2asn work. However, when the module is loaded Loading genome assembly and parsing soft-masked repetitive sequences fails. I had some issues with GLIBC originally when installing dependencies, but thought I solved the issue...seems like I didn't?

Does tbl2asn need to be run within the predict workflow or can the output i generated with the line above be read in when I try to relaunch (testing this now)?

[nextgenusfs]

I guess if it works interactively but not when you submit job to cluster than sounds like your GLIBC error is related to order in which modules are loaded or something associated with activating that environment. One of the last steps is to write the GBK output files (ie tbl2asn), so after that step the scripts reformat a few things and move files around in the proper places so they can be parsed properly by funannotate update or funannotate annotate, etc. Ideally would be great if it worked as intended.

[Thatguy027]

I don't load any modules from the cluster apart from glibc for this particular step.

Agreed, would be awesome if it worked in my hands ;_;

I am looking at the predict.py script and it's SO close to finishing. Please correct me if I'm wrong, but this seems like all that my run is missing at the moment:

shutil.copyfile(os.path.join(gag3dir, 'genome.gbf'), final_gbk)
shutil.copyfile(os.path.join(gag3dir, 'genome.tbl'), final_tbl)
shutil.copyfile(os.path.join(gag3dir, 'genome.val'), final_validation)
shutil.copyfile(os.path.join(gag3dir, 'errorsummary.val'), final_error)
lib.tbl2allout(final_tbl, MaskGenome, final_gff, final_proteins,
               final_transcripts, final_cds_transcripts, final_fasta)
lib.annotation_summary(MaskGenome, final_stats, tbl=final_tbl,
                       transcripts=Transcripts, proteins=Exonerate,
                       database=FUNDB, command=' '.join(sys.argv),
                       organism=organism_name)
total = lib.countGFFgenes(final_gff)    lib.log.info("Collecting final annotation files for {:,} total gene models".format(total))
lib.log.info("Funannotate predict is finished, output files are in the %s/predict_results folder" % (args.out))

This is probably out of the scope of an issue page, but is it possible for you to let me know which files to move over the results folder so I can utilize your downstream utilities? This is what is in the misc folder (all not listed, just since EVM output):

(funannotate) [thatguy0@n6459 fun_predict]$ ls -lat predict_misc/
total 1247716
drwxr-xr-x   4 thatguy0 kruglyak      4096 May 25 19:26 tbl2asn
drwxr-xr-x   5 thatguy0 kruglyak      4096 May 25 18:57 ..
drwxr-xr-x  11 thatguy0 kruglyak     12288 May 25 00:57 .
-rw-r--r--   1 thatguy0 kruglyak    121981 May 25 00:57 trnascan.no-overlaps.gff3
-rw-r--r--   1 thatguy0 kruglyak         0 May 25 00:57 assembly-gaps.bed.sorted.gff3
-rw-r--r--   1 thatguy0 kruglyak  31268809 May 25 00:57 evm.cleaned.gff3.sorted.gff3
-rw-r--r--   1 thatguy0 kruglyak    210770 May 25 00:57 trnascan.gff3.sorted.gff3
-rw-r--r--   1 thatguy0 kruglyak  31290510 May 25 00:57 evm.cleaned.gff3
-rw-r--r--   1 thatguy0 kruglyak    628883 May 25 00:57 bad_models.gff
-rw-r--r--   1 thatguy0 kruglyak    333377 May 25 00:57 genome.repeats.to.remove.gff
-rw-r--r--   1 thatguy0 kruglyak  33529503 May 25 00:57 evm.round1.gff3.sorted.gff
-rw-r--r--   1 thatguy0 kruglyak   2850230 May 25 00:57 repeatmasker.bed.sorted.bed
-rw-r--r--   1 thatguy0 kruglyak     33539 May 25 00:57 repeat.gene.models.txt
-rw-r--r--   1 thatguy0 kruglyak  17443804 May 25 00:57 repeats.xml
-rw-r--r--   1 thatguy0 kruglyak  10019816 May 25 00:56 evm.round1.proteins.fa
-rw-r--r--   1 thatguy0 kruglyak       185 May 25 00:56 weights.evm.txt
-rw-r--r--   1 thatguy0 kruglyak 122591758 May 25 00:56 gene_predictions.gff3
-rw-r--r--   1 thatguy0 kruglyak  23728480 May 25 00:56 augustus.evm.gff3
-rw-r--r--   1 thatguy0 kruglyak  24381370 May 25 00:56 augustus.evm.gff3.bak
-rw-r--r--   1 thatguy0 kruglyak  29363196 May 25 00:56 genemark.evm.gff3
-rw-r--r--   1 thatguy0 kruglyak  29363196 May 25 00:56 genemark.evm.gff3.bak
-rw-r--r--   1 thatguy0 kruglyak  29363196 May 25 00:56 genemark.temp.gff
-rw-r--r--   1 thatguy0 kruglyak  20369699 May 25 00:56 hints.ALL.gff
-rw-r--r--   1 thatguy0 kruglyak  27568706 May 25 00:56 hints.all.sort.tmp
-rw-r--r--   1 thatguy0 kruglyak  27568706 May 25 00:55 hints.all.tmp
-rw-r--r--   1 thatguy0 kruglyak   5754265 May 25 00:55 hints.P.gff
-rw-r--r--   1 thatguy0 kruglyak 210339823 May 25 00:55 proteins.combined.fa
-rw-r--r--   1 thatguy0 kruglyak 107111469 May 25 00:55 genome.softmasked.fa
-rw-r--r--   1 thatguy0 kruglyak         0 May 25 00:55 assembly-gaps.bed
-rw-r--r--   1 thatguy0 kruglyak   2850230 May 25 00:55 repeatmasker.bed
drwxr-xr-x   2 thatguy0 kruglyak      4096 May 24 21:53 1
-rw-r--r--   1 thatguy0 kruglyak  33551746 May 24 20:47 evm.round1.gff3
drwxr-xr-x 132 thatguy0 kruglyak     16384 May 24 20:11 EVM
-rw-r--r--   1 thatguy0 kruglyak    210786 May 24 20:02 trnascan.gff3
-rw-r--r--   1 thatguy0 kruglyak     38488 May 24 20:02 tRNAscan.len-filtered.out
-rw-r--r--   1 thatguy0 kruglyak     38548 May 24 20:02 tRNAscan.out

[nextgenusfs]

I can tell you where stuff should go -- but the problem is that this same step is at the end of update/annotate scripts as well. I could modify the code so it doesn't completely fail at this step, ie tbl2asn fails but it is kind of an important step as that is what generates all of the data for NCBI submission.

I wonder if it is the genome size, if you run funannotate test -t predict --cpus X does that also fail at the same step?

[nextgenusfs]

Hi @Thatguy027, I reorganized the last few steps there so all files should be in proper place before tbl2asn dies. I also added a step to try to run tbl2asn single threaded as a backup, I'm not sure if this will solve your problem or not. But would be great if you could test and provide some feedback, update with pip:

python -m pip install git+https://github.com/nextgenusfs/funannotate.git --upgrade --no-deps --force