Diamond (?) error

[nhartwic]

Attempting to do a prediction run. I'm currently installing funannotate 1.8.1 using conda and then exeucting a script that looks like...

funannotate setup -i uniprot -d fun_db/
funannotate setup -i repeats -d fun_db/
funannotate setup -i busco_outgroups -d fun_db/

funannotate predict  -i sp9504v2.softmasked.fasta \
    -o sp9504v2 --species sp9504v2 -d $(pwd)/fun_db/ \
    --busco_db $(pwd)/liliopsida_odb10 --organism other --repeats2evm \
    --name Sp9504 --augustus_species maize \
    --busco_seed_species maize --max_intronlen 10000 \
    --cpus 16

I seem to be getting an error during diamond execution or immediately after. I've attached part of the stdout from the run that includes the error and the diamond call: fun_error.txt. I'm happy to provide more information as needed.

Anyone have any ideas what is going wrong here?

[hyphaltip]

This looks like there are no protein alignments. Diamond would have complained if your diamond protein DB was borked. It seems like you are using the default which should be uniprot and surely some proteins are aligning.

Can you check what the BUSCO stats are? Eg /data1/snakemake/sp9504v2/predict_misc/busco/run_*/short_summary_*

maybe see what the files sizes are for the p2g.diamond.out in the predict_misc folder as well. Perhaps a general
ls -l of the predict_misc folder would help us to see filesizes etc.

[nhartwic]

This contains a recursive listing of the predict_misc directory at the time when the job failed: job_Ik14AZqLqp29.predict_misc.txt

The portion you are interested in is probably...

/data1/snakemake/p2g_1201:
total 209M
drwxr-xr-x 2 root root 4.0K Oct  6 20:50 scaffolds
drwxr-xr-x 2 root root 4.0K Oct  6 20:50 failed
-rw-r--r-- 1 root root 209M Oct  6 20:50 diamond.dmnd
-rw-r--r-- 1 root root    0 Oct  6 20:50 diamond.matches.tab

/data1/snakemake/p2g_1201/scaffolds:
total 0

/data1/snakemake/p2g_1201/failed:
total 0

These filenames don't seem to match what you are expecting, and I seem to be getting no matches for whatever reason. From previous runs, I expect the busco score for this genome to be around 70%. The genome itself is high quality, its just very distantly related to the other monocots in liliopsida.

[hyphaltip]

hi - the diamond is definitely failing to find hits not sure how/why from the error messages though.

[nextgenusfs]

Not sure either, could it be running out of memory? I would try to run that diamond command outside of funannotate and see if you get a more informative error in an interactive terminal.

[nextgenusfs]

Hi @nhartwic, @estolle tracked this down to what looks like a segfault from a large contig (>26 MB). So this seems perhaps like a diamond version issue, might get resolution from downgrading to an older version of diamond, or @estolle was able to compile the latest in master and update the diamond command to fix the issue.

[nhartwic]

Apologies for not updating sooner. Got side tracked. I ended up reverting to an older version of funannotate based on an old conda environment, but doing so required enabling the use of 'broken' labeled packages in conda forge as one of the GXX related packages had its label changed during the intervening time when I originally built the env and when I was trying to build a container.

Seems seem to be working now. I never got a 1.8.1 build to work. I tried changing the diamond versions and conda kept complaining about package conflicts.

I'm fairly confident that my issue isn't related to memory limits. Tibanna tracks memory usage in intervals of every few minutes and I never saw a huge spike. Granted, its possible every few minutes is insufficient resolution to see some spike in memory that could be causing problems.

[nextgenusfs]

Okay -- with conda you can use the --force to update a package to a different version than was is specified in the recipe. I did pin >=2.0.4 on diamond which is why it was complaining. I will not be so stringent with the next release on diamond version to give users a bit more flexibility within the constraints of conda. Not sure if same issue that @estolle had or not, there was a clear 'seg fault' in the output.

[nhartwic]

Well, apparently the older funannotate env, 1.7.4, is broken too. New error though.

Traceback (most recent call last):
  File "/opt/conda/envs/myenv/bin/funannotate", line 660, in <module>
    main()
  File "/opt/conda/envs/myenv/bin/funannotate", line 650, in main
    mod.main(arguments)
  File "/opt/conda/envs/myenv/lib/python2.7/site-packages/funannotate/predict.py", line 1594, in main
    args.out, 'predict_misc'), GlimmerPredictions)
  File "/opt/conda/envs/myenv/lib/python2.7/site-packages/funannotate/library.py", line 5223, in runGlimmerHMM
    glimmer2gff3(glimmerRaw, output)
  File "/opt/conda/envs/myenv/lib/python2.7/site-packages/funannotate/library.py", line 5152, in glimmer2gff3
    '\t')
ValueError: need more than 2 values to unpack

Glimmer output at a surface level looks normal.

-rw-r--r--    1 root root  35K Oct 25 09:03 glimmer.exons
drwxr-xr-t    2 root root 4.0K Oct 25 09:04 glimmerhmm
-rw-r--r--    1 root root  945 Oct 25 09:10 glimmerhmm.log
-rw-r--r--    1 root root 186M Oct 25 14:34 glimmerHMM.output.raw
-rw-r--r--    1 root root 3.4M Oct 25 14:34 glimmerhmm-predictions.gff3

At this point, I'm kind of over trying to debug older versions of funannotate and associated environments. I'm back to 1.8.1. Downgrading to Diamond 2.0.2 didn't work. I guess I'll jump over to the other issue to discuss getting diamond to work.

[nextgenusfs]

Sorry @nhartwic, I just pushed a commit to master that will let you use tblastn to pre-filter the protein to genome hits before running exonerate. This will be slower, but it should not die on you. The original method I wrote used tblastn, but I got away from that because multi-threading has been broken in tblastn for like the last 5 years -- so as long as you use the old version I've pinned in the condo recipe it should be fine.

So you can install the master over the top of your v1.8.1 by just running this from your conda environment:

python -m pip install --no-deps --force git+https://github.com/nextgenusfs/funannotate.git

And then you can pass --p2g_prefilter tblastn to funannotate predict to use that method. Diamond v2.0.4 is likely to work fine for all other tasks in funannotate as they are blastp searches.

[nhartwic]

Well, rebuilt and attempted some test runs. Ran into some memory limits so scaled up my machine. Then everything appeared to run correctly, but upon further investigation, something went wrong.

Funanntoate never errored out but failed to produce the expected output files. This is the script that is getting generated by my workflow at the moment...

# pull reads from sra, align to reference and merge in prep for passing to funanotate
fasterq-dump SRR2045957 -o SRR2045957
fasterq-dump SRR647050 -o SRR647050
hisat2-build salk-tm-dev/duckweed/landoltia/Landoltia_punctata_v3.softmasked.fasta hisat2_index
hisat2  -p 32 -x hisat2_index -1 SRR2045957_1.fastq -2 SRR2045957_2.fastq | samtools sort > alignments.0.bam
hisat2  -p 32 -x hisat2_index -1 SRR647050_1.fastq -2 SRR647050_2.fastq | samtools sort > alignments.1.bam
samtools merge alignments.bam alignments.0.bam alignments.1.bam

# pull busco db to be used
wget https://busco-data.ezlab.org/v4/data/lineages/liliopsida_odb10.2020-09-10.tar.gz
tar -zxf liliopsida_odb10.2020-09-10.tar.gz

# pull funannotate dbs to be used (I'm pretty sure busco_outgroups isn't needed, but whatever)
funannotate setup -i uniprot -d fun_db/
funannotate setup -i repeats -d fun_db/
funannotate setup -i busco_outgroups -d fun_db/

funannotate predict  -i salk-tm-dev/duckweed/landoltia/Landoltia_punctata_v3.softmasked.fasta \
    -o Landoltia_punctata_v3 --species Landoltia_punctata_v3 -d $(pwd)/fun_db/ \
    --busco_db $(pwd)/liliopsida_odb10 --organism other --repeats2evm \
    --name Lp5562.a03 --augustus_species maize \
    --busco_seed_species maize --max_intronlen 10000 \
    --cpus 32 --rna_bam alignments.bam  --p2g_prefilter tblastn

...This is being run inside a singularity container built from a docker container hosted at...

docker://nolanhartwick/salk-containers:fun

...Here is a quick listing of the predict_results folder...

total 11M
-rw-r--r-- 1 root root  11M Nov  4 06:41 Landoltia_punctata_v3.tbl
-rw-r--r-- 1 root root 1.4K Nov  4 06:42 Landoltia_punctata_v3.stats.json
-rw-r--r-- 1 root root  651 Nov  4 06:42 maize.parameters.json

Here is a recursive listing of every file in the funannotate output directory.

Everything in the log looks normal at a glance. The last logged messages are...

[06:42 AM]: Collecting final annotation files for 22,085 total gene models
[06:42 AM]: Funannotate predict is finished, output files are in the Landoltia_punctata_v3/predict_results folder
[06:42 AM]: Your next step to capture UTRs and update annotation using PASA:

  funannotate update -i Landoltia_punctata_v3 --cpus 32 \
            --left illumina_forward_RNAseq_R1.fastq.gz \
            --right illumina_forward_RNAseq_R2.fastq.gz \
            --jaccard_clip

[06:42 AM]: Training parameters file saved: Landoltia_punctata_v3/predict_results/maize.parameters.json
[06:42 AM]: Add species parameters to database:

  funannotate species -s maize -a Landoltia_punctata_v3/predict_results/maize.parameters.json

  ESC[4mSource         Weight   CountESC[0m
  Augustus       1        15016
  Augustus HiQ   2        7539
  GlimmerHMM     1        48205
  snap           1        1525
  Total          -        72285

This may be related to #506 ? I honestly have know idea.

Here is the full log I have. Its a bit of a mess as it interweaves reports on my EC2 node with stdout from funannotate and a few other tools that my script runs before invoking funannotate. But it should be complete.

EDIT: Cleaned up the log file so that it no longer has the recursive listing or the top calls.
This file now contains only the setup for my script (pulling and building the singularity container) and the stdout associated with the commands from the script The actual funanntoate log lines start on line 3920.

[nextgenusfs]

Could you post the funannotate-predict.log file located in Landoltia_punctata_v3/logfiles/. Should be a little bit easier to read as well as have the calls to all the dependencies.

So if the tbl output file is in the predict_results folder but not the rest of the results, it might suggest that the ncbi tool tbl2asn died.

[nextgenusfs]

And a --no-progress clearly needs to be an option if running in non-interactive mode like this.....

[nextgenusfs]

It also looks like snap failed to some extent if it only predicted 1525 models might be related to the install, looks like we talked about this before #386?