ragtag/scaffold and ragtag/patch modules and tests

[nschan]

Here are two modules for ragtag, one for assembly scaffolding to a reference, and one for assembly patching based on a target sequence. These were local modules in genomeassembler which I am beginning to migrate to nf-core/modules.

[jfy133]

Overall pretty good, just some optmisation suggetions to bring in line with the nf-core specifications: https://nf-co.re/docs/guidelines/components/modules#capturing-stdout-and-stderr

[nschan]

I don't know why test conda 12 fails, I think something is difficult with ragtag via conda. When I created the test results conda took extremely long (killed it after 1.5 hours).

[jfy133]

@nschan how much data does the module produce?

    >   tail: 'test/ragtag.patch.err' has become inaccessible: No such file or directory

And what does the tail of this file look like locally?

[jfy133]

Almost there (barring the timing out conda test)

[jfy133]

Please add metas for the other files in separate input channels too rather than a single tuple.

[nschan]

OK, I can do that. Can you clarify for me what the advantage of having more channels with their own meta is in this case? To me it makes sense to have the exclude file somehow linked to the assembly, since it probably needs to be crafted for each one individually, and I prefer (personally) to have fewer input channels, but maybe I am missing an advantage of having multiple input channels?

[nschan]

And to clarify more: should i do:

    input:
    tuple val(meta), path(assembly, name: 'assembly/*'), path(reference, name: 'reference/*')
    tuple val(meta2), path(exclude), path(skip), path(hard_skip)

or

    tuple val(meta), path(assembly, name: 'assembly/*'), path(reference, name: 'reference/*')
    tuple val(meta2), path(exclude) 
    tuple val(meta3), path(skip), path(hard_skip)

or something different?

[jfy133]

The latter, or but also spliut the assembly and refernece too

[nschan]

@nschan how much data does the module produce?

The work directory of the test is 270 KB

    >   tail: 'test/ragtag.patch.err' has become inaccessible: No such file or directory

And what does the tail of this file look like locally?

This file is empty

[jfy133]

My feeling

    >   tail: 'test/ragtag.patch.err' has become inaccessible: No such file or directory
    ```
    
    It's to do with the funky `tail -F` thing, I would suggest trying to redirect with `>` at the end instead and see what happens (or try removing the logging for now and see if thehn it runs through fine)

[mashehu]

looks like you were right, @jfy133!

[jfy133]

Interesting... maybe that pattern isn't so robust then... and unfortuantely we have no blame history of that file because we did that big restructure 🤦

I'm going to take a random guess.. @mahesh-panchal were you by any chance the author of: https://nf-co.re/docs/guidelines/components/modules#capturing-stdout-and-stderr ?

[jfy133]

But otherwise @nschan maybe try next just ragtag.py <...> > ${prefix}.log for now?

[mahesh-panchal]

Interesting... maybe that pattern isn't so robust then... and unfortuantely we have no blame history of that file because we did that big restructure 🤦

I'm going to take a random guess.. @mahesh-panchal were you by any chance the author of: https://nf-co.re/docs/guidelines/components/modules#capturing-stdout-and-stderr ?

Yes, that was me: nf-core/website#2136.

[nschan]

So, just to clarify patch.err is created by ragtag, I had to touch the file first to prevent tail -F from complaining; it is also emitted.

[mahesh-panchal]

Wait. Not the tail -F part.

Actually I reviewed that it seems: nf-core/website#2345

[mahesh-panchal]

@pontus Any insight on why the tail -F was causing issues?

[nschan]

But otherwise @nschan maybe try next just ragtag.py <...> > ${prefix}.log for now?

I opted for the tee method, which seems ok. Please let me know if there is anything else that I should change for these modules @jfy133

[pontus]

@pontus Any insight on why the tail -F was causing issues?

Do you have a link to the check where this happened readily available? (I would expect the setup shown to give some ugly things in logs but expect them to be red herrings typically.)

[nschan]

Here is one @pontus https://github.com/nf-core/modules/actions/runs/14303829077/job/40083278515?pr=8198

[jfy133]

One last comment but is nto a blocker, just optimisation suggestion

[jfy133]

I still don't really follow the logic here, why load them into a directory as defined in target if you then just use the list of input fastas directly ? And again, didn't fully undersstand why you need to rename the files - is really trying to put in two files as genome.fa going to be so common? I would still rather just decompress with gunzip, and give them direcly to the command.

I would also not cp the reference file as for large reference genomes that's a lot of HDD space to use, maybe just symlink to get the correct name?

However this is not a blocker here.

[nschan]

I still don't really follow the logic here, why load them into a directory as defined in target if you then just use the list of input fastas directly ? And again, didn't fully undersstand why you need to rename the files - is really trying to put in two files as genome.fa going to be so common? I would still rather just decompress with gunzip, and give them direcly to the command.

I do not know how common it is that someone uses two files with the same name for both target and reference, but it doesn't hurt to support that I guess? Loading them into a different directory is purely to avoid having a file with the same name twice; not sure if there is something I am not getting about what you are asking?

I would also not cp the reference file as for large reference genomes that's a lot of HDD space to use, maybe just symlink to get the correct name?

I have changed to symlinking.

[pontus]

@nschan Thanks, unfortunately, that doesn't make sense to me.

My guess is it's failing because the nextflow process somehow doesn't end as expected when it reaches its end but rather hangs around waiting for the tail to go to away. If that's correct I think all backgrounding of persistent processes are likely to cause issues spuriously.

It would be interesting to see what happens if one kills the tail after it has served its purpose (e.g. as below).

tail -F something >&2 &
tailpid=$!
do stuff
...
kill -TERM "$tailpid"

[pontus]

(I'm not sure what is best practice, this module seems to rely fairly heavily on the nf-core default options for process.shell to do error detection, should that be expected to be mentioned somewhere?)

[nschan]

So it seems that killing tail when the tool ran, as suggested by @pontus, solves the issue.

(I'm not sure what is best practice, this module seems to rely fairly heavily on the nf-core default options for process.shell to do error detection, should that be expected to be mentioned somewhere?)

To be honest, I do not really understand; I was under the impression that these modules are in line with what most nf-core modules do?