Update modules install to support version control by ErikDanielsson · Pull Request #1116 · nf-core/tools

ErikDanielsson · 2021-06-16T12:14:37Z

Updated nf-core modules install command to accommodate version control for modules, as suggested in #868.

New flag --latest installs the latest version of the module.
New flag --force overwrites the installed version of the module
New option --sha <commit sha> to install specific version of module.
questionary select for interactive selection of version

PR checklist

This comment contains a description of changes (with reason)
CHANGELOG.md is updated
If you've fixed a bug or added code that should be tested, add tests!
Documentation in docs is updated

…ules-install

codecov · 2021-06-16T12:16:56Z

Codecov Report

Merging #1116 (5b7c79b) into dev (afbed88) will decrease coverage by 0.33%.
The diff coverage is 67.39%.

@@            Coverage Diff             @@
##              dev    #1116      +/-   ##
==========================================
- Coverage   70.36%   70.02%   -0.34%     
==========================================
  Files          36       38       +2     
  Lines        4545     4748     +203     
==========================================
+ Hits         3198     3325     +127     
- Misses       1347     1423      +76

Impacted Files	Coverage Δ
nf_core/__main__.py	`60.85% <57.14%> (-0.21%)`	⬇️
nf_core/modules/pipeline_modules.py	`69.19% <60.55%> (-10.34%)`	⬇️
nf_core/modules/module_utils.py	`63.10% <63.10%> (ø)`
nf_core/modules/modules_repo.py	`90.74% <90.74%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update afbed88...5b7c79b. Read the comment docs.

KevinMenden

Really nice! This looks already pretty good.

The main suggestion I have here is to make the code robuster and surround especially the GitHub API call with a try-except and try to capture any errors that you expect.
You might also do this when reading/writing the json file, just to be on eht esafe side. But definitely for the api call.

KevinMenden · 2021-06-16T12:48:17Z

Ah and one more thing we should add some logging to inform the user that e.g. a modules.json file has been created.

Co-authored-by: Kevin Menden <kevin.menden@live.com>

KevinMenden

@ErikDanielsson tested this a bit and I think this is really nice!
So feel free to remove the draft state of that PR.

KevinMenden · 2021-06-18T07:41:43Z

Okay did it myself 😁

Co-authored-by: Kevin Menden <kevin.menden@live.com>

…hout interactive prompt

ErikDanielsson · 2021-06-21T14:38:48Z

It works for me when trying it on fastqc, which has quite a lot of commits. It crashes when it reaches the end of the commit history though -- will look into that.

…ols into update-modules-install

ErikDanielsson · 2021-06-22T07:58:56Z

The latest commit addresses that the directory structure of the modules repo was changed with nf-core/modules#80. Hence commits prior to this PR should not be fetched.

KevinMenden

Okay this looks good to me! 👍
Still need to update the changelog.
I'll approve now but want to test around with it before merging.

KevinMenden

Really nice, I think we're almost there.
Only a bit of code around force and sha 👍

KevinMenden · 2021-06-22T13:07:21Z

 @click.option("-t", "--tool", type=str, metavar="<tool> or <tool/subtool>")
-def install(ctx, pipeline_dir, tool):
+@click.option("-l", "--latest", is_flag=True, default=False, help="Install the latest version of the module")
+@click.option("--force", is_flag=True, default=False, help="Force installation of module if module already exists")
+def install(ctx, pipeline_dir, tool, latest, force):


We need an option to specify the sha here, e.g. -s, --sha as discussed on slack

KevinMenden · 2021-06-22T13:16:19Z

+            if not force:
+                log.error(
+                    "Module directory already exists but module {} is not present in 'module.json'".format(module_dir)
+                )
+                return False


Here it would be nice to ask the user whether to overwrite (we should have something similar somewhere else, I think nf-core modules create-test-yml).
Basically a simple question whether to force overwrite and if yes, set 'force' to true.

ErikDanielsson · 2021-06-22T15:53:13Z

All options are added now, I think, but they still need thorough testing. I think we will have to refactor pipeline_modules.py into one file per command at some point -- its quite a mess now.

KevinMenden

Hi @ErikDanielsson , nice work!
Just a couple of minor comments.

I've tested around with it a bit and we need to add the option to update all modules, as we won't include the nf-core modules update command now. But I would actually like to treat this as a separate PR and get this one here merged now, so we can continue working on the features.

So I'm approving it now and then I say we get this one merged in! We might find some other issues along the way but can then still solve them before the release.

Nice work!

KevinMenden · 2021-06-23T06:49:09Z

+        if self.latest and self.sha is not None:
+            log.error("Cannot use '--sha' and '--latest' at the same time!")


we should probably exit here

Co-authored-by: Kevin Menden <kevin.menden@live.com>

…ols into update-modules-install

ErikDanielsson added 2 commits June 16, 2021 14:09

Added support for modules.json for modules install commandq

4765d9a

Merge branch 'dev' of github.com:ErikDanielsson/tools into update-mod…

916911c

…ules-install

ErikDanielsson requested a review from KevinMenden June 16, 2021 12:15

KevinMenden reviewed Jun 16, 2021

View reviewed changes

Comment thread nf_core/modules/pipeline_modules.py Outdated

KevinMenden reviewed Jun 16, 2021

View reviewed changes

Comment thread nf_core/modules/pipeline_modules.py Outdated

KevinMenden requested changes Jun 16, 2021

View reviewed changes

KevinMenden reviewed Jun 16, 2021

View reviewed changes

Comment thread nf_core/modules/pipeline_modules.py Outdated

ErikDanielsson and others added 3 commits June 16, 2021 14:53

Remove redundant code

bb62ddb

Co-authored-by: Kevin Menden <kevin.menden@live.com>

Implement suggestions from code review

8af92af

Implement suggestions from code review

db39da3

KevinMenden reviewed Jun 16, 2021

View reviewed changes

Comment thread nf_core/modules/pipeline_modules.py

Apply changes from code review

3351841

Co-authored-by: Kevin Menden <kevin.menden@live.com>

KevinMenden reviewed Jun 18, 2021

View reviewed changes

Comment thread nf_core/modules/pipeline_modules.py Outdated

KevinMenden reviewed Jun 18, 2021

View reviewed changes

Comment thread nf_core/modules/pipeline_modules.py Outdated

KevinMenden approved these changes Jun 18, 2021

View reviewed changes

KevinMenden marked this pull request as ready for review June 18, 2021 07:41

ErikDanielsson and others added 9 commits June 18, 2021 15:37

Added API call for full git log

199216f

Rewrote code for module.json creation

ca5cb84

Moved ModulesRepo to separate file

2c9a39e

Resolve merge conflict

770b595

Apply suggestions from code review

278fe86

Co-authored-by: Kevin Menden <kevin.menden@live.com>

Bug fixes after testing with rnaseq pipeline

a353a43

Added support for pagination of commits

a9a8d17

Update function descriptions

e04cf4e

Implemented installation of different pipeline versions

0dbe865

KevinMenden reviewed Jun 21, 2021

View reviewed changes

Comment thread nf_core/modules/module_utils.py Outdated

Added new flag '--latest' to install the latest version of module wit…

ef73868

…hout interactive prompt

ErikDanielsson added 3 commits June 21, 2021 16:39

Merge branch 'update-modules-install' of github.com:ErikDanielsson/to…

db8b81c

…ols into update-modules-install

Only display 'older commits' if there are any

38b314d

Remove redundant code

cfad229

ErikDanielsson added the DSL2 label Jun 21, 2021

Restriced commit log to only be shown from given date

1a5c215

ErikDanielsson added 2 commits June 22, 2021 10:02

Minor fixes

bc6366e

Fix index of out range error

ad7482d

KevinMenden approved these changes Jun 22, 2021

View reviewed changes

ErikDanielsson added 2 commits June 22, 2021 11:25

Updated CHANGELOG.md

2a07d52

Fix CHANGELOG.md

ee5bbea

KevinMenden reviewed Jun 22, 2021

View reviewed changes

Comment thread CHANGELOG.md

ErikDanielsson added 5 commits June 22, 2021 13:41

Add '--force' option

1ea9be8

Remove duplicated code

286724d

Added commit SHA to message, and bug fix of '--force'

d033d3d

Bug fixes for '--force'

dabdc64

Added '--sha' flag to specify tool version

a9a769e

KevinMenden reviewed Jun 22, 2021

View reviewed changes

ErikDanielsson added 2 commits June 22, 2021 17:42

Refactor modules install

0c4cda5

Update tests with new flags

89339e1

Make questionary select show installed version

5087619

KevinMenden approved these changes Jun 23, 2021

View reviewed changes

ErikDanielsson and others added 4 commits June 23, 2021 09:16

Apply suggestions from code review

e39ff09

Co-authored-by: Kevin Menden <kevin.menden@live.com>

Apply changes from code review

2586097

Merge branch 'update-modules-install' of github.com:ErikDanielsson/to…

4dce523

…ols into update-modules-install

Black

5b7c79b

ErikDanielsson merged commit d6f616e into nf-core:dev Jun 23, 2021

ErikDanielsson deleted the update-modules-install branch July 25, 2022 09:02

		if self.latest and self.sha is not None:
		log.error("Cannot use '--sha' and '--latest' at the same time!")

Uh oh!

Conversation

ErikDanielsson commented Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

codecov Bot commented Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

KevinMenden left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KevinMenden commented Jun 16, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KevinMenden left a comment

Choose a reason for hiding this comment

Uh oh!

KevinMenden commented Jun 18, 2021

Uh oh!

Uh oh!

ErikDanielsson commented Jun 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ErikDanielsson commented Jun 22, 2021

Uh oh!

KevinMenden left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KevinMenden left a comment

Choose a reason for hiding this comment

Uh oh!

KevinMenden Jun 22, 2021

Choose a reason for hiding this comment

Uh oh!

KevinMenden Jun 22, 2021

Choose a reason for hiding this comment

Uh oh!

ErikDanielsson commented Jun 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KevinMenden left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KevinMenden Jun 23, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ErikDanielsson commented Jun 16, 2021 •

edited

Loading

codecov Bot commented Jun 16, 2021 •

edited

Loading

ErikDanielsson commented Jun 21, 2021 •

edited

Loading

ErikDanielsson commented Jun 22, 2021 •

edited

Loading