support 'west init --topdir'#854
support 'west init --topdir'#854thorsten-klein wants to merge 3 commits intozephyrproject-rtos:mainfrom
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #854 +/- ##
==========================================
+ Coverage 85.95% 86.03% +0.08%
==========================================
Files 11 11
Lines 3453 3474 +21
==========================================
+ Hits 2968 2989 +21
Misses 485 485
|
652408c to
8a2e15e
Compare
|
I really want to review this one because I already spent a lot of time on #774 and #775 before but @thorsten-klein is just submitting way too much code, I can't keep up!
Can you give me a few more days? |
Sorry 🤞
Yes, sure 👍🏻 Thank you in advance for your time and input! |
a08dec0 to
99377aa
Compare
Seconded, and I'll take this opportunity to thank you for all your contributions @thorsten-klein. |
|
Let's wait for feedback from @marc-hb, as he requested some more time. |
marc-hb
left a comment
There was a problem hiding this comment.
This is particularly useful when using --local
I downloaded this PR and tested the other, --manifest-url option and it does not seem to work for me. I ran out of time and did not try --local yet. I didn't look at the code either.
Describing directory nesting clearly and accurately is not easy - it is basically a "specification" except it has to be user-friendly too. So I will look at the code only when that "specification" is complete and unambiguous.
This help looks like the two last ones can be both used at the same time:
west init [-m URL] [--mr REVISION] [--mf FILE] [-o=GIT_CLONE_OPTION] [-t WORKSPACE_DIR] [directory]
But it's not actually possible:
FATAL ERROR: --topdir cannot be combined with positional argument [directory]
When using either separately, each one seems to have the same effect. So, there does not seem to be any new feature in the remote case, which means it's still not possible to west init a manifest two levels down or more in the remote case? Is this an oversight? The help does not seem to say. I don't see why --local would be more flexible than --manifest-url. I would hope the --manifest-url and --local options work the same immediately after the git clone operation.
|
|
Thank you for the review 🙏
Yes, exactly. Nevertheless, in remote case those arguments that do the same are mutually exclusive (see error message you got).
This has already worked before in remote case, since the positional argument Thank you 👍🏻 |
99377aa to
552f964
Compare
|
I think I have resolved all review comments in the code👍🏻 |
Has it? Please share one example of a |
5d88b67 to
9647d77
Compare
9647d77 to
88035ae
Compare
88035ae to
553ce1a
Compare
553ce1a to
482e8d4
Compare
|
From #854 (comment)
Also, it's very time-consuming and error prone to have high-level design discussions buried in in tens of comments about typos and other, low-level implementation details. Let's discuss user interface and other high-level design points in issue #774 rather than in this PR. #774 is a bit noisy because of some initial misunderstandings but it's not as noisy as here. |
482e8d4 to
bb01594
Compare
In an ideal world, sure. But I think in this particular case we are already too far ahead in the implementation and design to change the location of the discussions in my opinion. Reading your comments, I don't think we are that far from agreeing to a solution here, so @thorsten-klein perhaps we can try to resolve the last points (specifically this comment). |
It's not really a "change" because all the BTW I'm writing "high-level design" as in "looking at the --help, not the code". This is only about defining what the command-line parameters mean, it's not advanced maths! It's just faster to discuss and define this before writing and reviewing the code, that's all.
Agreed, and I also love being proven wrong. I'm only upset about the time-consuming way to get there. EDIT, trying to be fair: @thorsten-klein did not initially intend to implement |
There was a problem hiding this comment.
All the combinatorial explosion and complexity was clearly and vastly underestimated. I think the only way to get back on track is to follow the "top-down", more formal approach that should have been used in the first place:
- Make an extensive list of all user input possibilities. NO CODE here.
- Complete and unambiguous definition of all resulting behaviors that can immediately answer all questions like the ones above. NO CODE here.
- "Tests first" approach with extensive coverage.
- Code LAST (design first)
Steps 3. and 4. should be PAUSED until 1. and 2. are discussed and agreed: PLEASE DO NOT PUSH until they are. Prototyping is great. Reviewing prototypes while wrongly believing they are close to final has been crazy time-consuming, let's please stop?
| If no `-m / --manifest-url` is provided, west uses Zephyr URL by default: | ||
| {MANIFEST_URL_DEFAULT}. | ||
|
|
||
| The topdir (where `.west` is created) is determined as follows: |
There was a problem hiding this comment.
Just found this, funny enough: https://interrupt.memfault.com/blog/practical_zephyr_west
Notice that we no longer specify the current directory in the call to west init using “.”. In fact, the directory - optionally passed as the last argument to west init - is interpreted differently by West when using the --local flag or the -m arguments:
With --local, the directory specifies the path to the local manifest repository.
Without the --local flag, the directory refers to the topdir and thus the folder in which to create the workspace (defaulting to the current working directory in this case).
Awkward, but this approach is probably used due to legacy reasons. If no --local flag is used and no repository is specified using -m, West defaults to using the Zephyr repository.
src/west/app/project.py
Outdated
| if args.topdir: | ||
| topdir = Path(args.topdir) | ||
| if args.directory: | ||
| if not Path(args.directory).is_relative_to(topdir): |
There was a problem hiding this comment.
So you just removed abspath() from the earlier version if not Path(abspath(args.directory)).is_relative_to following my comment in #854 (comment) but that did not fix relative directory arguments. Now relative directory arguments all fail because is_relative_to() fails silently with relative arguments:
>>> Path('d1/d2').is_relative_to('/')
False
>>> Path('/d1/d2').is_relative_to('/')
TrueIt's confusing to have relative arguments not being relative (!) but it's documented in Pathlib. There is a test coverage gap for this use case (and you changed that code without testing manually).
This raises a subtle question: are relative directory arguments relative to topdir, or relative to the current directory? I would say the former but I don't have a strong opinion. Either way it needs to be clearly specified.
src/west/app/project.py
Outdated
| f"directory '{args.directory}' must be relative " | ||
| f"to west topdir '{args.topdir}'" | ||
| ) | ||
| manifest_subdir = Path(os.path.relpath(args.directory, topdir)) |
There was a problem hiding this comment.
Please avoid os.path unless absolutely necessary. It's complicated enough to deal with one Python module (Pathlib), please don't mix both.
src/west/app/project.py
Outdated
| manifest_path = PurePath(urlparse(manifest_url).path).name | ||
| mf_path = PurePath(urlparse(manifest_url).path).name | ||
|
|
||
| manifest_path: Path = Path(subdir) / mf_path |
There was a problem hiding this comment.
I could be wrong, but for args.directory consistency, I expected these two commands to converge to the exact same state. I just tested and they produce something different right now:
west init -l -t ~/tmp/topdir ~/tmp/topdir/extra/level/zephyr/
# ... should end in the same final state as
west init -m http://..../zephyr/ -t ~/tmp/topdir ~/tmp/topdir/extra/level/zephyr/Once the behavior for absolute paths is finalized, there is the delicate question of relative paths. When args.directory is relative, what is it relative to? Is it relative to topdir? Or is it relative to the current directory where the command is run? Either way, this command always fails right now:
cd ~/tmp/topdir/
west init -m .... -t ~/tmp/topdir extra/level
FATAL ERROR: directory 'extra/level' must be relative to west topdir '~/tmp/topdir'That's because Pathlib is_relative_to() is not intuitive at all, see my other comment about that.
bb01594 to
c127022
Compare
I almost agree (only the last argument has to omit I have added a test to prove that the commands really result in the same final state (= same tree). |
c127022 to
cb9714f
Compare
Can you give some reason(s) why you prefer this variant? Not only it make these two commands above inconsistent with each other, this also makes the second one inconsistent with Also, this makes it impossible to rename without an extra rename action after cloning (which is probably why This is not an obscure implementation detail that could easily be changed later, this is literally defining the main user interface of this new feature. After a gazillion of prototypes force-pushed that I manually tested and reviewed :-((( Once that user interface choice is made and people start using this feature, there is no way back (except for a painful deprecation, best avoided).
So, in answer to this request of mine, "I prefer this variant" without any elaboration was your only plain English reply AND you force-pushed at least twice, piling up on the already out of control "evolog" and untrackable progress[1,2]. Did you miss or misunderstand what I asked there or is it really impossible to change your approach? [1] I did not look at these new force pushes yet. I bet even more will come. |
There will be more revisions because the latest version cb9714f fails on Windows: As a coincidence, I just had to work on similar portability issues and found a way to test Windows without Windows: |
I do not prefer this as well. Until now I thought that this was existing behavior. But it seems to be not. |
Ping? Is it |
multiple combinations of arguments are tested as they can be provided to 'west init'.
Added new argument for west init to specify the west topdir when a workspace is inizialized.
Added new argument for west init to specify the west topdir when a workspace is inizialized.
cb9714f to
6e3defd
Compare
|
I’ve separated the tests into individual commits so it’s easier to verify that existing behavior remains unchanged. |
|
| assert actual.west_commands == expected.west_commands | ||
|
|
||
|
|
||
| def tree(path: Path, prefix="") -> str: |
There was a problem hiding this comment.
I think https://anytree.readthedocs.io/en/latest/intro.html provides this for free and it is already a Zephyr requirement.
Don't get me wrong: I am NOT suggesting to add new west dependency but only to add a new west TEST dependency
Fix detection of some existing (and very unlikely) .west/ "pollution" at an intermediate level, for details see first comment at zephyrproject-rtos#935 This pollution is very unlikely because it's pretty difficult to have the manifest clone more than one level down before zephyrproject-rtos#774 / zephyrproject-rtos#854 - so it's difficult to have "intermediate" levels. Signed-off-by: Marc Herbert <Marc.Herbert@gmail.com>
This was a
This bug looks like it has been there forever. No one noticed before you because:
It resets two values in
Help review #936 :-)
It wasn't intentional because it depended on the current directory |
Fixes #774 (Successor of #775)
This PR introduces a new
--topdiroption forwest init, allowing users to explicitly specify the west workspace directory (topdir) during initialization.This is particularly useful when using
--local, as the current behavior always sets the parent of the given directory as the topdir. In addition, west is currently unable to determine bothmanifest.pathandmanifest.filefrom a singledirectoryargument as the path can be interpreted in multiple ways (e.g. it can be split at various places).With the new
--topdirargument, users can now directly control where the west workspace is created.Parameterized tests for
west init --topdirare added as well.Moreover, the help texts and descriptions are (hopefully) simplified.