Skip to content

support 'west init --topdir'#854

Open
thorsten-klein wants to merge 3 commits intozephyrproject-rtos:mainfrom
thorsten-klein:west-init-topdir
Open

support 'west init --topdir'#854
thorsten-klein wants to merge 3 commits intozephyrproject-rtos:mainfrom
thorsten-klein:west-init-topdir

Conversation

@thorsten-klein
Copy link
Copy Markdown
Contributor

@thorsten-klein thorsten-klein commented Oct 5, 2025

Fixes #774 (Successor of #775)

This PR introduces a new --topdir option for west init, allowing users to explicitly specify the west workspace directory (topdir) during initialization.

This is particularly useful when using --local, as the current behavior always sets the parent of the given directory as the topdir. In addition, west is currently unable to determine both manifest.path and manifest.file from a single directory argument as the path can be interpreted in multiple ways (e.g. it can be split at various places).

With the new --topdir argument, users can now directly control where the west workspace is created.

Parameterized tests for west init --topdir are added as well.

Moreover, the help texts and descriptions are (hopefully) simplified.

@codecov
Copy link
Copy Markdown

codecov bot commented Oct 5, 2025

Codecov Report

❌ Patch coverage is 93.33333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.03%. Comparing base (11302e1) to head (bb01594).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/west/app/project.py 93.33% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #854      +/-   ##
==========================================
+ Coverage   85.95%   86.03%   +0.08%     
==========================================
  Files          11       11              
  Lines        3453     3474      +21     
==========================================
+ Hits         2968     2989      +21     
  Misses        485      485              
Files with missing lines Coverage Δ
src/west/app/project.py 80.91% <93.33%> (+0.39%) ⬆️

Copy link
Copy Markdown
Contributor

@mbolivar mbolivar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me

Copy link
Copy Markdown
Collaborator

@pdgendt pdgendt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@pdgendt
Copy link
Copy Markdown
Collaborator

pdgendt commented Oct 8, 2025

@marc-hb I'm not sure if @fouge is very active? I think we can move forward with this?

@thorsten-klein thorsten-klein force-pushed the west-init-topdir branch 3 times, most recently from 652408c to 8a2e15e Compare October 10, 2025 20:37
pdgendt
pdgendt previously approved these changes Oct 13, 2025
@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Oct 14, 2025

I really want to review this one because I already spent a lot of time on #774 and #775 before but @thorsten-klein is just submitting way too much code, I can't keep up!

west did not use to be like this, help! :-D

Can you give me a few more days?

@thorsten-klein
Copy link
Copy Markdown
Contributor Author

@thorsten-klein is just submitting way too much code, I can't keep up.

Sorry 🤞

Can you give me a few more days?

Yes, sure 👍🏻 Thank you in advance for your time and input!

@thorsten-klein thorsten-klein force-pushed the west-init-topdir branch 2 times, most recently from a08dec0 to 99377aa Compare October 15, 2025 06:08
@carlescufi
Copy link
Copy Markdown
Member

@thorsten-klein is just submitting way too much code, I can't keep up.

Seconded, and I'll take this opportunity to thank you for all your contributions @thorsten-klein.

Copy link
Copy Markdown
Member

@carlescufi carlescufi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though I'd prefer --topdir was only compatible with -l, this is not a blocking issue for me. Thanks!

@pdgendt
Copy link
Copy Markdown
Collaborator

pdgendt commented Oct 15, 2025

Let's wait for feedback from @marc-hb, as he requested some more time.

Copy link
Copy Markdown
Collaborator

@marc-hb marc-hb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is particularly useful when using --local

I downloaded this PR and tested the other, --manifest-url option and it does not seem to work for me. I ran out of time and did not try --local yet. I didn't look at the code either.

Describing directory nesting clearly and accurately is not easy - it is basically a "specification" except it has to be user-friendly too. So I will look at the code only when that "specification" is complete and unambiguous.

This help looks like the two last ones can be both used at the same time:

west init [-m URL] [--mr REVISION] [--mf FILE] [-o=GIT_CLONE_OPTION] [-t WORKSPACE_DIR] [directory]

But it's not actually possible:

FATAL ERROR: --topdir cannot be combined with positional argument [directory]

When using either separately, each one seems to have the same effect. So, there does not seem to be any new feature in the remote case, which means it's still not possible to west init a manifest two levels down or more in the remote case? Is this an oversight? The help does not seem to say. I don't see why --local would be more flexible than --manifest-url. I would hope the --manifest-url and --local options work the same immediately after the git clone operation.

@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Oct 16, 2025

This is particularly useful when using --local

I downloaded this PR and tested the other, --manifest-url option and it does not seem to work for me. I ran out of time and did not try --local yet. I didn't look at the code either.

Describing directory nesting clearly and accurately is not easy - it is basically a "specification" except it has to be user-friendly too. So I will look at the code only when that "specification" is complete and unambiguous (EDIT and of course: working for me).

This help looks like the two last ones can be both used at the same time:

west init [-m URL] [--mr REVISION] [--mf FILE] [-o=GIT_CLONE_OPTION] [-t WORKSPACE_DIR] [directory]

But it's not actually possible:

FATAL ERROR: --topdir cannot be combined with positional argument [directory]

When using either separately, each one seems to have the same effect. So, there does not seem to be any new feature in the remote case, which means it's still not possible to west init a manifest two levels down or more in the remote case? Is this an oversight? The help does not seem to say. I don't see why --local would be more flexible than --manifest-url. I would hope the --manifest-url and --local options work the same immediately after the git clone operation.

@thorsten-klein
Copy link
Copy Markdown
Contributor Author

thorsten-klein commented Oct 16, 2025

Thank you for the review 🙏

When using either separately, each one seems to have the same effect. So, there does not seem to be any new feature in the remote case, which means it's still not possible to west init a manifest two levels down or more in the remote case? Is this an oversight? The help does not seem to say. I don't see why --local would be more flexible than --manifest-url. I would hope the --manifest-url and --local options work the same immediately after the git clone operation.

Yes, exactly. --topdir and positional argument directory do exactly the same in remote case.
I have proposed to get rid of this positional argument at all, but it will not be accepted for now because it is a breaking change (what I fully understand).

Nevertheless, in remote case those arguments that do the same are mutually exclusive (see error message you got).
I have applied your suggested change, which totally make sense to point out in the help that they both cannot be combined ([--topdir DIR | directory]) 👍🏻

it's still not possible to west init a manifest two levels down or more in the remote case?

This has already worked before in remote case, since the positional argument [directory] already exists and it does exactly the same as --topdir (see above). The main value in this change lays in the local case (and is the pre-work to potentially remove the positional argument [directory] in future)

Thank you 👍🏻

@thorsten-klein
Copy link
Copy Markdown
Contributor Author

I think I have resolved all review comments in the code👍🏻

@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Oct 17, 2025

it's still not possible to west init a manifest two levels down or more in the remote case?

This has already worked before in remote case, ...

Has it? Please share one example of a west init --manifest-url ... command that can produce this:

workspace
├── extra
│   └── level
│       └── manifest_clone
│           └── .git
└── .west

@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Jan 19, 2026

From #854 (comment)

but again it would have been more efficient to have this very discussion about the user interface and design first BEFORE writing the corresponding code, rather then "reserve-engineering" the code at code review time to discover that the UI design was not formalized.

Also, it's very time-consuming and error prone to have high-level design discussions buried in in tens of comments about typos and other, low-level implementation details. Let's discuss user interface and other high-level design points in issue #774 rather than in this PR. #774 is a bit noisy because of some initial misunderstandings but it's not as noisy as here.

@carlescufi
Copy link
Copy Markdown
Member

From #854 (comment)

but again it would have been more efficient to have this very discussion about the user interface and design first BEFORE writing the corresponding code, rather then "reserve-engineering" the code at code review time to discover that the UI design was not formalized.

Also, it's very time-consuming and error prone to have high-level design discussions buried in in tens of comments about typos and other, low-level implementation details. Let's discuss user interface and other high-level design points in issue #774 rather than in this PR. #774 is a bit noisy because of some initial misunderstandings but it's not as noisy as here.

In an ideal world, sure. But I think in this particular case we are already too far ahead in the implementation and design to change the location of the discussions in my opinion. Reading your comments, I don't think we are that far from agreeing to a solution here, so @thorsten-klein perhaps we can try to resolve the last points (specifically this comment).

@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Jan 22, 2026

But I think in this particular case we are already too far ahead in the implementation and design to change the location of the discussions in my opinion

It's not really a "change" because all the --topdir user interface and high-level design discussion happened in #774 in the first place and the majority of it is still there. Even after this gets merged, someone wondering about the user interface would rather stick to #774 and not have to get lost in typos and other implementation details to find rationale(s).

BTW I'm writing "high-level design" as in "looking at the --help, not the code". This is only about defining what the command-line parameters mean, it's not advanced maths! It's just faster to discuss and define this before writing and reviewing the code, that's all.

I don't think we are that far from agreeing to a solution here

Agreed, and I also love being proven wrong. I'm only upset about the time-consuming way to get there.

EDIT, trying to be fair: @thorsten-klein did not initially intend to implement --topdir and cloning at the same time and graciously accepted to add that during the review. But that entire discussion should still have happened before writing any code.

Copy link
Copy Markdown
Collaborator

@marc-hb marc-hb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the combinatorial explosion and complexity was clearly and vastly underestimated. I think the only way to get back on track is to follow the "top-down", more formal approach that should have been used in the first place:

  1. Make an extensive list of all user input possibilities. NO CODE here.
  2. Complete and unambiguous definition of all resulting behaviors that can immediately answer all questions like the ones above. NO CODE here.
  3. "Tests first" approach with extensive coverage.
  4. Code LAST (design first)

Steps 3. and 4. should be PAUSED until 1. and 2. are discussed and agreed: PLEASE DO NOT PUSH until they are. Prototyping is great. Reviewing prototypes while wrongly believing they are close to final has been crazy time-consuming, let's please stop?

If no `-m / --manifest-url` is provided, west uses Zephyr URL by default:
{MANIFEST_URL_DEFAULT}.

The topdir (where `.west` is created) is determined as follows:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just found this, funny enough: https://interrupt.memfault.com/blog/practical_zephyr_west

Notice that we no longer specify the current directory in the call to west init using “.”. In fact, the directory - optionally passed as the last argument to west init - is interpreted differently by West when using the --local flag or the -m arguments:

With --local, the directory specifies the path to the local manifest repository.
Without the --local flag, the directory refers to the topdir and thus the folder in which to create the workspace (defaulting to the current working directory in this case).
Awkward, but this approach is probably used due to legacy reasons. If no --local flag is used and no repository is specified using -m, West defaults to using the Zephyr repository.

if args.topdir:
topdir = Path(args.topdir)
if args.directory:
if not Path(args.directory).is_relative_to(topdir):
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you just removed abspath() from the earlier version if not Path(abspath(args.directory)).is_relative_to following my comment in #854 (comment) but that did not fix relative directory arguments. Now relative directory arguments all fail because is_relative_to() fails silently with relative arguments:

>>> Path('d1/d2').is_relative_to('/')
False
>>> Path('/d1/d2').is_relative_to('/')
True

It's confusing to have relative arguments not being relative (!) but it's documented in Pathlib. There is a test coverage gap for this use case (and you changed that code without testing manually).

This raises a subtle question: are relative directory arguments relative to topdir, or relative to the current directory? I would say the former but I don't have a strong opinion. Either way it needs to be clearly specified.

f"directory '{args.directory}' must be relative "
f"to west topdir '{args.topdir}'"
)
manifest_subdir = Path(os.path.relpath(args.directory, topdir))
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please avoid os.path unless absolutely necessary. It's complicated enough to deal with one Python module (Pathlib), please don't mix both.

manifest_path = PurePath(urlparse(manifest_url).path).name
mf_path = PurePath(urlparse(manifest_url).path).name

manifest_path: Path = Path(subdir) / mf_path
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be wrong, but for args.directory consistency, I expected these two commands to converge to the exact same state. I just tested and they produce something different right now:

west init -l                      -t ~/tmp/topdir ~/tmp/topdir/extra/level/zephyr/
# ... should end in the same final state as
west init -m http://..../zephyr/  -t ~/tmp/topdir ~/tmp/topdir/extra/level/zephyr/

Once the behavior for absolute paths is finalized, there is the delicate question of relative paths. When args.directory is relative, what is it relative to? Is it relative to topdir? Or is it relative to the current directory where the command is run? Either way, this command always fails right now:

cd ~/tmp/topdir/
west init -m  ....    -t ~/tmp/topdir extra/level
FATAL ERROR: directory 'extra/level' must be relative to west topdir '~/tmp/topdir'

That's because Pathlib is_relative_to() is not intuitive at all, see my other comment about that.

@thorsten-klein
Copy link
Copy Markdown
Contributor Author

thorsten-klein commented Mar 5, 2026

west init -l -t ~/tmp/topdir ~/tmp/topdir/extra/level/zephyr/
... should end in the same final state as
west init -m http://..../zephyr/ -t ~/tmp/topdir ~/tmp/topdir/extra/level/zephyr/

I almost agree (only the last argument has to omit /zephyr):

west init -l                      -t ~/tmp/topdir ~/tmp/topdir/extra/level/zephyr
# ... should end in the same final state as
west init -m http://..../zephyr/  -t ~/tmp/topdir ~/tmp/topdir/extra/level

I have added a test to prove that the commands really result in the same final state (= same tree).

@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Mar 6, 2026

I almost agree (only the last argument has to omit /zephyr):

west init -l                      -t ~/tmp/topdir ~/tmp/topdir/extra/level/zephyr
# ... should end in the same final state as
west init -m http://..../zephyr/  -t ~/tmp/topdir ~/tmp/topdir/extra/level

Can you give some reason(s) why you prefer this variant? Not only it make these two commands above inconsistent with each other, this also makes the second one inconsistent with git clone:

git clone http://..../zephyr/                   ~/tmp/topdir/extra/level
ls ~/tmp/topdir/extra/level/
west.yml README ...

Also, this makes it impossible to rename without an extra rename action after cloning (which is probably why git clone made that choice)

This is not an obscure implementation detail that could easily be changed later, this is literally defining the main user interface of this new feature. After a gazillion of prototypes force-pushed that I manually tested and reviewed :-((( Once that user interface choice is made and people start using this feature, there is no way back (except for a painful deprecation, best avoided).

Steps 3. and 4. should be PAUSED until 1. and 2. are discussed and agreed: PLEASE DO NOT PUSH until they are.

So, in answer to this request of mine, "I prefer this variant" without any elaboration was your only plain English reply AND you force-pushed at least twice, piling up on the already out of control "evolog" and untrackable progress[1,2]. Did you miss or misunderstand what I asked there or is it really impossible to change your approach?

[1] I did not look at these new force pushes yet. I bet even more will come.
[2] compounded by zephyrproject-rtos/zephyr#39194

@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Mar 6, 2026

[1] I did not look at these new force pushes yet. I bet even more will come.

There will be more revisions because the latest version cb9714f fails on Windows:

https://github.com/zephyrproject-rtos/west/actions/runs/22708722014/job/65841352565?pr=854

       flags = _setup_test_init(manifest_repo, local_dir, topdir, directory, manifest_file)
        _, stderr = cmd_raises(['init'] + flags, expected_error)
>       assert expected_stderr in stderr
E       assert "FATAL ERROR: directory 'extra/dir' must be inside west topdir" in "FATAL ERROR: directory 'extra\\dir' must be inside west topdir 'D:\\a\\west\\west\\check out\\somedir'\n"

As a coincidence, I just had to work on similar portability issues and found a way to test Windows without Windows:

@thorsten-klein
Copy link
Copy Markdown
Contributor Author

Can you give some reason(s) why you prefer this variant

I do not prefer this as well. Until now I thought that this was existing behavior. But it seems to be not.
So I agree that it should be as you suggest 👍🏻

@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Mar 9, 2026

Once the behavior for absolute paths is finalized, there is the delicate question of relative paths. When args.directory is relative, what is it relative to? Is it relative to topdir? Or is it relative to the current directory where the command is run?

cd some_dir/
west init .... -t ~/tmp/topdir manifest/dir

Ping? Is it some_dir/manifest/dir or is it ~/tmp/topdir/manifest/dir? The latter "looks" better but the former allows completion... Which one is more consistent with other and existing cases and how?

multiple combinations of arguments are tested as they can be provided to
'west init'.
Added new argument for west init to specify the west topdir when a
workspace is inizialized.
Added new argument for west init to specify the west topdir when a
workspace is inizialized.
@thorsten-klein
Copy link
Copy Markdown
Contributor Author

thorsten-klein commented Mar 25, 2026

I’ve separated the tests into individual commits so it’s easier to verify that existing behavior remains unchanged.
That said, I’m unsure how to proceed with this PR and would appreciate some guidance.

@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Mar 26, 2026

That said, I’m unsure how to proceed with this PR and would appreciate some guidance.

  1. First things first: finish defining the user interface (in the help string). I sincerely hope the question of what relative arguments are relative to is the last question that is lacking an answer, see my "ping" just above. Don't worry about the current test failures yet. Fixing them could be a waste of time before the user interface is completely defined. Especially fixing failures like "X should be inside Y" when relative paths have not been defined yet.

  2. Try to "use it in anger" and try break it by interactively testing all the user inputs you can think of. This is how I kept finding bugs, missing features and unspecified behaviors the whole time. Make a list of edge cases than need test coverage.

  3. As a last step, look at existing test failures and add new tests for any missing coverage. I can probably investigate and help with Windows-specific failures but not before the user interface is completely defined.

assert actual.west_commands == expected.west_commands


def tree(path: Path, prefix="") -> str:
Copy link
Copy Markdown
Collaborator

@marc-hb marc-hb Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think https://anytree.readthedocs.io/en/latest/intro.html provides this for free and it is already a Zephyr requirement.

Don't get me wrong: I am NOT suggesting to add new west dependency but only to add a new west TEST dependency

marc-hb added a commit to marc-hb/west that referenced this pull request Mar 26, 2026
Fix detection of some existing (and very unlikely) .west/ "pollution" at
an intermediate level, for details see first comment at
zephyrproject-rtos#935

This pollution is very unlikely because it's pretty difficult to have
the manifest clone more than one level down before zephyrproject-rtos#774 / zephyrproject-rtos#854 - so it's
difficult to have "intermediate" levels.

Signed-off-by: Marc Herbert <Marc.Herbert@gmail.com>
@marc-hb
Copy link
Copy Markdown
Collaborator

marc-hb commented Mar 26, 2026

I’ve tried with the code to fail if the topdir is already initialized (when using --local). However, it seems that some tests (e.g. test_update_some_with_imports) rely on the current behavior and perform a re-initialization of some already initialized workspace.

I'm amazed that such "reinit" situations exist, is it still the case? If yes, can you elaborate and provide a simple reproduction? This looks like a serious bug.

This was a west init bug (and inconsistency). Filed in #935 and fix submitted in #936

Does it predate this PR?

This bug looks like it has been there forever. No one noticed before you because:

  • users probably don't re-initialize workspaces by mistake much.
  • as long as you nothing had changed, the re-initialization was harmless

Does it simply deletes the existing .west/ directory?

It resets two values in .west/config without even changing the rest.

What shall I do?

Help review #936 :-)

It looks like this behavior might have been intentional, so I make it fail only when the new argument --topdir is used.

It wasn't intentional because it depended on the current directory west init was run from. That's too inconsistent to be intentional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

west init gives wrong config path for a manifest file in nested directories

5 participants