support running and debugging pytest for local tree

[thorsten-klein]

Why?

I'm not sure how tests for the local source tree are currently debugged.
In my setup, I’m unable to run pytest directly on the local tree, and debugging is difficult because it’s not possible to step through since subprocess is used in the current test setup.

Proposed Changes

This change allows running pytest directly, instead of only being able to run it via uv run poe test.
pytest can test either the installed West package or the local source tree, depending on how it’s invoked.
This simplifies test execution and improves compatibility with pytest integrations (e.g. in IDEs).

With this setup, pytest is fully in charge for setting up the Python environment, so testing becomes straightforward:

# Test the installed West package
pytest

# Test the local source tree
pytest -o pythonpath=src
# or
PYTHONPATH=src pytest

Background

During this work, I faced that many tests used subprocess, which interfered with testing the local copy.
Subprocesses do not inherit the full Python environment configured by pytest, causing a mix between modules from the installed West package and those from the local source tree. Therefore I removed the use of subprocesses in tests wherever possible. Where subprocesses are still used, the python module (west/app/main.py) is called instead of west whereby it is ensured now that its correct modules are used by prepending correct PYTHONPATH. With those changes only one Python environment managed by pytest is used, ensuring consistent behavior and enabling proper debugging.

Additional infos

cmd() captures only Python-level stdout and does not include output from any subprocesses launched internally.

If the code under test starts one or more subprocesses and the test verifies this combined stdout, the older cmd_subprocess() behavior must be used, as this function captures all stdout, both from Python itself and from any spawned subprocesses.

To simplify invoking west with corresponding modules, the main.py script can now be executed directly.
This approach is used in tests by cmd_subprocess(), but also end users can also invoke main.py directly in their local tree if needed.

[pdgendt]

I'm not sure I like where this is going. Testing west should focus on the package. I think we could add some documentation bits for people who really, really want to use pytest directly that they should install an editable version with

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install -e .

[thorsten-klein]

Note: A few other tests invoke the hardcoded west executable directly, rather than using cmd().
As a result, they only work if west is installed and available in PATH.

For consistency and robustness, these tests should be refactored so that pytest passes even when west is not installed.
Would you like me to address this in the current PR, or handle it in a follow-up PR?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.40%. Comparing base (50cbc34) to head (26fd91d).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #859      +/-   ##
==========================================
- Coverage   84.41%   84.40%   -0.02%     
==========================================
  Files          11       11              
  Lines        3382     3385       +3     
==========================================
+ Hits         2855     2857       +2     
- Misses        527      528       +1     

Files with missing lines	Coverage Δ
src/west/app/main.py	75.91% <100.00%> (+0.31%)	⬆️

... and 1 file with indirect coverage changes

[pdgendt]

I recommend you check https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code too.

[thorsten-klein]

I'm not sure I like where this is going. Testing west should focus on the package. I think we could add some documentation bits for people who really, really want to use pytest directly that they should install an editable version with

Installation in editable mode does not work for me:

ERROR: Project file:///home/user/work/GIT/west has a 'pyproject.toml' and
its build backend is missing the 'build_editable' hook. Since it does not have
a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode.
Consider using a build backend that supports PEP 660.

Am I doing something wrong?

I recommend you check https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code too.

In this documentation it says:

If you don’t use an editable install and are relying on the fact that Python by default 
puts the current directory in sys.path to import your package, you can execute 
python -m pytest to execute the tests against the local copy directly, without using pip.

So this is exactly what I want to do. I want to run python -m pytest and execute the tests against the local copy.
When the tests are run in tox, then they run against the installed package.
Therefore I adapted the tests to prefer the installed package (if west in correct version exists), and otherwise fall back to the local copy (main.py).

In many other Python open-source projects, it’s possible to run pytest directly, and I find that very convenient.

if you ran pip install -e . before, the installed west executable will probably be used, while not desired.

If the installed west version matches, then the tests should test against this installed package.
So when installed with pip install -e . the west version should always be up-to-date.

The main problem seems to lay in current tests that call subprocess, which does not inherit the sys.path from the test environment. Therefore pytest does not work as expected, since different west modules are imported in the test versus the called west process.

Running pytest directly becomes particularly useful when debugging in an IDE, where for example some paths from ~/.bashrc are not set when IDE is started via GUI.
Currently, I have to either start my IDE inside a prepared virtual environment or I have to install the tool I want to test into my IDE environment. That feels wrong.

---

Out of curiosity: Have you ever considered distinguishing between different test levels and handling them different (e.g., unit tests vs. integration tests)? (Does not matter)

[pdgendt]

Installation in editable mode does not work for me:

ERROR: Project file:///home/user/work/GIT/west has a 'pyproject.toml' and
its build backend is missing the 'build_editable' hook. Since it does not have
a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode.
Consider using a build backend that supports PEP 660.

Am I doing something wrong?

So what I do to test (requires an environment!):

$ python3.10 -m venv .venv
$ source .venv/bin/activate
$ pip install -e .

In many other Python open-source projects, it’s possible to run pytest directly, and I find that very convenient.

I get that, but I also want to make sure developers are running the version they're expecting to run. Maybe we should check POE_ACTIVE and in that case require an installed version.

Running pytest directly becomes particularly useful when debugging in an IDE, where for example some paths from ~/.bashrc are not set when IDE is started via GUI. Currently, I have to either start my IDE inside a prepared virtual environment or I have to install the tool I want to test into my IDE environment. That feels wrong.

Right, but I'd like to avoid having to mess with the system path if possible. There should be some recommended practices for this, no? There are so many packages out there, we can't be the only with this problem.

I guess that's why the myriad of task runners exist.

[thorsten-klein]

Right, but I'd like to avoid having to mess with the system path if possible. There should be some recommended practices for this, no? There are so many packages out there, we can't be the only with this problem.

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()?
This is at least how conan seems to do it:
Ref:

• https://github.com/conan-io/conan/blob/develop2/test/integration/command/test_build.py (using TestClient.run() which is analog to west's cmd())
• https://github.com/conan-io/conan/blob/develop2/conan/test/utils/tools.py#L569 (implementation of TestClient.run calls _run_cli() )
• https://github.com/conan-io/conan/blob/develop2/conan/test/utils/tools.py#L553 (implementation of the _run_cli method without any subprocess)

With this, pytest can completely ensure which modules are tested, because there are no other processes that have different PYTHONPATH.
This is also the benefit of the src layout, as described in your mentioned link (https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code)

pytest # test installed package
PYTHONPATH=src pytest # test local package

The benefit of this would also be that the tests become fully debuggable.
Currently. subprocess cannot be fully debugged, since the whole west logic happens in this other process, which is only one python call.

[pdgendt]

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()? This is at least how conan seems to do it: Ref:

Yes please, but I guess that this is even more work than the current proposal?

With this, pytest can completely ensure which modules are tested, because there are no other processes that have different PYTHONPATH. This is also the benefit of the src layout, as described in your mentioned link (docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code)

pytest # test installed package
PYTHONPATH=src pytest # test local package

The benefit of this would also be that the tests become fully debuggable. Currently. subprocess cannot be fully debugged, since the whole west logic happens in this other process, which is only one python call.

Getting rid of the subprocess would be nice, we can also get rid of the patch for coverage.

[thorsten-klein]

I will give it a try 👍

[mbolivar]

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()?

Just FYI that's technically an abstraction violation; nothing in west.app is API

[pdgendt]

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()?

Just FYI that's technically an abstraction violation; nothing in west.app is API

But that doesn't mean we can't rely on it for testing, right? From a practical point of view it would solve some of the issues we have with testing/coverage.

[mbolivar]

No, it's just white-box testing

[thorsten-klein]

I use main.main() now for testing, which takes argv as argument. This argv is the west command line API which should be stable, so it can be used for testing 😇

[marc-hb]

These changes allow running pytest directly, simplifying test execution and making debugging even easier.

Shorter commands are always nicer but it's IMHO not enough to justify a PR that big. Is there some bigger problem that this PR solves for you? I remember I used to struggle to use pudb with tox but now this seems enough for me:

uv run pytest -k narrow -s

[marc-hb]

Is there some bigger problem that this PR solves for you?

I scanned the comments and found a couple problem descriptions that should have been in commit messages. And then some more:

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()?

Yes please, but I guess that this is even more work than the current proposal?

Prototyping and experimenting is great, but once a PR leaves the draft status it should be clear 1. what is the problem solved 2. why and how that solution was chosen.

For earlier stage discussions please use drafts, issues, discord,...

Some good inspiration:
https://github.com/zephyrproject-rtos/zephyr/blob/67435e394f0d025b8/.github/ISSUE_TEMPLATE/002_enhancement.yml

[pdgendt]

Running with uv also results in the same errors.

EDIT: I actually have this on main 🤔 need to look into this.

@thorsten-klein see #862

[Copilot]

Pull Request Overview

Enable running and debugging pytest directly against the local source tree without relying on subprocess-based invocations, improving IDE integration and consistency of environment handling.

• Replace subprocess-based test helpers with in-process CLI invocation (west.app.main), adding a subprocess variant only where needed
• Update tests to expect SystemExit, capture stderr explicitly, and add coverage for module execution and forall env vars
• Adjust main entrypoint to prepend src to sys.path when run as a script/module; update coverage paths and README instructions

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/test_project.py	Replace CalledProcessError with SystemExit, switch to cmd_subprocess where necessary, add test_forall_env_vars, and separate stderr assertions.
tests/test_main.py	Add tests for version output via in-process and subprocess calls; add module-run test verifying sys.path injection and exit code.
tests/test_help.py	Simplify help output checks by removing platform newline normalization (now captured consistently).
tests/test_config.py	Migrate exception expectations from CalledProcessError to SystemExit and update cmd_raises usage.
tests/test_alias.py	Update exception expectations, but assertions inspect exit code instead of error output.
tests/conftest.py	Introduce _cmd, cmd, cmd_raises, cmd_subprocess helpers; capture stdout/stderr; run west.app.main directly; optional subprocess mode.
src/west/app/main.py	Prepend src to sys.path when executed as main to ensure local modules are used.
pyproject.toml	Fix coverage omit globs to match paths nested under any directory.
README.rst	Document how to run pytest against installed package vs local source using pythonpath=src.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[thorsten-klein]

Gentle ping :-)

I have adopted the PR description to explain the "Why".

Is there anything else you’d like me to change on this branch?
I’d really appreciate if this PR could be merged soon, as it allows me to fully debug the tests.

[pdgendt]

Needs a rebase

[thorsten-klein]

Needs a rebase

Done 👍🏻

[carlescufi]

This looks reasonable to me altogether. I don't love the addition to main.py, but I have read the comments and it seems the better way of handling this.

[marc-hb]

I have no problem with the general idea but I have a few requests, see below.

Also, as the original author, I think we really need @mbolivar to approve the general idea. Not to review every line but at least approve the conceptual changes.

Finally, I didn't have time to check every code change yet, sorry.

[marc-hb]

Suggested change

	cmd = cmd.split() if isinstance(cmd, str) else cmd
	cmd = cmd if isinstance(cmd, list) else cmd.split()

2 reasons:

• more obvious that list is the preferred option
• if cmd is neither list or str by mistake, fail faster.

[marc-hb]

Please mention briefly that this is best avoided because it makes it impossible or harder to use a debugger, etc. That's the one of the main purposes of this PR!

[marc-hb]

Naive question sorry: isn't there some logging function around here?

[thorsten-klein]

I don't think so. There are print statements at other places as well in conftest.py.

[marc-hb]

stderr and stdout always race with each other, so this is a bad idea. Was it already like this? Either way, this deserves an elaboration: explaining that it is generally bad, then explaining why it is not too bad in this particular context.

[thorsten-klein]

It was always this case, and I think the "racing" is not problematic as west does not invoke async subprocesses.
I will adapt the text little bit to point out the rare case when this function shall be used.

[marc-hb]

This split between stdout and stderr looks unrelated to this commit. Can you please submit it in a separate PR?

[thorsten-klein]

I have adapter the refactored cmd() function to capture combined stdout and stderr.

[marc-hb]

The comment must explain what is the effect of this when people run west from a pip3 pipx/uv/uvx install or from their distro.

sys.path is security-sensitive, all use cases must be carefully considered.

You made this a separate commit. I love smaller commits and smaller PRs [*] but I wonder what happens when I try to use the previous, big refactoring without this commit?

[*] https://docs.zephyrproject.org/latest/contribute/contributor_expectations.html#defining-smaller-prs

[thorsten-klein]

There is no effect when people run west. As the comment says it has only an effect when running this script. I will adapt the comment little bit to point out that it only affects if running this script directly

[marc-hb]

This is apparently not enough, see fixup:

• conftest.py: test the local west, not the installed one #883

[marc-hb]

Please correct me: this 3rd commit and test addition is only to test the previous, sys.path commit? In other words, this is test code to test... code required for testing? Test coverage is great but this feels overkill... if the previous commit is really required for testing, then it's already going to be covered regularly, no? If not, then maybe that previous commit is not required that much.

Please correct me.

[thorsten-klein]

Yes, this is just a test for the second commit.

Without the second commit, the cmd_subprocess does not use the correct local modules.

I am not sure if I see this as a test code. With this change also users can invoke west by running the main.py script directly. No need for them to install west or setting PYTHONPATH.

I will add this to the PR description.

out of date

[pdgendt]

I have no problem with the general idea but I have a few requests, see below.

Also, as the original author, I think we really need @mbolivar to approve the general idea. Not to review every line but at least approve the conceptual changes.

Ping @mbolivar, we have the necessary approvals, but I agree with @marc-hb, I'd like to know your opinion.

[mbolivar]

@pdgendt if you're happy with it, I don't object. I did it this way in large part because of #149 -- the global state made it really tricky to try to do everything from one pytest process

[marc-hb]

Is #149 not a problem anymore? AFAICT it's still open because of west.log.

[mbolivar]

Is #149 not a problem anymore? AFAICT it's still open because of west.log

You're going to hate this answer...

At least where tests are concerned, west.log isn't a problem, because we have no tests for it in west/tests :)

[marc-hb]

Thanks! Considering it's deprecated, fine by me. It bet its logic was not very complicated and not crying for test coverage either.