--action_env ignored when `cfg = "exec"` is used

[Flamefire]

Description of the bug:

We need to pass some environment variables like "$CPATH" to the compiler when building TensorFlow with Bazel. This is cumbersome itself and has led to hard-to-debug issues like #12059 in the past already.

Now we again see failures caused by action-env values not passed to the compiler invocation in TensorFlow 2.8.4 which I tracked down to tensorflow/tensorflow@07cbc7b which changes cfg = "host" to cfg = "exec"

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Build TensorFlow 2.8.4 with Bazel 4.2.2 passing --action-env=CPATH and observe that it is not passed to some compiler invocations resulting in e.g.:

In file included from bazel-out/k8-opt-exec-50AE0418/bin/tensorflow/core/framework/dataset_options.pb.cc:4:
bazel-out/k8-opt-exec-50AE0418/bin/tensorflow/core/framework/dataset_options.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory
   10 | #include <google/protobuf/port_def.inc>

So can you provide information on how to use --action-env (or similar) in such circumstances?

An explanation on what is actually being done with the change to "exec" from "host" would also be very welcome. In our case we are not cross-compiling so host, target and build machine are all the same.

I would clearly classify this behavior as a bug because the documentation states:

Specifies the set of environment variables available during the execution of all actions.

But obviously there are now actions where those are missing but they should be in "all actions"

Which operating system are you running Bazel on?

REHL 7

[fmeum]

Does passing in CPATH via --host_action_env work? The user manual you link to does seem to be outdated, the full command-line reference mentions that --action_env applies to the target configuration whereas --host_action_env applies to host and exec configuration.

[Flamefire]

Does passing in CPATH via --host_action_env work? The user manual you link to does seem to be outdated

Yes that seems to work. But I guess now I have to duplicate all action_env into host_action_env. We are using --distinct_host_configuration=false which looks like it should work. But now can't find it in the docs.

[fmeum]

@Flamefire --distinct_host_configuration is a no-op at HEAD. Whether to bring it back has already been discussed in the context of Tensorflow at some point, but I can't find the issue anymore. @meteorcloudy who I think was part of that discussion.

[meteorcloudy]

@fmeum I believe it's #14848 (comment)
/cc @gregestren who was also in that discussion.

[Flamefire]

Seems my comment didn't got saved so again the most important bits as multiple people spent days trying to figure out why TF started to fail to compile due to the wrong/unset env variables (I finally found the commit that broke/changed it by bisecting over 2000 commits):

We are in, what should be, the most simple situation: Build a software (TensorFlow) with Bazel on the machine it is supposed to run on. On other build systems it is a variation of configure && make && make install but for Bazel which by default deletes all environment variables we need to pass commandline options to have them set (this is about e.g. $CPATH and $LIBRARY_PATH)

Previously --action_env has mostly worked as it was documented as "environment variables available during the execution of all actions". Above I linked the documentation of Bazel 4.2.2 which is the version of Bazel I've been using. @fmeum Mentioned that this is wrong. As behavior of Bazel changes considerably between versions it would be good to have correct documentation for each specific version and I would count this at least as a bug report against the 4.2.2 documentation if the observed behavior is intended and hence the new documentation (which doesn't seem to be specific to a version?) is correct.

However even before the change in TF it was brittle as the used environment depended on many other factors, for example:

• Use of tools vs exec_tools: Rename exec_tools to tools tensorflow/tensorflow#44901
• Unset use_default_shell_env: Add use_default_shell_env = True to all ctx.actions.run_shell rules tensorflow/tensorflow#44549
• Now the cfg attr

I don't fully understand the difference between "host" and "exec" configurations, they sound very similar. Also the observed behavior surprises me: --action_env applies to "target" configurations per the new docs but reverting the TF commit (so changing cfg = "exec" back to cfg = "host") makes the environment variables be passed. But with cfg = "exec" I need to set them via --host_action_env. Why is that? The last part makes sense per the new docs but why --action_env does apply to "host" cfgs isn't clear to me. Has this changed? Or is this the expected effect of --distinct_host_configuration=false? In which Bazel version will this be a no-op?

Next question is why use_default_shell_env isn't set by default? The name implies that.

And finally I'd like to ask what the "correct" way would be to build something with Bazel on the machine it is to be run i.e. a local, **non-**crosscompilation build.

Things that come to mind:

• Keep/Restore the behavior of --action_env applying to all actions and introduce a separate --target_action_env similar to --host_action_env
• Have a flag like --no-distinct-configurations so that "host", "target" and "exec" are all the same avoiding potential rebuilds and issues like we have seen with unexpected changes to action environments

I would also greatly appreciate an easy way to find out why a specific C++ file is compiled (or library linked) with specific env variables (i.e. in a specific configuration). E.g. it would help if --subcommands would show the configuration name of an action and there would be a way to query how & why a specific file is built. It was very confusing that a library was build with missing env variables but calling bazel build on the seemingly obvious target had the env variables and hence worked. Turned out that the library was build again as part of a dependency of a dependency of a tool which had somewhere cfg = "exec". But I haven't found a way in the documentation to find that from a source tree and/or with bazel. So the only way I found feasible is find the commit which changed the behavior and check those changes.

I hope that helps and Merry Christmas! 🎅

[gregestren]

Generally exec and host should be basically the same. I say basically because exec is intentionally supposed to generalize the concept of 'host. In particular, it's possible to have multiple execution architectures per build. exec supports this, while host doesn't.

I understand you're saying this doesn't matter for your use case, where all architectures are the same.

action_env should pass the same exact way to exec as it does to host. I suspect the reason you're not seeing that is --distinct_host_configuration=false, as you mention.

As @fmeum mentioned, --distinct_host_configuration=false is a no-op after a certain Bazel version (I'd have to check which). For any Bazel after that version I'd expect --action_env to behave equivalent for host and exec.

But if your baseline is an older version where --distinct_host_configuration=false works, that flag makes host an alias for the target configuration. So even though --action_env only applies to the target configuration, that kicks in here. We've never had a way to make exec an alias for the target configuration, hence the difference.

Most important, I think, is if you really need some equivalent of --distinct_host_configuration=false going forward. TensorFlow has argued for that kind of functionality. I think we could revive that by making an exec transition a no-op. And if it works for TensorFlow that's great. But I'd want to double check we're sure that the semantics that implies are okay. Particularly that any flags that should only apply to a host action (like, say, --compilation_mode=opt instead of --compilation_mode=dbg will not trigger in this mode. So there's some fundamental awkwardness there.

[Flamefire]

For any Bazel after that version I'd expect --action_env to behave equivalent for host and exec.

You mean "behave equivalent for" as "have no effect on", correct? As I understood --host_action_env must be used for host and exec.

Most important, I think, is if you really need some equivalent of --distinct_host_configuration=false going forward. TensorFlow has argued for that kind of functionality. I think we could revive that by making an exec transition a no-op.

A small step would be to have --action_env (i.e. no prefix like host_) pass those variables to literally *all actions. Or a flag with that effect, e.g. --all_action_env. Our current workaround is to duplicate --action_env to --host_action_env which makes the commandline huge and may at some point hit a limit.

What would help us even more is to have no such "transitions", i.e. keep all environment variables from the current environment for all actions. We have modules which on load set up paths in $CPATH, $PATH, $LD_LIBRARY_PATH etc. as there are multiple versions of the same software installed side-by-side so only one can be "active" at a time which means not having a central location for all software installations, i.e. not installing everything into /usr or similar.

Particularly that any flags that should only apply to a host action (like, say, --compilation_mode=opt instead of --compilation_mode=dbg will not trigger in this mode. So there's some fundamental awkwardness there.

I don't understand that part. Can you elaborate or provide a link to the docs what flags only apply to some actions and the implications?

[gregestren]

For any Bazel after that version I'd expect --action_env to behave equivalent for host and exec.

You mean "behave equivalent for" as "have no effect on", correct? As I understood --host_action_env must be used for host and exec.

Yes. According to my reading --action_env doesn't propagate to either the host or exec configs. While --host_action_env exclusively applies to the host / exec configs.

The only exception to this (the only place where host and exec could diverge) is when setting --distinct_host_configuration=false on a sufficiently old Bazel version.

A small step would be to have --action_env (i.e. no prefix like host_) pass those variables to literally *all actions. Or a flag with that effect, e.g. --all_action_env. Our current workaround is to duplicate --action_env to --host_action_env which makes the commandline huge and may at some point hit a limit.

What would help us even more is to have no such "transitions", i.e. keep all environment variables from the current environment for all actions. We have modules which on load set up paths in $CPATH, $PATH, $LD_LIBRARY_PATH etc. as there are multiple versions of the same software installed side-by-side so only one can be "active" at a time which means not having a central location for all software installations, i.e. not installing everything into /usr or similar.

Doing some spelunking, #4008 looks pretty relevant - an old bug that looks like it's asking for the same sort of thing. That was fixed by #12122 which led to the current --action_env / --host_action_env split. Which I understand you're saying is insufficient.

@meteorcloudy asked me at #12122 (comment) if we could just propagate --action-env to the host / exec config. I didn't give a direct answer because I don't consider myself an authority on the --action_env feature. I don't feel I fully understand the implications of such a change one way or another.

What we need is a clear owner of the --action_env feature who can evaluate that and recommend. @meteorcloudy any idea who that would be?

Particularly that any flags that should only apply to a host action (like, say, --compilation_mode=opt instead of --compilation_mode=dbg will not trigger in this mode. So there's some fundamental awkwardness there.

I don't understand that part. Can you elaborate or provide a link to the docs what flags only apply to some actions and the implications?

I don't think there's a clear API or documentation for this now. The authoritative source for now is looking at the code: https://cs.opensource.google/search?q=%22FragmentOptions%20getExec%22&sq=&ss=bazel.

For example https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/analysis/config/CoreOptions.java;l=985;drc=3d29b2e14f143cd56dfee4d49d991a6e330f3a85 has the line that propagates --host_action_env.