Skip to content

Fix processing short pip flags in environment dependencies #3708

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrewnester merged 6 commits into from
Oct 3, 2025
Merged

Fix processing short pip flags in environment dependencies #3708

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0bf34b0
Fix processing short pip flags in environment dependencies
andrewnester Oct 2, 2025
3013c11
changelog
andrewnester Oct 2, 2025
f62f9e3
handle local path
andrewnester Oct 2, 2025
5df9ec5
local to remote translation for pip flags
andrewnester Oct 2, 2025
fe23869
fix
andrewnester Oct 2, 2025
fa7327c
fixes
andrewnester Oct 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions NEXT_CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,7 @@
### Dependency updates

### Bundles
* Fix processing short pip flags in environment dependencies ([#3708](https://github.com/databricks/cli/pull/3708))
Copy link
Contributor

@pietern pietern Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR now also handles relative paths for -e. We can include additional entries for the same PR, for different user-visible changes (eg call out -e specifically).

Copy link
Contributor Author

@andrewnester andrewnester Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed here fa7327c

* Add support for referencing local files in -e pip flag for environment dependencies ([#3708](https://github.com/databricks/cli/pull/3708))

### API Changes
3 changes: 3 additions & 0 deletions acceptance/bundle/environments/dependencies/databricks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ resources:
client: "1"
dependencies:
- "-r ./requirements.txt"
- "-e ./file.py"
- "-i http://myindexurl.com"
- "--index-url http://myindexurl.com"
- "test_package"
- "test_package==2.0.1"
- "test_package>=2.0.1"
Expand Down
Empty file added acceptance/bundle/environments/dependencies/file.py
View file
Open in desktop
Empty file.
6 changes: 6 additions & 0 deletions acceptance/bundle/environments/dependencies/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ Deployment complete!
"client": "1",
"dependencies": [
"-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
"-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
"-i http://myindexurl.com",
"--index-url http://myindexurl.com",
"test_package",
"test_package==2.0.1",
"test_package>=2.0.1",
Expand Down Expand Up @@ -79,6 +82,9 @@ Deployment complete!
"client": "1",
"dependencies": [
"-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
"-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
"-i http://myindexurl.com",
"--index-url http://myindexurl.com",
"test_package",
"test_package==2.0.1",
"test_package>=2.0.1",
Expand Down
25 changes: 14 additions & 11 deletions bundle/config/mutator/normalize_paths.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (

"github.com/databricks/cli/bundle"
"github.com/databricks/cli/bundle/config/mutator/paths"
"github.com/databricks/cli/bundle/libraries"
"github.com/databricks/cli/libs/diag"
"github.com/databricks/cli/libs/dyn"
)
Expand Down Expand Up @@ -85,18 +86,20 @@ func collectGitSourcePaths(b *bundle.Bundle) []dyn.Path {
}

func normalizePath(path string, location dyn.Location, bundleRootPath string) (string, error) {
// Handle requirements file paths with -r flag
reqPath, ok := strings.CutPrefix(path, "-r ")
if ok {
// Normalize the path part
reqPath = strings.TrimSpace(reqPath)
normalizedPath, err := normalizePath(reqPath, location, bundleRootPath)
if err != nil {
return "", err
}
// Handle local file paths used inside pip flags
for _, flag := range libraries.PipFlagsWithLocalPaths {
reqPath, ok := strings.CutPrefix(path, flag+" ")
if ok {
// Normalize the path part
reqPath = strings.TrimSpace(reqPath)
normalizedPath, err := normalizePath(reqPath, location, bundleRootPath)
if err != nil {
return "", err
}

// Reconstruct the path with -r flag
return "-r " + normalizedPath, nil
// Reconstruct the path with -r flag
return flag + " " + normalizedPath, nil
}
}

pathAsUrl, err := url.Parse(path)
Expand Down
8 changes: 8 additions & 0 deletions bundle/config/mutator/normalize_paths_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,14 @@ func TestNormalizePath_requirementsFile(t *testing.T) {
assert.Equal(t, "-r requirements.txt", value)
}

func TestNormalizePath_environmentDependency(t *testing.T) {
tmpDir := t.TempDir()
location := dyn.Location{File: filepath.Join(tmpDir, "resources", "job_1.yml")}
value, err := normalizePath("-e ../file.py", location, tmpDir)
assert.NoError(t, err)
assert.Equal(t, "-e file.py", value)
}

func TestLocationDirectory(t *testing.T) {
loc := dyn.Location{File: "file", Line: 1, Column: 2}
dir, err := locationDirectory(loc)
Expand Down
4 changes: 2 additions & 2 deletions bundle/config/mutator/paths/job_libraries_paths_visitor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,9 @@ func jobLibrariesRewritePatterns() []jobRewritePattern {
dyn.Key("dependencies"),
dyn.AnyIndex(),
),
TranslateModeEnvironmentRequirements,
TranslateModeEnvironmentPipFlag,
func(s string) bool {
_, ok := libraries.IsLocalRequirementsFile(s)
_, _, ok := libraries.IsLocalPathInPipFlag(s)
Copy link
Contributor

@pietern pietern Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make the same change for pipeline environment dependencies?

Copy link
Contributor

@pietern pietern Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't handle any flags there yet, so could be a separate follow-up PR with dedicated changelog entry.

Copy link
Contributor Author

@andrewnester andrewnester Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll do this as a follow-up for this PR

return !ok
},
},
Expand Down
4 changes: 2 additions & 2 deletions bundle/config/mutator/paths/translation_mode.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,6 @@ const (
// This allows for disambiguating between paths and PyPI package names.
TranslateModeLocalRelativeWithPrefix

// TranslateModeEnvironmentRequirements translates a local requirements file path to be absolute.
TranslateModeEnvironmentRequirements
// TranslateModeEnvironmentPipFlag translates a local file path in a pip flag to be absolute.
TranslateModeEnvironmentPipFlag
)
10 changes: 6 additions & 4 deletions bundle/config/mutator/translate_paths.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,10 @@ func (t *translateContext) rewritePath(
opts translateOptions,
) (string, error) {
// If the input is a local requirements file, we need to translate it to an absolute path.
if reqPath, ok := libraries.IsLocalRequirementsFile(input); ok {
var flag string
if reqPath, flagPrefix, ok := libraries.IsLocalPathInPipFlag(input); ok {
input = reqPath
flag = flagPrefix
}

// We assume absolute paths point to a location in the workspace
Expand Down Expand Up @@ -160,10 +162,10 @@ func (t *translateContext) rewritePath(
interp, err = t.translateLocalRelativePath(ctx, input, localPath, localRelPath)
case paths.TranslateModeLocalRelativeWithPrefix:
interp, err = t.translateLocalRelativeWithPrefixPath(ctx, input, localPath, localRelPath)
case paths.TranslateModeEnvironmentRequirements:
case paths.TranslateModeEnvironmentPipFlag:
interp, err = t.translateFilePath(ctx, input, localPath, localRelPath)
// Add the -r flag to the path to indicate it's a requirements file used for environment dependencies.
interp = "-r " + interp
// Add the flag prefix to the path to indicate it's a local file used in pip flags for environment dependencies.
interp = flag + " " + interp
default:
return "", fmt.Errorf("unsupported translate mode: %d", opts.Mode)
}
Expand Down
36 changes: 20 additions & 16 deletions bundle/libraries/local_path.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,17 +58,11 @@ func IsLibraryLocal(dep string) bool {
}
}

// If the dependency starts with --, it's a pip flag option which is a valid
// entry for environment dependencies but not a local path
if containsPipFlag(dep) {
return false
}

// If the dependency is a requirements file, it can either be a local path or a remote path.
// Even though the path to requirements.txt can be local we don't return true in this function anyway
// If the dependency starts with - or --, it's a pip flag option which is a valid
// entry for environment dependencies. Even though the path in the flag can be local we don't return true in this function anyway
// and don't treat such path as a local library path.
// Instead we handle translation of these paths in a separate code path in TranslatePath mutator.
if strings.HasPrefix(dep, "-r") {
if containsPipFlag(dep) {
return false
}

Expand All @@ -80,18 +74,28 @@ func IsLibraryLocal(dep string) bool {
return IsLocalPath(dep)
}

func IsLocalRequirementsFile(dep string) (string, bool) {
dep, ok := strings.CutPrefix(dep, "-r ")
if !ok {
return "", false
var PipFlagsWithLocalPaths = []string{
"-r",
"-e",
}

func IsLocalPathInPipFlag(dep string) (string, string, bool) {
for _, flag := range PipFlagsWithLocalPaths {
dep, ok := strings.CutPrefix(dep, flag+" ")
Copy link
Contributor

@pietern pietern Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dep aliases the input. It works because of := but is error-prone. Recommend to use a different name.

if ok {
dep = strings.TrimSpace(dep)
return dep, flag, IsLocalPath(dep)
}
}

dep = strings.TrimSpace(dep)
return dep, IsLocalPath(dep)
return "", "", false
}

func containsPipFlag(input string) bool {
re := regexp.MustCompile(`--[a-zA-Z0-9-]+`)
// Trailing space means the the flag takes an argument or there's multiple arguments in input
// Alternatively it could be a flag with no argument and no space after it
// For example: -r myfile.txt or --index-url http://myindexurl.com or -i
re := regexp.MustCompile(`--?[a-zA-Z0-9-]+(\s|$)`)
Copy link
Contributor

@pietern pietern Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This made me realize we also need ^ on the regex (or ^\s*), or we risk also matching foo-bar.

While unlikely to be a valid filename, it is never a flag.

Copy link
Contributor Author

@andrewnester andrewnester Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true, it should be ^\s* indeed, will add an use case for this

return re.MatchString(input)
}

Expand Down
8 changes: 7 additions & 1 deletion bundle/libraries/local_path_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@ func TestIsLibraryLocal(t *testing.T) {
{path: "s3://mybucket/path/to/package", expected: false},
{path: "dbfs:/mnt/path/to/package", expected: false},
{path: "beautifulsoup4", expected: false},
{path: "-e some/local/path", expected: false},
Copy link
Contributor

@lennartkats-db lennartkats-db Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some more tricky cases that we should cover here :-/ AI might help identify and handle them

-i http://myindexurl.com
--trusted-host pypi.org
--platform win_amd64
-e ${workspace.file_path}
-e /Workspace/path/to
-e /Volumes/path/to

Btw, Andrew, please consider how to incorporate such cases. It could be that what you have here is just net-better than what we have now and that we should have a followup with refinements. It could also be that we just need to revise this PR to cover things like -i properly, also to make sure we don't regress anything.

Copy link
Contributor Author

@andrewnester andrewnester Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have quite a good coverage for this overall, but I've added some additional cases for unit / acceptacne tests

{path: "-i http://myindexurl.com", expected: false},
{path: "--index-url http://myindexurl.com", expected: false},
{path: "-i", expected: false},
{path: "--index-url", expected: false},
{path: "-i -e", expected: false},

// Check the possible version specifiers as in PEP 440
// https://peps.python.org/pep-0440/#public-version-identifiers
Expand Down Expand Up @@ -129,7 +135,7 @@ func TestIsLocalRequirementsFile(t *testing.T) {

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, isLocal := IsLocalRequirementsFile(tt.input)
got, _, isLocal := IsLocalPathInPipFlag(tt.input)
require.Equal(t, tt.expected, got)
require.Equal(t, tt.isLocal, isLocal)
})
Expand Down
Loading