Added support for --bind flag in bundle generate
#3782
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| bundle: | ||
| name: auto-bind-test | ||
|
|
||
| workspace: | ||
| root_path: /tmp/${UNIQUE_NAME} | ||
|
|
||
| include: | ||
| - resources/*.yml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
|
|
||
| === Create a pre-defined job: | ||
| Created job with ID: [NUMID] | ||
|
|
||
| >>> [CLI] workspace mkdirs /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME] | ||
|
|
||
| >>> [CLI] workspace import /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME]/test --file test.py --language PYTHON | ||
|
|
||
| === Generate and bind in one step: | ||
| >>> [CLI] bundle generate job --key test_job --existing-job-id [NUMID] --config-dir resources --source-dir src --bind | ||
| File successfully saved to src/test.py | ||
| Job configuration successfully saved to resources/test_job.job.yml | ||
| Updating deployment state... | ||
| Successfully bound job with an id '[NUMID]' | ||
|
|
||
| >>> ls src/ | ||
| test.py | ||
|
|
||
| >>> cat resources/test_job.job.yml | ||
| resources: | ||
| jobs: | ||
| test_job: | ||
| name: auto-bind-job-[UNIQUE_NAME] | ||
| tasks: | ||
| - task_key: test | ||
| new_cluster: | ||
| azure_attributes: | ||
| availability: ON_DEMAND_AZURE | ||
| enable_elastic_disk: true | ||
| node_type_id: [NODE_TYPE_ID] | ||
| num_workers: 1 | ||
| spark_version: 13.3.x-snapshot-scala2.12 | ||
| email_notifications: {} | ||
| notebook_task: | ||
| notebook_path: ../src/test.py | ||
| source: WORKSPACE | ||
| run_if: ALL_SUCCESS | ||
| timeout_seconds: 0 | ||
| email_notifications: {} | ||
| max_concurrent_runs: 1 | ||
| queue: | ||
| enabled: true | ||
| timeout_seconds: 0 | ||
| webhook_notifications: {} | ||
|
|
||
| === Deploy the bound job: | ||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/tmp/[UNIQUE_NAME]/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| === Destroy the bundle: | ||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete job test_job | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/tmp/[UNIQUE_NAME] | ||
|
|
||
| Deleting files... | ||
| Destroy complete! | ||
|
|
||
| === Check that job is bound and does not exist after bundle is destroyed: | ||
| >>> errcode [CLI] jobs get [NUMID] --output json | ||
| Error: Job [NUMID] does not exist. | ||
|
|
||
| Exit code: 1 | ||
|
|
||
| === Delete the tmp folder: | ||
| >>> [CLI] workspace delete /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME]/test | ||
|
|
||
| >>> [CLI] workspace delete /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| title "Create a pre-defined job:\n" | ||
|
|
||
| PYTHON_NOTEBOOK_DIR="/Workspace/Users/${CURRENT_USER_NAME}/python-${UNIQUE_NAME}" | ||
| PYTHON_NOTEBOOK="${PYTHON_NOTEBOOK_DIR}/test" | ||
|
|
||
| JOB_ID=$($CLI jobs create --json ' | ||
| { | ||
| "name": "auto-bind-job-'${UNIQUE_NAME}'", | ||
| "tasks": [ | ||
| { | ||
| "task_key": "test", | ||
| "new_cluster": { | ||
| "spark_version": "'${DEFAULT_SPARK_VERSION}'", | ||
| "node_type_id": "'${NODE_TYPE_ID}'", | ||
| "num_workers": 1 | ||
| }, | ||
| "notebook_task": { | ||
| "notebook_path": "'${PYTHON_NOTEBOOK}'" | ||
| } | ||
| } | ||
| ] | ||
| }' | jq -r '.job_id') | ||
|
|
||
| echo "Created job with ID: $JOB_ID" | ||
|
|
||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
| cleanup() { | ||
| title "Delete the tmp folder:" | ||
| trace $CLI workspace delete ${PYTHON_NOTEBOOK} | ||
| trace $CLI workspace delete ${PYTHON_NOTEBOOK_DIR} | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| trace $CLI workspace mkdirs "${PYTHON_NOTEBOOK_DIR}" | ||
| trace $CLI workspace import "${PYTHON_NOTEBOOK}" --file test.py --language PYTHON | ||
|
|
||
| title "Generate and bind in one step:" | ||
| trace $CLI bundle generate job --key test_job --existing-job-id $JOB_ID --config-dir resources --source-dir src --bind | ||
| trace ls src/ | ||
| trace cat resources/test_job.job.yml | ||
|
|
||
| title "Deploy the bound job:" | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "Destroy the bundle:" | ||
| trace $CLI bundle destroy --auto-approve | ||
|
|
||
| title "Check that job is bound and does not exist after bundle is destroyed:" | ||
| trace errcode $CLI jobs get "${JOB_ID}" --output json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # Databricks notebook source | ||
| print("Test notebook") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # This test is using a workspace import API to load a notebook file. | ||
| # This API has a logic on how to accept notebook files and distinguishes them from regular python files. | ||
| # To succeed locally we would need to replicate this logic in the fake_workspace | ||
|
Contributor
Author
There was a problem hiding this comment. Certainly, there are a bunch of other tests that need this, worth doing this as a separate PR |
||
| Local = false | ||
| Cloud = true | ||
|
|
||
| Ignore = [ | ||
| "databricks.yml", | ||
| "resources/*", | ||
| "src/*", | ||
| ".databricks", | ||
| ] | ||
|
|
||
| [EnvMatrix] | ||
| DATABRICKS_BUNDLE_ENGINE = ["terraform"] | ||
|
|
||
|
|
||
| [Env] | ||
| # MSYS2 automatically converts absolute paths like /Users/$username/$UNIQUE_NAME to | ||
| # C:/Program Files/Git/Users/$username/UNIQUE_NAME before passing it to the CLI | ||
| # Setting this environment variable prevents that conversion on windows. | ||
| MSYS_NO_PATHCONV = "1" | ||
|
|
||
| [[Repls]] | ||
| Old = '\\' | ||
| New = '/' | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.