[Python] Support large pipeline options via file (Fixes #37370) #37379

Mathdee · 2026-01-21T21:34:48Z

Issue:
As described in #37370, running pipelines with large options on Dataflow causes fork/exec /usr/local/bin/python: argument list too long.
This occurs because the bootloader passes the complete JSON config via PIPELINE_OPTIONS environment variable --> exceeds the OS ARG_MAX limit.

The Fix:
This change uses an identical pattern from the Go SDK (Issue #27839, Commit e31e885) to Python.

boot.go(file): Writing the pipeline options to a temp file (pipeline_options.json) and sets the PIPELINE_OPTIONS_FILE environment variable.
sdk_worker_main.py(file): This then checks for PIPELINE_OPTIONS_FILE and loads the configs from the disk if present.

Outcome:

Fixes [Feature Request]: Support large pipeline options in Python SDK #37370
Feature Parity with Java and Go SDK's for handling large pipeline options
Verified with unit tests that ensured priority of file-based loading.

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

gemini-code-assist · 2026-01-21T21:35:05Z

Summary of Changes

Hello @Mathdee, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a robust solution for handling large pipeline options in the Python SDK, which previously led to an 'argument list too long' error due to operating system limitations on environment variable size. By transitioning from direct environment variable passing to a file-based approach, where options are written to a temporary file and its path is referenced, the system can now accommodate extensive configurations without encountering runtime failures. This enhancement ensures greater stability and consistency across SDKs.

Highlights

Addresses Argument List Too Long Error: This pull request resolves the fork/exec /usr/local/bin/python: argument list too long error that occurs when running Dataflow pipelines with excessively large pipeline options, by preventing the complete JSON config from being passed directly via the PIPELINE_OPTIONS environment variable.
File-based Pipeline Options: Pipeline options are now written to a temporary file named pipeline_options.json by the Go bootloader. The path to this file is then passed to the Python SDK worker via the PIPELINE_OPTIONS_FILE environment variable.
Python SDK Worker Adaptation: The Python SDK worker (sdk_worker_main.py) has been updated to check for the PIPELINE_OPTIONS_FILE environment variable and load the pipeline configuration from the specified file if present, ensuring compatibility with the new mechanism.
Feature Parity: This change brings the Python SDK into feature parity with the Java and Go SDKs for handling large pipeline options, using an identical pattern previously implemented in the Go SDK.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

Mathdee · 2026-01-22T00:02:40Z

All linting and integration checks are passing. Ready for review

github-actions · 2026-01-22T00:33:54Z

Assigning reviewers:

R: @claudevdm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

claudevdm · 2026-01-23T17:58:22Z

@shunping can you take a pass at this since you have more context?

shunping · 2026-01-24T01:44:45Z

Ack! Will do

shunping · 2026-01-28T16:48:07Z

sdks/python/container/boot.go

+// Assumes the options string is JSON formatted.
+//
+// Stores the file name in question in PIPELINE_OPTIONS_FILE for access by the SDK.
+func MakePipelineOptionsFileAndEnvVar(options string) (string, error) {


Any reason to redefine this function? I see we already have that at

beam/sdks/go/container/tools/pipeline_options.go

Line 28 in 928ad7e

func MakePipelineOptionsFileAndEnvVar(options string) error {

, and we already imported the Go package in this boot.go at

beam/sdks/python/container/boot.go

Line 38 in 928ad7e

"github.com/apache/beam/sdks/v2/go/container/tools"

shunping · 2026-01-28T16:56:56Z

sdks/python/container/boot.go

+	if optionsFile, err := MakePipelineOptionsFileAndEnvVar(options); err != nil {
+		logger.Fatalf(ctx, "Failed to create a pipeline options file: %v", err)
+	} else {
+		os.Setenv("PIPELINE_OPTIONS_FILE", optionsFile)


In either the original MakePipelineOptionsFileAndEnvVar or your own copy (which seems to be unnecessary), the environment variable is already set. I think we don't need to do that again here.

shunping · 2026-01-28T16:58:00Z

sdks/python/apache_beam/runners/worker/sdk_worker_main.py

+
+  pipeline_options_dict = _load_pipeline_options(options_json)
+
+  #pipeline_options_dict = _load_pipeline_options(


Remove unnecessary comments.

shunping · 2026-01-28T17:02:03Z

sdks/python/container/boot.go

 	// (3) Invoke python

-	os.Setenv("PIPELINE_OPTIONS", options)
+	//Commented out -->


Clean up comments

shunping · 2026-01-28T17:04:21Z

sdks/python/container/boot.go

+	//Commented out -->
+	//os.Setenv("PIPELINE_OPTIONS", options)
+
+	//To prevent crashes if options > ARG_MAX -->


Could you follow the comment style of this file to polish your comment here?

Something like:

// Write the JSON string of pipeline options into a file to prevent `argument list too long` error.

shunping · 2026-01-28T18:19:43Z

sdks/python/container/boot.go

+	// --> Write the large JSON content into a file on disk
+	// --> environment only receives the short filename
+	if optionsFile, err := MakePipelineOptionsFileAndEnvVar(options); err != nil {
+		logger.Fatalf(ctx, "Failed to create a pipeline options file: %v", err)


For consistency, could we use the same error message as Java and GO Sdks?

beam/sdks/go/container/boot.go

Lines 198 to 199 in 928ad7e

if err := tools.MakePipelineOptionsFileAndEnvVar(options); err != nil {

logger.Fatalf(ctx, "Failed to load pipeline options to worker: %v", err)

beam/sdks/java/container/boot.go

Lines 126 to 127 in 928ad7e

if err := tools.MakePipelineOptionsFileAndEnvVar(options); err != nil {

logger.Fatalf(ctx, "Failed to load pipeline options to worker: %v", err)

shunping

Thanks for submitting the PR. Please check out my inline comments.

…ge with JAVA and GO SDKs

Mathdee · 2026-01-30T00:16:19Z

@shunping I saw the re-run failed again, so I dug into the logs. It looks like it's hitting the exact same timeout (DEADLINE_EXCEEDED) in test_builtin_combines.
Since that test is totally unrelated to the pipeline options logic (and the other 7,610 tests passed), this seems like a flake in the test environment.
Is there a way to merge despite this, or should we try triggering it one more time?

shunping · 2026-01-30T00:19:02Z

Let's give it another try. :)

Mathdee · 2026-01-30T00:56:00Z

All checks passed! Ready for final review/merge when you are. Thanks again @shunping.

Support large pipeline options in Python SDK apache#37370

9c5d868

github-actions bot added python docker runners labels Jan 21, 2026

Mathdee added 4 commits January 21, 2026 17:06

reformatted code to match the project's standards to pass checks

0707d08

Fixed PythonFormatterPreCommitscript error"

258ddf8

Fix import order: move google.protobuf above apache_beam

6a9cd2f

Removed blank line between import and google import

937fd54

github-actions bot added the Next Action: Reviewers label Jan 22, 2026

shunping reviewed Jan 28, 2026

View reviewed changes

shunping requested changes Jan 28, 2026

View reviewed changes

shunping added Next Action: Author and removed Next Action: Reviewers labels Jan 28, 2026

Addressed the review comments: cleaned up comments, align error messa…

f1aa586

…ge with JAVA and GO SDKs

github-actions bot added Next Action: Reviewers and removed Next Action: Author labels Jan 28, 2026

Mathdee added 4 commits January 28, 2026 15:36

Fixed Formatting Error

171cfad

Fixed line lenght that causes lint error

4666684

rerun tests

e64900c

Rerun tests

86cfb0c


		pipeline_options_dict = _load_pipeline_options(options_json)

		#pipeline_options_dict = _load_pipeline_options(

	if err := tools.MakePipelineOptionsFileAndEnvVar(options); err != nil {
	logger.Fatalf(ctx, "Failed to load pipeline options to worker: %v", err)

[Python] Support large pipeline options via file (Fixes #37370) #37379

Are you sure you want to change the base?

[Python] Support large pipeline options via file (Fixes #37370) #37379

Conversation

Mathdee commented Jan 21, 2026

Uh oh!

gemini-code-assist bot commented Jan 21, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

Mathdee commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 22, 2026

Uh oh!

claudevdm commented Jan 23, 2026

Uh oh!

shunping commented Jan 24, 2026

Uh oh!

shunping Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shunping Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

shunping Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

shunping Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

shunping Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

shunping Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

shunping left a comment

Choose a reason for hiding this comment

Uh oh!

Mathdee commented Jan 30, 2026

Uh oh!

shunping commented Jan 30, 2026

Uh oh!

Mathdee commented Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shunping Jan 28, 2026 •

edited

Loading