Skip to content

Fix GCC OOM when building by reducing parallelism.#8592

Closed
kgpai wants to merge 1 commit into
facebookincubator:mainfrom
kgpai:reduce_parallelism_2
Closed

Fix GCC OOM when building by reducing parallelism.#8592
kgpai wants to merge 1 commit into
facebookincubator:mainfrom
kgpai:reduce_parallelism_2

Conversation

@kgpai
Copy link
Copy Markdown
Contributor

@kgpai kgpai commented Jan 30, 2024

Why

Builds OOM with the following error : https://app.circleci.com/pipelines/github/facebookincubator/velox/43449/workflows/0ecff160-1cb9-4084-9f1e-21ae5b484321/jobs/302064 . We reduced the parallelism recently but dint reduce it for all jobs (#8530) - In this PR we set it for all debug jobs (which are most likely to OOM )

How
We reduce parallelism as a stop gap to fix this issue and unblock folks. We have an issue here for a long term fix : #8539

@netlify
Copy link
Copy Markdown

netlify Bot commented Jan 30, 2024

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 6471d22
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/65b8404ea69cb90008c66891

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 30, 2024
Copy link
Copy Markdown
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should get to the bottom of this. This change could increase CI time.

Copy link
Copy Markdown
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kgpai can we only reduce the MAX_LINK_JOBS ?

@kgpai
Copy link
Copy Markdown
Contributor Author

kgpai commented Jan 30, 2024

@majetideepak
It does not look like it failed in link step no :

/usr/bin/ccache /opt/rh/gcc-toolset-9/root/bin/g++  -DBOOST_ALL_NO_LIB -DBOOST_CONTEXT_DYN_LINK -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_REGEX_DYN_LINK -DBOOST_SYSTEM_DYN_LINK -DBOOST_THREAD_DYN_LINK -DFOLLY_HAVE_INT128_T=1 -DGFLAGS_IS_A_DLL=0 -DSIMDJSON_THREADS_ENABLED=1 -I../../. -I../../velox/external/xxhash -I_deps/xsimd-src/include -I_deps/simdjson-src/include -isystem ../../velox -isystem ../../velox/external -isystem /usr/include/libdwarf -mavx2 -mfma -mavx -mf16c -mlzcnt -std=c++17 -mbmi2 -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qualifiers        -Wno-implicit-fallthrough          -Wno-empty-body          -Wno-class-memaccess          -Wno-comment          -Wno-int-in-bool-context          -Wno-redundant-move          -Wno-array-bounds          -Wno-maybe-uninitialized          -Wno-unused-result          -Wno-format-overflow          -Wno-strict-aliasing          -Wno-type-limits          -Wno-stringop-overflow          -Wno-stringop-overread          -Wno-return-type -Werror -g -fPIC   -fdiagnostics-color=always -Wno-deprecated-declarations -std=gnu++17 -MD -MT velox/functions/prestosql/registration/CMakeFiles/velox_functions_prestosql.dir/ArithmeticFunctionsRegistration.cpp.o -MF velox/functions/prestosql/registration/CMakeFiles/velox_functions_prestosql.dir/ArithmeticFunctionsRegistration.cpp.o.d -o velox/functions/prestosql/registration/CMakeFiles/velox_functions_prestosql.dir/ArithmeticFunctionsRegistration.cpp.o -c ../../velox/functions/prestosql/registration/ArithmeticFunctionsRegistration.cpp

Will just reducing max link jobs help ?

@majetideepak
Copy link
Copy Markdown
Collaborator

It does not look like it failed in link step no :

Thanks for the clarification.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

@kgpai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

@kgpai merged this pull request in 7bca88a.

@conbench-facebook
Copy link
Copy Markdown

Conbench analyzed the 1 benchmark run on commit 7bca88a8.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

FelixYBW pushed a commit to FelixYBW/velox that referenced this pull request Feb 12, 2024
…#8592)

Summary:
**Why**

Builds OOM with the following error : https://app.circleci.com/pipelines/github/facebookincubator/velox/43449/workflows/0ecff160-1cb9-4084-9f1e-21ae5b484321/jobs/302064 . We reduced the parallelism recently but dint reduce it for all jobs (facebookincubator#8530) - In this PR we set it for all _debug_ jobs (which are most likely to OOM )

**How**
 We reduce parallelism as a stop gap to fix this issue and unblock folks.  We have an issue here for a long term fix : facebookincubator#8539

Pull Request resolved: facebookincubator#8592

Reviewed By: Yuhta

Differential Revision: D53240651

Pulled By: kgpai

fbshipit-source-id: 539030018ea3f18d5a3696327f679e7369235291
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Builds OOM'ing out when building ArithmeticFunctionRegistration.cpp

5 participants