PGO optimization results for LLVM-based projects

[zamazan4ik]

Hi!

LLVM right now supports PGO only for Clang. I want to share PGO results for other LLVM-based projects.

Here I want to share my results with applying PGO on clangd (version - current main branch). According to my local tests on AMD 5900x/48 Gib RAM/Fedora 38/Clang 16 (for building clangd), clangd in Release mode without PGO finishes indexing llvm-project sources in ~10 minutes (9m55s), and clangd in Release mode with PGO (fprofile-instr-generate/-fprofile-instr-use) finishes indexing llvm-project in ~8 minutes (7m50s - 7m55s). Tests were performed multiple times, with clangd cache reset between runs, on the latest main branch. Compilation options: CC=clang CXX=clang++ cmake -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/zamazan4ik/open_source/install_clangd -DLLVM_INCLUDE_BENCHMARKS=1 -DCMAKE_POSITION_INDEPENDENT_CODE=1 -DLLVM_USE_LINKER=lld -G "Ninja" ../llvm-project/llvm (PGO mode just additionally sets -fprofile-instr-generate/-fprofile-instr-use options).

I think the results are quite good to add PGO support to clangd to the repository (as it's already done with clang itself). Another article about applying PGO on clangd you can find here: JetBrains blog.

Additionally want to note that Clangd in Instrumentation mode works toooo slow, so I just waited for indexing ~200 files from the LLVM repo. I think that's enough (and the benchmark confirms it).

[llvmbot]

@llvm/issue-subscribers-clangd

[zamazan4ik]

I think similar projects like Clang Tidy and Clang Static Analyzer also should have a performance boost from PGO.

[zamazan4ik]

On the same hardware/software setup I got the following results for optimizing Clang-Tidy (version - current main branch). As a bench workload, I chose to scan BOLT sources with all Clang-Tidy checks (except alpha). Measurements were done with time utility.

The results are the following:

• Release: ./run-clang-tidy -checks="*" -p /home/zamazan4ik/open_source/build_bolt 5319,09s user 49,67s system 2142% cpu 4:10,58 total
• Release + PGO: ./run-clang-tidy -checks="*" -p /home/zamazan4ik/open_source/build_bolt 4691,02s user 53,03s system 2041% cpu 3:52,43 total
• (just for the history) Instrumentation mode (-fprofile-instr-generate): ./run-clang-tidy -checks="*" -p /home/zamazan4ik/open_source/build_bolt 12124,95s user 65,60s system 2057% cpu 9:52,44 total

[kadircet]

FWIW, I don't think this is a clangd specific issue, but rather a release issue, and probably it should be discussed as an RFC in discourse rather than here. As people can already have whatever profiling they want when building LLVM, and I believe the proposal here is to have some PGO with the default LLVM releases and/or the way distros are building LLVM related packages.
So this extra complexity should be vetted by people that maintain those build systems/scripts.

---

regarding clangd, I believe the indexing is not the most latency-sensitive workflow we have. It's done in the background, on the idle cores and despite being quite useful for a good experience, having a full index of the codebase as soon as possible is not detrimental to clangd functionality.

the rather latency-sensitive workflows actually involve interactive interactions like code completion/signature help and main file ast builds (similar to indexing, but with a preamble).

so i think having some profiling based on those interactions would be better, but I guess any profiles are fine as long as they don't clearly regress performance of latency-sensitive interactions i mentioned above.

[zamazan4ik]

FWIW, I don't think this is a clangd specific issue, but rather a release issue, and probably it should be discussed as an RFC in discourse rather than here. As people can already have whatever profiling they want when building LLVM, and I believe the proposal here is to have some PGO with the default LLVM releases and/or the way distros are building LLVM related packages.
So this extra complexity should be vetted by people that maintain those build systems/scripts.

Right now I am talking about improving PGO optimization for other LLVM projects to the same level as Clang has now (CMake-specific scripts in the LLVM repo). I am not talking about the builds in different Linux distributions - that's a different talk for every distro since they need to choose the balance between "performance improvements" and "maintainability costs".

regarding clangd, I believe the indexing is not the most latency-sensitive workflow we have. It's done in the background, on the idle cores and despite being quite useful for a good experience, having a full index of the codebase as soon as possible is not detrimental to clangd functionality.

Anyway, it improves a lot experience with checkouting pretty large codebases on the local machines and switching between fast-evolving branches (in this case reindexing for multiple files is required, so even in this case the UX will be improved as well).

the rather latency-sensitive workflows actually involve interactive interactions like code completion/signature help and main file ast builds (similar to indexing, but with a preamble).

I didn't perform such benchmarks but Jetbrains did (warning - on Windows): link.