refactor(tpch): Handle in-output errors for string functions (Part 2) #12064

anandamideShakyan · 2025-01-11T19:56:27Z

PR Description:

Detect and address input/output errors that can result in undefined behavior. Ensure proper error handling for I/O functions that may fail and leave variables uninitialized. Neglecting to check the status of these functions before using their outputs (e.g., memory buffers, file descriptors, etc.) can lead to undefined program behavior. This update enforces checks on commonly used I/O functions to validate their return values and prevent improper usage.

This is the second PR of the several changes for this refactoring.

netlify · 2025-01-11T19:56:45Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`3e86148`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/67fe14798368710008b038e5

czentgr · 2025-01-14T05:52:44Z

velox/tpch/gen/dbgen/CMakeLists.txt

 target_compile_definitions(dbgen PRIVATE DBNAME=dss MAC ORACLE TPCH)
 target_include_directories(dbgen PRIVATE include)
-
+target_link_libraries(dbgen PUBLIC Folly::folly)


This should be INTERFACE instead of PUBLIC. We don;t actually need to link to Folly here. We only want it to get to the header for the macro FOLLY_UNLIKELY.

Sorry that's actually not true, I was unsure and made a little test, see below.

The a folly header is included in a header but also an implementation file so we need to link as PUBLIC to make folly's include dir available both to targets depending on this target (aka anyone using this header) and when building build.cpp.

INTERFACE will not add folly's include dir to the compiler invocation, only to the one of targets:

cmake_minimum_required(VERSION 3.20) project(test) set(CMAKE_EXPORT_COMPILE_COMMANDS ON) add_library(main a.cpp) add_library(lib_a INTERFACE) target_include_directories(lib_a INTERFACE ${CMAKE_CURRENT_LIST_DIR}/include) target_link_libraries(main INTERFACE lib_a)

INTERFACE

[ { "directory": "~/scratch", "command": "/usr/bin/c++ -o CMakeFiles/main.dir/a.cpp.o -c ~/scratch/a.cpp", "file": "~/scratch/a.cpp", "output": "CMakeFiles/main.dir/a.cpp.o" } ]

PUBLIC

[ { "directory": "~/scratch", "command": "/usr/bin/c++ -I~/scratch/include -o CMakeFiles/main.dir/a.cpp.o -c ~/scratch/a.cpp", "file": "~/scratch/a.cpp", "output": "CMakeFiles/main.dir/a.cpp.o" } ]

czentgr · 2025-01-14T05:53:44Z

velox/tpch/gen/dbgen/include/dbgen/dss.h

+#define PR_DATE(tgt, yr, mn, dy)                             \
+do {                                                         \
+  int res = sprintf(tgt, "19%02ld-%02ld-%02ld", yr, mn, dy); \
+  if(FOLLY_UNLIKELY(res < 0)) {                              \


How come the header for this macro include isn't needed?
It seems to have been needed in the build.cpp file.
How come? Is it not actually needed everywhere because it comes in from some other include?

It might be coming from some other file, I will include it explicitly.

aditi-pandit · 2025-01-14T21:09:52Z

velox/tpch/gen/dbgen/build.cpp

-  sprintf(target + 3, "%03d", static_cast<int>(acode));
-  sprintf(target + 7, "%03d", static_cast<int>(exchg));
-  sprintf(target + 11, "%04d", static_cast<int>(number));
+  int res = sprintf(target, "%02d", static_cast<int>(10 + (ind % NATIONS_MAX)));


Why not use fmt::format instead of the complications with sprintf ?

Done, but still we need to convert std::string back to c style right and store the output in target.

aditi-pandit · 2025-01-14T21:11:34Z

velox/tpch/gen/dbgen/include/dbgen/dss.h

-#define PR_DATE(tgt, yr, mn, dy) sprintf(tgt, "%02d-%02d-19%02d", mn, dy, yr)
+#define PR_DATE(tgt, yr, mn, dy)                               \
+  do {                                                         \
+    int res = sprintf(tgt, "19%02ld-%02ld-%02ld", yr, mn, dy); \


fmt::format would be simpler here as well.

One doubt though, fmt::format returns a cpp string whereas tgt is supposed to be a c style null terminated string(char array) . So do you want me to use fmt::format followed by strncpy as in :

#define PR_DATE(tgt, yr, mn, dy) \ do { \ std::string formatted = fmt::format("{:02d}-{:02d}-19{:02d}", mn, dy, yr); \ std::strncpy(tgt, formatted.c_str(), formatted.size() + 1); \ tgt[formatted.size() + 1] = '\0' } while (0)

czentgr · 2025-02-03T15:08:03Z

velox/exec/tests/AggregateSpillBenchmarkBase.h

 public:
  explicit AggregateSpillBenchmarkBase(std::string spillerType)
-      : spillerType_(spillerType){};
+      : spillerType_(spillerType) {};


For these changes here and the MultiFragmentTest.
looks like only format changes. Is that because the format-fix was making these while you undid your previous changes?

Yeah. The format fix is adding this space, what should I be doing retain the old formatting or keep it as it is?

czentgr · 2025-02-03T15:09:15Z

velox/tpch/gen/dbgen/build.cpp

+      static_cast<int>(exchg),
+      static_cast<int>(number));
+  std::strncpy(target, formatted.c_str(), sizeof(target) - 1);
+  target[sizeof(target) - 1] = '\0';


strncpy already adds a null-terminator. This is not needed.

czentgr · 2025-02-03T15:17:01Z

velox/tpch/gen/dbgen/build.cpp


  if (!bInit) {
-    sprintf(szFormat, C_NAME_FMT, 9, &HUGE_FORMAT[1]);
+    auto result = sprintf(szFormat, C_NAME_FMT, 9, &HUGE_FORMAT[1]);


Perhaps we want consistency in the naming of the variable. It is called res everywhere else.

Actually, there is another variable named res just below, so I didn't want to cause any scoping issues.

There is no scoping issue really. This one is local to this block only.
The compiler will tell you if there are problems.

velox/tpch/gen/dbgen/build.cpp

anandamideShakyan · 2025-02-24T09:32:37Z

@czentgr waiting for your review on this one.

libianoss · 2025-03-04T13:25:47Z

@czentgr can you help to review this PR based on @anandamideShakyan 's change? Thanks.

czentgr · 2025-03-07T15:55:43Z

@anandamideShakyan please review the CI failures. Rebase if necessary.

anandamideShakyan · 2025-03-17T14:56:03Z

@czentgr resolved the CI failures, waiting for your review.

czentgr

Thanks!

prestodb-ci · 2025-04-04T08:40:36Z

@ethanyzhang imported this issue into IBM GitHub Enterprise

velox/tpch/gen/dbgen/build.cpp

velox/tpch/gen/dbgen/include/dbgen/dss.h

velox/tpch/gen/dbgen/build.cpp

velox/tpch/gen/dbgen/CMakeLists.txt

velox/tpch/gen/dbgen/build.cpp

czentgr · 2025-04-07T19:53:51Z

velox/tpch/gen/dbgen/build.cpp

 #include "dbgen/dsstypes.h" // @manual

 #include <math.h>
+#include "common/base/Exceptions.h"


The target still needs fmt as a dependency. It might be able to stay "private".

This should be "velox/common/base/Exceptions.h" , please fix.

Done. This was added as an automatic import by clion when I used VELOX_CHECK_GT macro, not sure why common/base/Exceptions.h was used in the import path, it is strange.

Doesn't this line:include_directories(SYSTEM velox) in root CmakeLists.txt allow us to pass just "common/base/Exceptions.h" as file path?

velox/tpch/gen/dbgen/CMakeLists.txt

kgpai · 2025-04-11T00:07:44Z

Can you rebase to latest main @anandamideShakyan ?

facebook-github-bot · 2025-04-15T00:08:48Z

@kgpai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-04-16T18:48:45Z

@kgpai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-04-17T07:24:11Z

@kgpai merged this pull request in 4ddf2dd.

…facebookincubator#12064) Summary: # PR Description: Detect and address input/output errors that can result in undefined behavior. Ensure proper error handling for I/O functions that may fail and leave variables uninitialized. Neglecting to check the status of these functions before using their outputs (e.g., memory buffers, file descriptors, etc.) can lead to undefined program behavior. This update enforces checks on commonly used I/O functions to validate their return values and prevent improper usage. This is the second PR of the several changes for this refactoring. Pull Request resolved: facebookincubator#12064 Reviewed By: pedroerp Differential Revision: D73005009 Pulled By: kgpai fbshipit-source-id: 5cce140545fb9f682b036ce86875bd2c1b1b5a06

anandamideShakyan requested review from assignUser and majetideepak as code owners January 11, 2025 19:56

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 11, 2025

czentgr reviewed Jan 14, 2025

View reviewed changes

aditi-pandit reviewed Jan 14, 2025

View reviewed changes

czentgr reviewed Feb 3, 2025

View reviewed changes

anandamideShakyan force-pushed the dev4 branch from 96c9c3f to d597761 Compare February 16, 2025 14:25

assignUser reviewed Feb 22, 2025

View reviewed changes

velox/tpch/gen/dbgen/build.cpp Outdated Show resolved Hide resolved

anandamideShakyan changed the title ~~Refactor: handle in-output errors for string functions (Part 2)~~ refactor: Handle in-output errors for string functions (Part 2) Feb 24, 2025

anandamideShakyan force-pushed the dev4 branch from 21f4746 to b4f386e Compare February 25, 2025 07:26

anandamideShakyan force-pushed the dev4 branch from b4f386e to 18c33b5 Compare March 10, 2025 03:40

anandamideShakyan force-pushed the dev4 branch 2 times, most recently from 8c599a5 to 722535e Compare March 21, 2025 04:24

czentgr approved these changes Mar 21, 2025

View reviewed changes

anandamideShakyan force-pushed the dev4 branch from 722535e to 00b98a1 Compare March 22, 2025 06:53

anandamideShakyan force-pushed the dev4 branch from 00b98a1 to 1f446f5 Compare April 1, 2025 03:30

majetideepak reviewed Apr 4, 2025

View reviewed changes

velox/tpch/gen/dbgen/build.cpp Outdated Show resolved Hide resolved

velox/tpch/gen/dbgen/build.cpp Outdated Show resolved Hide resolved

velox/tpch/gen/dbgen/include/dbgen/dss.h Outdated Show resolved Hide resolved

anandamideShakyan force-pushed the dev4 branch from 63edca4 to 3fa46fe Compare April 5, 2025 22:14

majetideepak reviewed Apr 7, 2025

View reviewed changes

velox/tpch/gen/dbgen/build.cpp Outdated Show resolved Hide resolved

majetideepak reviewed Apr 7, 2025

View reviewed changes

velox/tpch/gen/dbgen/CMakeLists.txt Outdated Show resolved Hide resolved

majetideepak reviewed Apr 7, 2025

View reviewed changes

velox/tpch/gen/dbgen/build.cpp Outdated Show resolved Hide resolved

anandamideShakyan force-pushed the dev4 branch 3 times, most recently from 4aa27d4 to bae1d95 Compare April 7, 2025 14:29

anandamideShakyan force-pushed the dev4 branch from 69ff478 to 1042eb2 Compare April 7, 2025 14:38

majetideepak approved these changes Apr 7, 2025

View reviewed changes

majetideepak changed the title ~~refactor: Handle in-output errors for string functions (Part 2)~~ refactor(tpch): Handle in-output errors for string functions (Part 2) Apr 7, 2025

majetideepak added the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Apr 7, 2025

czentgr approved these changes Apr 7, 2025

View reviewed changes

czentgr reviewed Apr 7, 2025

View reviewed changes

anandamideShakyan force-pushed the dev4 branch 2 times, most recently from ea293ba to a9c6fc2 Compare April 8, 2025 05:16

majetideepak reviewed Apr 8, 2025

View reviewed changes

velox/tpch/gen/dbgen/CMakeLists.txt Outdated Show resolved Hide resolved

anandamideShakyan force-pushed the dev4 branch from 2b6a49f to 426ac02 Compare April 9, 2025 04:28

anandamideShakyan force-pushed the dev4 branch from 426ac02 to 9af0c5f Compare April 11, 2025 04:34

anandamideShakyan force-pushed the dev4 branch from 9af0c5f to ba384b6 Compare April 15, 2025 06:58

refactor(tpch): Handle in-output errors for string functions (Part 2)

3e86148

anandamideShakyan force-pushed the dev4 branch from ba384b6 to 3e86148 Compare April 15, 2025 08:10

facebook-github-bot closed this in 4ddf2dd Apr 17, 2025

facebook-github-bot added the Merged label Apr 17, 2025

prestodb-ci mentioned this pull request Apr 17, 2025

Rebase branch velox_pr_rebase (937e0a1) with oss-main (4ddf2dd) oap-project/velox#552

Closed

1 task

refactor(tpch): Handle in-output errors for string functions (Part 2) #12064

refactor(tpch): Handle in-output errors for string functions (Part 2) #12064

Uh oh!

Conversation

anandamideShakyan commented Jan 11, 2025

PR Description:

Uh oh!

netlify bot commented Jan 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for meta-velox canceled.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anandamideShakyan Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

anandamideShakyan commented Feb 24, 2025

Uh oh!

libianoss commented Mar 4, 2025

Uh oh!

czentgr commented Mar 7, 2025

Uh oh!

anandamideShakyan commented Mar 17, 2025

Uh oh!

czentgr left a comment

Choose a reason for hiding this comment

Uh oh!

prestodb-ci commented Apr 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anandamideShakyan Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kgpai commented Apr 11, 2025

Uh oh!

netlify bot commented Jan 11, 2025 •

edited

Loading

anandamideShakyan Feb 4, 2025 •

edited

Loading

anandamideShakyan Apr 15, 2025 •

edited

Loading