Strengthen undefined behavior prevention in checkSignedBitfieldOverflow (#2418)

This one was found by
[afl++](https://github.com/AFLplusplus/AFLplusplus). Executing `bitfield
0 set i64 0 1` triggers UBSan at the `int64_t minincr = min - value;`
calculation. To fix the undefined behavior in the `minincr` calculation
and strengthen the protection in the `maxincr` calculation, we cast
both, the minuend and the subtrahend, to an unsigned int, do the
calculation, and then cast the result back into a signed int.

Signed-off-by: Fusl <[email protected]>

Author: Fusl

src/bitops.c

@@ -491,12 +491,10 @@ int checkSignedBitfieldOverflow(int64_t value, int64_t incr, uint64_t bits, int
     int64_t max = (bits == 64) ? INT64_MAX : (((int64_t)1 << (bits - 1)) - 1);
     int64_t min = (-max) - 1;
 
-    /* Note that maxincr and minincr could overflow, but we use the values
-     * only after checking 'value' range, so when we use it no overflow
-     * happens. 'uint64_t' cast is there just to prevent undefined behavior on
-     * overflow */
-    int64_t maxincr = (uint64_t)max - value;
-    int64_t minincr = min - value;
+    /* max/min and value are signed integers but to avoid undefined behavior
+     * we temporarily cast them to unsigned integers before subtracting. */
+    int64_t maxincr = (int64_t)((uint64_t)max - (uint64_t)value);
+    int64_t minincr = (int64_t)((uint64_t)min - (uint64_t)value);
 
     if (value > max || (bits != 64 && incr > maxincr) || (value >= 0 && incr > 0 && incr > maxincr)) {
         if (limit) {