feat: add LinearCurve variant for adaptive parallelism strategy #24022
+359
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduce a new AdaptiveParallelismStrategy variant `LinearCurve` that allows fine-grained control of target parallelism via piecewise linear interpolation. This enables smooth, custom scaling of parallelism ratios based on available parallelism, improving flexibility over existing fixed or bounded strategies. - Add `LinearCurve` and `LinearCurvePoint` data structures with interpolation - Implement parsing of `linear_curve(a:r,...)` syntax with validation and padding - Support serialization/deserialization preserving existing string format style - Calculate target parallelism by interpolating ratio and scaling current value - Add comprehensive unit tests covering parsing, interpolation, errors, and ordering Signed-off-by: Peng Chen <[email protected]>
Refactor interpolation and padding logic in the linear curve strategy for better correctness and code clarity. Modernize Rust idioms to enhance readability and maintainability. Add comprehensive unit and integration tests to verify interpolation behavior and system parameter effects. - Replace explicit range checks with idiomatic `.contains()` usage - Use `if let ... && ...` pattern and `.unwrap_or_else()` for cleaner code - Add detailed comments explaining interpolation and padding logic - Ensure padding points at 0 and 1 cores are properly inserted via interpolation - Add unit tests for interpolation helper and LinearCurve interpolation - Enhance integration tests with standard config and total cores encapsulation - Add new integration test covering linear curve adaptive strategy behavior Signed-off-by: Peng Chen <[email protected]>
Escape square brackets in the `LinearCurve` struct doc comment from `[0,1]` to `\[0,1\]` to prevent Rustdoc from interpreting them as Markdown links. - Fixes potential Rustdoc warnings about malformed link syntax - Ensures correct rendering of the target ratio range in generated docs - Improves documentation clarity without affecting code behavior Signed-off-by: Peng Chen <[email protected]>
Enhance error reporting in the `parse_linear_curve` function by using the `AsReport` trait to format parsing errors more clearly and consistently. - Import `thiserror_ext::AsReport` to enable richer error descriptions - Replace direct error display with `e.as_report()` in error formatting - Improve clarity and detail of error messages when parsing fails - Maintain backward compatibility with no behavioral changes or API impact Signed-off-by: Peng Chen <[email protected]>
Improve error handling in `parse_linear_curve` by explicitly specifying the error types in the `map_err` closures when parsing numeric values from strings. - Annotate parse errors as `ParseIntError` and `ParseFloatError` explicitly - Enhance code clarity and compiler type inference for error handling - Maintain detailed error reporting via `as_report()` without changing logic - Address potential compiler or linter warnings about ambiguous error types Signed-off-by: Peng Chen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Summary
This PR introduces a new
LinearCurvevariant to theAdaptiveParallelismStrategyenum, enabling flexible piecewise-linear mapping from available parallelism to target parallelism ratio. It enhances the system's ability to finely control parallelism scaling behavior beyond existing fixed or ratio-based strategies.Details
LinearCurveandLinearCurvePointstructs representing piecewise-linear curves with validated points sorted by available parallelism.LINEAR_CURVE(0:0,10:0.5,20:0.8)with automatic padding at boundaries for robustness.compute_target_parallelismto handle the new strategy, impacting adaptive parallelism decisions across the system.By the way, this curve doesn't necessarily need to keep increasing; it can even be used to draw a sine curve.
Checklist
Documentation
Release note
Add a linear_curve to the adaptive parallelism strategy, allowing users to customize the scaling curve.