pwilson/Add TPCH SF3000 Support with Automatic Resource Validation #122
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
patdevinwilson
commented
Nov 3, 2025
patdevinwilson
commented
- Problem: No validation for large scale factors
- Solution: Automatic resource checks + comprehensive docs
- Testing: Successfully running SF3000 generation
- Impact: 6 files, +613 lines, fully backward compatible
- Benefits: Prevents failures, better UX, automation-friendly
- Add SF3000_SUPPORT.md with comprehensive documentation - Update setup_benchmark_data_and_tables.sh with automatic warnings for large scale factors - Add validation in generate_data_files.py for SF3000+ generations - Update README.md files with SF3000 examples and requirements - Automatic checks for disk space, memory, and time requirements - Interactive confirmation prompts for SF3000+ generations Users can now generate SF3000 (~3TB) datasets with appropriate warnings about: - 4TB+ disk space requirements - 256GB+ RAM recommendations - 8-24 hour generation time - Resource availability checks
- Add SF3000 example command with resource requirements comment - Add SCALE FACTORS section showing requirements for different scale factors - Update both setup scripts to display comprehensive scale factor information - Help text now shows: SF1-SF100, SF1000 (~1TB), SF3000 (~3TB) requirements
- Add sys.stdin.isatty() check to detect non-interactive mode - Auto-proceed with SF3000 generation when running in background - Fixes EOFError when running via nohup or automated scripts - Interactive mode still prompts for confirmation as before
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.