Skip to content

HADOOP-18246. Reduce lower limit on s3a prefetching/caching block size to 1 byte. #5120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steveloughran merged 6 commits into from
Feb 2, 2023
Merged

HADOOP-18246. Reduce lower limit on s3a prefetching/caching block size to 1 byte. #5120

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0429180
Made the lower limit to 1
Nov 4, 2022
cdbed32
Lowered the Prefetch Limit, unchanged the Default size
Nov 8, 2022
563f0ff
Added the lower limit information in the Prefetch Readme
Nov 8, 2022
ddba5cd
Added new line after period in readme.
Nov 8, 2022
bf73b01
Updated the documentation of general s3a client configuration.
Nov 9, 2022
5705e99
Improved block description in s3a client configuration
Nov 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -525,8 +525,7 @@ public void initialize(URI name, Configuration originalConf)

this.prefetchEnabled = conf.getBoolean(PREFETCH_ENABLED_KEY, PREFETCH_ENABLED_DEFAULT);
long prefetchBlockSizeLong =
longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE,
PREFETCH_BLOCK_DEFAULT_SIZE);
longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE, 1);
if (prefetchBlockSizeLong > (long) Integer.MAX_VALUE) {
throw new IOException("S3A prefatch block size exceeds int limit");
}
Expand Down
6 changes: 6 additions & 0 deletions hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@ Multiple blocks may be read in parallel.
|`fs.s3a.prefetch.block.size` |Size of a block |`8M` |
|`fs.s3a.prefetch.block.count` |Number of blocks to prefetch |`8` |

Although, default size of the block for prefetching the input stream is 8 MB,
minimum size allowed to set is 1 byte for a block.
User should set the block size with the understanding that smaller block sizes increases the number of blocks.
Thus, smaller block size affects the performance by increasing the overhead for reading and prefetching
each block.
Copy link
Contributor

@ahmarsuhail ahmarsuhail Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Although, default size of the block for prefetching the input stream is 8 MB,
minimum size allowed to set is 1 byte for a block.
User should set the block size with the understanding that smaller block sizes increases the number of blocks.
Thus, smaller block size affects the performance by increasing the overhead for reading and prefetching
each block.
The default size of a block is 8MB, and the minimum allowed block size is 1 byte.
Decreasing block size will increase the number of blocks to be read for a file.
A smaller block size may negatively impact performance as the number of prefetches required will increase.


### Key Components

`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of
Expand Down