Skip to content

Commit 7a8fb11

Browse files
committed
Merge branch 'branch-3.3' of github.com:apache/hadoop into HADOOP-18805-branch-3.3
2 parents 9db8bd1 + f439801 commit 7a8fb11

3 files changed

Lines changed: 6 additions & 2 deletions

File tree

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -543,8 +543,7 @@ public void initialize(URI name, Configuration originalConf)
543543
DEFAULT_MULTIPART_UPLOAD_ENABLED);
544544
this.prefetchEnabled = conf.getBoolean(PREFETCH_ENABLED_KEY, PREFETCH_ENABLED_DEFAULT);
545545
long prefetchBlockSizeLong =
546-
longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE,
547-
PREFETCH_BLOCK_DEFAULT_SIZE);
546+
longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE, 1);
548547
if (prefetchBlockSizeLong > (long) Integer.MAX_VALUE) {
549548
throw new IOException("S3A prefetch block size exceeds int limit");
550549
}

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1107,6 +1107,7 @@ options are covered in [Testing](./testing.md).
11071107
<value>8MB</value>
11081108
<description>
11091109
The size of a single prefetched block of data.
1110+
Decreasing this will increase the number of prefetches required, and may negatively impact performance.
11101111
</description>
11111112
</property>
11121113

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,10 @@ Multiple blocks may be read in parallel.
4343
|`fs.s3a.prefetch.block.size` |Size of a block |`8M` |
4444
|`fs.s3a.prefetch.block.count` |Number of blocks to prefetch |`8` |
4545

46+
The default size of a block is 8MB, and the minimum allowed block size is 1 byte.
47+
Decreasing block size will increase the number of blocks to be read for a file.
48+
A smaller block size may negatively impact performance as the number of prefetches required will increase.
49+
4650
### Key Components
4751

4852
`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of

0 commit comments

Comments
 (0)