Skip to content

HADOOP-18565. Completes outstanding items for the SDK V2 upgrade. #5421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steveloughran merged 42 commits into from
Apr 24, 2023
Merged

HADOOP-18565. Completes outstanding items for the SDK V2 upgrade. #5421

Changes from 2 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
c3c8f00
update to TM GA
ahmarsuhail Jan 3, 2023
1dd2523
use bundled transfer manager
ahmarsuhail Jan 9, 2023
db7e5e1
adds transfer listener to upload
ahmarsuhail Jan 9, 2023
e27316c
adds support for custom signers
ahmarsuhail Dec 9, 2022
08ad2fe
fixes ITestXAttr
ahmarsuhail Dec 12, 2022
4e70fb8
removes mockmaker
ahmarsuhail Dec 14, 2022
bd54e4d
fixes ACL and test301ContainsEndpoint
ahmarsuhail Dec 14, 2022
a70d864
address review comments
ahmarsuhail Dec 14, 2022
a1cc012
adds in generic instance creator for signers and cred providers
ahmarsuhail Feb 1, 2023
e359451
don't set default endpoint
ahmarsuhail Feb 1, 2023
403f06f
fixes failing tests
ahmarsuhail Feb 1, 2023
21613ec
fixes failing tests
ahmarsuhail Feb 2, 2023
2bf5710
removes v1 sdk bundle, only use core package
ahmarsuhail Feb 15, 2023
38cc68e
Update TM to use async client, not CRT
ahmarsuhail Feb 15, 2023
927dd19
fix failing tests
ahmarsuhail Feb 15, 2023
58850ba
adds in progress listener
ahmarsuhail Feb 16, 2023
78df27f
updates test for custom signer config
ahmarsuhail Feb 17, 2023
e24a8fe
adds in progress listener chaining
ahmarsuhail Feb 20, 2023
28f4dab
updates comments
ahmarsuhail Feb 20, 2023
2b8ac97
removes CRT dependency
ahmarsuhail Feb 20, 2023
c9d7dac
renames createS3ClientV2 to createS3Client
ahmarsuhail Feb 20, 2023
6fff7b8
removes crt version
ahmarsuhail Feb 20, 2023
5034365
removes crt dependency
ahmarsuhail Feb 20, 2023
6bfd380
fixes access point tests
ahmarsuhail Feb 21, 2023
75761d0
addresses review comments
ahmarsuhail Feb 21, 2023
77584b4
fix typo
ahmarsuhail Feb 21, 2023
4b7531d
fix checkstyle errors
ahmarsuhail Feb 22, 2023
e0cb320
fixes javadoc warnings + errors
ahmarsuhail Feb 22, 2023
01202ef
implements region caching
ahmarsuhail Feb 23, 2023
ca46d83
use new secure proxy config option
ahmarsuhail Feb 27, 2023
534bdd2
fixes failing tests
ahmarsuhail Feb 27, 2023
b3bcd5c
fixes checkstyle errors
ahmarsuhail Mar 1, 2023
9cf2f0d
removes listener from copy()
ahmarsuhail Mar 1, 2023
82bcb66
fixes failing test (put request length was set incorrectly)
ahmarsuhail Mar 9, 2023
9910bf8
fixes failing select scale test
ahmarsuhail Mar 10, 2023
6c80e9a
suppress warnings, fixes unit tests
ahmarsuhail Mar 16, 2023
a816414
changes as per review comments
ahmarsuhail Mar 20, 2023
3f6692f
updates ITestS3ABucketExistence
ahmarsuhail Mar 20, 2023
a1d8c52
fixes checkstyle issues
ahmarsuhail Mar 21, 2023
a055765
suppress inconsistent sync spotbug warning
ahmarsuhail Apr 17, 2023
d1805dd
set S3AsyncClient in synchronised method
ahmarsuhail Apr 20, 2023
b48709f
updates method name
ahmarsuhail Apr 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -975,10 +975,16 @@ private void bindAWSClient(URI name, boolean dtEnabled) throws IOException {

S3ClientFactory clientFactory = ReflectionUtils.newInstance(s3ClientFactoryClass, conf);
s3Client = clientFactory.createS3Client(getUri(), parameters);
s3AsyncClient = clientFactory.createS3AsyncClient(getUri(), parameters);
setS3AsyncClient(clientFactory, parameters);
transferManager = clientFactory.createS3TransferManager(s3AsyncClient);
}

// set in synchronized method to suppress spotbugs error.
Copy link
Contributor

@steveloughran steveloughran Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. give it a name like createS3AsyncClient as it is more than a simple setter
  2. comment should be a javadoc so IDEs preview it.

private synchronized void setS3AsyncClient(S3ClientFactory clientFactory,
S3ClientFactory.S3ClientCreationParameters parameters) throws IOException {
s3AsyncClient = clientFactory.createS3AsyncClient(getUri(), parameters);
}

/**
* Get the bucket region.
*
Expand Down