HADOOP-15409 #367

lqjack · 2018-04-25T11:08:47Z

change doesBucketExistV2 to verified the acl

danieldgf · 2018-04-26T06:56:48Z

厉害

rahulkinra · 2018-04-27T02:18:01Z

There are no conflicts in this right now.

steveloughran · 2018-05-01T12:08:14Z

I do want to merge this in, but we have a very strict policy for the object stores "Say which endpoint you ran the integration tests for that store against"? Jenkins can't do it, and while I'll do a full test run before I do the final merge, I don't want to be the person debugging the acutal patch.

see Policy_for_submitting_patches_which_affect_the_hadoop-aws_module

thanks

lqjack · 2018-05-02T16:18:55Z

@steveloughran Thanks your comment, I will append the test case .

lqjack · 2018-06-28T04:04:38Z

@steveloughran can I apply the credentials for test ?

append the Test case

steveloughran · 2018-06-28T14:19:40Z

afraid you get to use your own credentials. I'll do a final testrun before committing, but during dev you are going to have to pay. The tests are designed to keep costs down to cents; all data is deleted, the big downloads are on Amazon's own data, and if you don't run the scale tests (-Pscale) there's not much bulk data IO going on at al...

lqjack · 2018-06-29T01:39:48Z

@steveloughran Thanks reply.

steveloughran · 2018-07-27T21:34:59Z

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABlocksize.java

+    try {
+      fs.verifyBucketExists();
+    } catch (IOException e) {
+      fail(e.getMessage());


Given we actually call verifyBucketExists in the initialize() routine, it'll have already been tested in setup. I think we can get by without adding a new test here.

Ok， I will update the test case.

lqjack · 2018-08-09T06:46:27Z

@steveloughran I have removed the test case ,please reveiw and comment.

steveloughran · 2018-12-09T16:31:32Z

hey, it's great you've signed your patches. Can you submit your public key to the central GPG Servers?

~/P/h/h/hadoop-aws (incoming/PR-367-HADOOP-15409 ↩) git log1s
* gpg: Signature made Thu Aug  9 07:45:14 2018 BST
| gpg:                using RSA key 4AEE18F83AFDEB23
| gpg: requesting key 4AEE18F83AFDEB23 from hkps server hkps.pool.sks-keyservers.net
| gpg: Can't check signature: No public key
| 81f78c27798 - (HEAD -> incoming/PR-367-HADOOP-15409) remove IOException (4 months ago)

steveloughran · 2018-12-09T16:54:22Z

Oddly enough, one of the stack traces in troubleshooting_s3a shows doesBucketExist doing the ACK check,

zonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176)
  at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3381)
  at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1160)
  at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1150)
  at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:367)

The sdk may have switched a long time ago

steveloughran · 2018-12-09T17:16:18Z

Next patch for this will give you a choice of v1 or v2, so that third-party endpoints will work. Versions other than 1 or 2 get told off and then continue

 bin/hadoop fs -D fs.s3a.bucket.exists.version=3 -ls s3a://fdsd/
2018-12-09 17:11:58,015 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-12-09 17:11:58,537 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2018-12-09 17:11:58,598 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2018-12-09 17:11:58,598 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
2018-12-09 17:11:59,177 WARN s3a.S3AFileSystem: Unknown fs.s3a.bucket.exists.version version: 3
2018-12-09 17:12:02,729 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
ls: s3a://fdsd/: getFileStatus on s3a://fdsd/: com.amazonaws.services.s3.model.AmazonS3Exception: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID: 7D34514B3633D1EC; S3 Extended Request ID: AWimKqhxDqr10Q22QVnh71xN/ctC9UK+x7/FubEbbMIs8sjpXbTrCttzy9r1BUvUKeFM0E9Jlb8=), S3 Extended Request ID: AWimKqhxDqr10Q22QVnh71xN/ctC9UK+x7/FubEbbMIs8sjpXbTrCttzy9r1BUvUKeFM0E9Jlb8=:AllAccessDisabled
2018-12-09 17:12:03,163 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
2018-12-09 17:12:03,163 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
2018-12-09 17:12:03,163 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.

lqjack · 2018-12-10T01:57:15Z

@steveloughran the public key has imported. thanks.

Author: Jacob Maes <[email protected]> Author: Jacob Maes <[email protected]> Reviewers: Prateek Maheshwari <[email protected]> Closes apache#367 from jmakes/samza-1508

apache#367) * HDFS-16791. Add getEnclosingRoot() API to filesystem interface and implementations (apache#6198) The enclosing root path is a common ancestor that should be used for temp and staging dirs as well as within encryption zones and other restricted directories. Contributed by Tom McCormick ACLOVERRIDE due to OSS backport email issue * fixing merge conflicts * Fixing backport issues * ACLOVERRIDE due to OSS backport email issue --------- Co-authored-by: Tom <[email protected]> Co-authored-by: Tom McCormick <[email protected]>

github-actions · 2025-12-29T00:27:29Z

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.

HADOOP-15409

0c8af1b

change doesBucketExistV2 to verified the acl

HADOOP-15428

0c516b1

append the Test case

steveloughran reviewed Jul 27, 2018

View reviewed changes

lqjack added 2 commits August 9, 2018 14:44

Remove Test case

eaa93c2

remove IOException

81f78c2

github-actions bot added the Stale label Dec 29, 2025

github-actions bot closed this Dec 30, 2025

HADOOP-15409 #367

HADOOP-15409 #367

Uh oh!

Conversation

lqjack commented Apr 25, 2018

Uh oh!

danieldgf commented Apr 26, 2018

Uh oh!

rahulkinra commented Apr 27, 2018

Uh oh!

steveloughran commented May 1, 2018

Uh oh!

lqjack commented May 2, 2018

Uh oh!

lqjack commented Jun 28, 2018

Uh oh!

steveloughran commented Jun 28, 2018

Uh oh!

lqjack commented Jun 29, 2018

Uh oh!

steveloughran Jul 27, 2018

Choose a reason for hiding this comment

Uh oh!

lqjack Aug 3, 2018

Choose a reason for hiding this comment

Uh oh!

lqjack commented Aug 9, 2018

Uh oh!

steveloughran commented Dec 9, 2018

Uh oh!

steveloughran commented Dec 9, 2018

Uh oh!

steveloughran commented Dec 9, 2018

Uh oh!

lqjack commented Dec 10, 2018

Uh oh!

github-actions bot commented Dec 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants