Skip to content

HADOOP-19286: S3A: Support cross region access when S3 region/endpoint is set #7067

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steveloughran merged 5 commits into from
Oct 1, 2024
Merged

HADOOP-19286: S3A: Support cross region access when S3 region/endpoint is set #7067

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1c2e59c
HADOOP-19286: Support S3A cross region access when S3 region/endpoint…
shameersss1 Sep 24, 2024
f5f0bc5
Change the cross region access to sa-east-1
shameersss1 Sep 25, 2024
2c541df
Skip the cross region tests for third party stores
shameersss1 Sep 30, 2024
1f41548
null check to skip the test and remove non-aws region probe
shameersss1 Oct 1, 2024
d3a4e61
Add documentation changes
shameersss1 Oct 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1372,6 +1372,19 @@ private Constants() {
*/
public static final String XA_HEADER_PREFIX = "header.";

/**
* S3 cross region access enabled ?
* Value: {@value}.
*/

public static final String AWS_S3_CROSS_REGION_ACCESS_ENABLED =
"fs.s3a.cross.region.access.enabled";
/**
* Default value for S3 cross region access enabled: {@value}.
*/
public static final boolean AWS_S3_CROSS_REGION_ACCESS_ENABLED_DEFAULT = true;


/**
* AWS S3 region for the bucket. When set bypasses the construction of
* region through endpoint url.
Expand Down
14 changes: 10 additions & 4 deletions hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@
import static org.apache.hadoop.fs.s3a.Constants.AWS_REGION;
import static org.apache.hadoop.fs.s3a.Constants.AWS_S3_ACCESS_GRANTS_ENABLED;
import static org.apache.hadoop.fs.s3a.Constants.AWS_S3_ACCESS_GRANTS_FALLBACK_TO_IAM_ENABLED;
import static org.apache.hadoop.fs.s3a.Constants.AWS_S3_CROSS_REGION_ACCESS_ENABLED;
import static org.apache.hadoop.fs.s3a.Constants.AWS_S3_CROSS_REGION_ACCESS_ENABLED_DEFAULT;
import static org.apache.hadoop.fs.s3a.Constants.AWS_S3_DEFAULT_REGION;
import static org.apache.hadoop.fs.s3a.Constants.CENTRAL_ENDPOINT;
import static org.apache.hadoop.fs.s3a.Constants.FIPS_ENDPOINT;
Expand Down Expand Up @@ -330,7 +332,6 @@ private <BuilderT extends S3BaseClientBuilder<BuilderT, ClientT>, ClientT> void
builder.endpointOverride(endpoint);
LOG.debug("Setting endpoint to {}", endpoint);
} else {
builder.crossRegionAccessEnabled(true);
origin = "central endpoint with cross region access";
LOG.debug("Enabling cross region access for endpoint {}",
endpointStr);
Expand All @@ -343,7 +344,6 @@ private <BuilderT extends S3BaseClientBuilder<BuilderT, ClientT>, ClientT> void
// no region is configured, and none could be determined from the endpoint.
// Use US_EAST_2 as default.
region = Region.of(AWS_S3_DEFAULT_REGION);
builder.crossRegionAccessEnabled(true);
builder.region(region);
origin = "cross region access fallback";
} else if (configuredRegion.isEmpty()) {
Expand All @@ -354,8 +354,14 @@ private <BuilderT extends S3BaseClientBuilder<BuilderT, ClientT>, ClientT> void
LOG.debug(SDK_REGION_CHAIN_IN_USE);
origin = "SDK region chain";
}

LOG.debug("Setting region to {} from {}", region, origin);
boolean isCrossRegionAccessEnabled = conf.getBoolean(AWS_S3_CROSS_REGION_ACCESS_ENABLED,
AWS_S3_CROSS_REGION_ACCESS_ENABLED_DEFAULT);
// s3 cross region access
if (isCrossRegionAccessEnabled) {
builder.crossRegionAccessEnabled(true);
}
LOG.debug("Setting region to {} from {} with cross region access {}",
region, origin, isCrossRegionAccessEnabled);
}

/**
Expand Down
5 changes: 3 additions & 2 deletions hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,7 @@ public void testCustomUserAgent() throws Exception {
@Test
public void testRequestTimeout() throws Exception {
conf = new Configuration();
skipIfCrossRegionClient(conf);
// remove the safety check on minimum durations.
AWSClientConfig.setMinimumOperationDuration(Duration.ZERO);
try {
Expand Down Expand Up @@ -632,8 +633,8 @@ public static boolean isSTSSignerCalled() {
*/
private static void skipIfCrossRegionClient(
Configuration configuration) {
if (configuration.get(ENDPOINT, null) == null
&& configuration.get(AWS_REGION, null) == null) {
if (configuration.getBoolean(AWS_S3_CROSS_REGION_ACCESS_ENABLED,
AWS_S3_CROSS_REGION_ACCESS_ENABLED_DEFAULT)) {
skip("Skipping test as cross region client is in use ");
}
}
Expand Down
32 changes: 32 additions & 0 deletions hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@

import static org.apache.hadoop.fs.s3a.Constants.ALLOW_REQUESTER_PAYS;
import static org.apache.hadoop.fs.s3a.Constants.AWS_REGION;
import static org.apache.hadoop.fs.s3a.Constants.AWS_S3_CROSS_REGION_ACCESS_ENABLED;
import static org.apache.hadoop.fs.s3a.Constants.CENTRAL_ENDPOINT;
import static org.apache.hadoop.fs.s3a.Constants.ENDPOINT;
import static org.apache.hadoop.fs.s3a.Constants.FIPS_ENDPOINT;
Expand Down Expand Up @@ -346,6 +347,37 @@ public void testCentralEndpointAndDifferentRegionThanBucket() throws Throwable {
assertRequesterPaysFileExistence(newConf);
}

@Test
public void testWithOutCrossRegionAccess() throws Exception {
describe("Verify cross region access fails when disabled");
final Configuration newConf = new Configuration(getConfiguration());
// skip the test if the region is eu-west-2
String region = getFileSystem().getS3AInternals().getBucketMetadata().bucketRegion();
if (EU_WEST_2.equals(region)) {
Copy link
Contributor

@steveloughran steveloughran Sep 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I'd like a different region here a that is my region and I would like the test coverage.
  2. what happens with third party stores?

Copy link
Contributor Author

@shameersss1 shameersss1 Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Ack.
  2. As per the doc (https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/s3-cross-region.html)
When you reference an existing bucket in a request, such as when you use the putObject method, the SDK initiates a request to the Region configured for the client.

If the bucket does not exist in that specific Region, the error response includes the actual Region where the bucket resides. The SDK then uses the correct Region in a second request.

To optimize future requests to the same bucket, the SDK caches this Region mapping in the client.

So as per the implementation, It looks like it won't be supported by thirdparty store and each store have to implement it separately.

return;
}
// disable cross region access
newConf.setBoolean(AWS_S3_CROSS_REGION_ACCESS_ENABLED, false);
newConf.set(AWS_REGION, EU_WEST_2);
S3AFileSystem fs = new S3AFileSystem();
fs.initialize(getFileSystem().getUri(), newConf);
intercept(AWSRedirectException.class,
"does not match the AWS region containing the bucket",
() -> fs.exists(getFileSystem().getWorkingDirectory()));
}

@Test
public void testWithCrossRegionAccess() throws Exception {
describe("Verify cross region access succeed when enabled");
final Configuration newConf = new Configuration(getConfiguration());
// enable cross region access
newConf.setBoolean(AWS_S3_CROSS_REGION_ACCESS_ENABLED, true);
newConf.set(AWS_REGION, EU_WEST_2);
Copy link
Contributor

@steveloughran steveloughran Sep 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as above

Copy link
Contributor Author

@shameersss1 shameersss1 Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack

S3AFileSystem fs = new S3AFileSystem();
Copy link
Contributor

@steveloughran steveloughran Sep 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs to be in try-with-resources to close()

Copy link
Contributor Author

@shameersss1 shameersss1 Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack

fs.initialize(getFileSystem().getUri(), newConf);
fs.exists(getFileSystem().getWorkingDirectory());
}

@Test
public void testCentralEndpointAndSameRegionAsBucket() throws Throwable {
describe("Access public bucket using central endpoint and region "
Expand Down