Skip to content

HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport #6069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mukund-thakur merged 24 commits into from
Feb 22, 2024
Merged

HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport #6069

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
a4f240d
Production Code changes to support Checksum
Sep 14, 2023
146affa
Code Refracted
Sep 18, 2023
604be8f
Addressed Comments
Sep 19, 2023
c1ce19d
Added More tests
Sep 20, 2023
3b9d17f
Resolved Comments
Sep 20, 2023
03a7453
PR Checks
Sep 21, 2023
d422012
Addressed Comments
Sep 21, 2023
c09259d
Sending Relevant Exception To User
Sep 21, 2023
49d9468
Addressed Comments
Sep 28, 2023
a734abb
optimized to remove explicit array copy
Sep 28, 2023
ed0ab72
Yetus checks
Sep 28, 2023
68d53b1
Addressing Comments
Nov 5, 2023
9bd995a
PR Checks
Nov 6, 2023
4b8a665
Teardown test FS
Nov 6, 2023
42fa529
Compilation Error Fix
Nov 6, 2023
7caa63f
Exception handling refactored
Nov 14, 2023
f5e0ae6
PR Checks
Nov 14, 2023
45ccd08
Yetus Checks
Nov 14, 2023
ff2d5ab
Missed code change
Nov 14, 2023
cfa404f
Addressed Comments
Nov 24, 2023
405e5a7
Fixing Merge Conflicts
Jan 2, 2024
590a003
Compilation Error Fix
Jan 3, 2024
5cb0d2b
Backmerge with Trunk
Jan 23, 2024
59a905f
Addressing Comments
Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,9 @@ public class AbfsConfiguration{
FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, DefaultValue = DEFAULT_ENABLE_ABFS_CHECKSUM_VALIDATION)
private boolean isChecksumValidationEnabled;

private String clientProvidedEncryptionKey;
private String clientProvidedEncryptionKeySHA;

public AbfsConfiguration(final Configuration rawConfig, String accountName)
throws IllegalAccessException, InvalidConfigurationValueException, IOException {
this.rawConfig = ProviderUtils.excludeIncompatibleCredentialProviders(
Expand Down
10 changes: 9 additions & 1 deletion ...p-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,11 @@
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.Base64;
import java.util.List;
import java.util.Locale;
import java.util.UUID;
Expand All @@ -37,6 +41,7 @@
import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsInvalidChecksumException;
import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsDriverException;
import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
import org.apache.hadoop.fs.azurebfs.utils.NamespaceUtil;
import org.apache.hadoop.fs.store.LogExactlyOnce;
import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.Permissions;
import org.apache.hadoop.fs.azurebfs.extensions.EncryptionContextProvider;
Expand Down Expand Up @@ -1074,11 +1079,14 @@ public AbfsRestOperation read(final String path,
ContextEncryptionAdapter contextEncryptionAdapter,
TracingContext tracingContext) throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
addCustomerProvidedKeyHeaders(requestHeaders);

AbfsHttpHeader rangeHeader = new AbfsHttpHeader(RANGE,
String.format("bytes=%d-%d", position, position + bufferLength - 1));
requestHeaders.add(rangeHeader);
addEncryptionKeyRequestHeaders(path, requestHeaders, false,
contextEncryptionAdapter, tracingContext);
requestHeaders.add(new AbfsHttpHeader(RANGE,
Copy link
Contributor

@steveloughran steveloughran Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this going in here when like 1085 sets this header too?

Copy link
Contributor Author

@anujmodi2021 anujmodi2021 Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be outdated.
Caused by merge conflicts but it was fixed in the latest commit: 590a003

More merge conflicts need to be resolved now.
Will take them up.

String.format("bytes=%d-%d", position, position + bufferLength - 1)));
requestHeaders.add(new AbfsHttpHeader(IF_MATCH, eTag));

// Add request header to fetch MD5 Hash of data returned by server.
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.