Skip to content

HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs #2845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steveloughran merged 18 commits into from
Aug 4, 2021
Merged

HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs #2845

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
e0b34b3
mask oids
sumangala17 Mar 31, 2021
dc64112
basic test
sumangala17 Mar 31, 2021
8e3ea6e
spotbug viz
sumangala17 Mar 31, 2021
57c437e
move to utils
sumangala17 Apr 9, 2021
e97a2f9
Merge branch 'trunk' into HADOOP-17618
sumangala17 Apr 9, 2021
407143f
tests
sumangala17 Apr 9, 2021
7af9f80
minor change
sumangala17 Apr 9, 2021
d72d490
fix test assuming unique sig
sumangala17 Apr 9, 2021
42d76d4
temp
sumangala17 Apr 16, 2021
3a95de3
tests
sumangala17 Apr 19, 2021
d635f07
fix null query fail
sumangala17 Apr 20, 2021
1e39143
rm =
sumangala17 Apr 20, 2021
4359718
pr revw
sumangala17 Apr 22, 2021
c69b392
mask || server logs, revw
sumangala17 Apr 23, 2021
cb46432
chkstyle
sumangala17 Apr 23, 2021
6283044
chkstyle
sumangala17 Apr 23, 2021
6a75563
Merge branch 'trunk' into HADOOP-17618
sumangala17 Apr 23, 2021
2cd7519
Merge branch 'trunk' into HADOOP-17618
sumangala17 Aug 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .../hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ public class AbfsHttpOperation implements AbfsPerfLoggable {
private static final Logger LOG = LoggerFactory.getLogger(AbfsHttpOperation.class);

public static final String SIGNATURE_QUERY_PARAM_KEY = "sig=";
private static final String[] SAS_OID_PARAM_KEYS = {"skoid", "saoid", "suoid"};

private static final int CONNECT_TIMEOUT = 30 * 1000;
private static final int READ_TIMEOUT = 30 * 1000;
Expand Down Expand Up @@ -547,6 +548,7 @@ public static String encodedUrlStr(String url) {
public String getSignatureMaskedUrl() {
if (this.maskedUrl == null) {
this.maskedUrl = getSignatureMaskedUrl(this.url.toString());
maskSASObjectIDs();
}
return this.maskedUrl;
}
Expand All @@ -558,6 +560,24 @@ public String getSignatureMaskedEncodedUrl() {
return this.maskedEncodedUrl;
}

public void maskSASObjectIDs() {
int oidStartIdx, ampIdx, oidEndIndex, qpStrIdx;
Copy link

@vinaysbadami vinaysbadami Apr 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to point of first use

Copy link
Contributor Author

@sumangala-patki sumangala-patki Apr 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

for (String qpKey : SAS_OID_PARAM_KEYS) {
qpStrIdx = maskedUrl.indexOf('&' + qpKey);
Copy link

@vinaysbadami vinaysbadami Apr 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this.maskedUrl to be consistent with rest of file

if (qpStrIdx == -1) {
qpStrIdx = maskedUrl.indexOf('?' + qpKey);
if (qpStrIdx == -1) {
continue;
}
}
oidStartIdx = qpStrIdx + qpKey.length() + 1;
ampIdx = maskedUrl.indexOf("&", oidStartIdx);
oidEndIndex = (ampIdx != -1) ? ampIdx : maskedUrl.length();
maskedUrl = maskedUrl.substring(0, oidStartIdx + 5) + "XXXX" + maskedUrl
.substring(oidEndIndex);
}
}

public static class AbfsHttpOperationWithFixedResult extends AbfsHttpOperation {
/**
* Creates an instance to represent fixed results.
Expand Down
26 changes: 26 additions & 0 deletions ...re/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelegationSAS.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@
*/
public class ITestAzureBlobFileSystemDelegationSAS extends AbstractAbfsIntegrationTest {
private static final String TEST_GROUP = UUID.randomUUID().toString();
private static final String[] OID_KEYS = {"skoid", "saoid", "suoid"};

private static final Logger LOG =
LoggerFactory.getLogger(ITestAzureBlobFileSystemDelegationSAS.class);
Expand Down Expand Up @@ -410,6 +411,31 @@ public void testSignatureMask() throws Exception {
.startsWith("sig%3DXXXX");
}

@Test
public void testSASObjectIDMask() throws IOException {
final AzureBlobFileSystem fs = getFileSystem();
AbfsHttpOperation abfsHttpOperation = fs.getAbfsClient().getAclStatus("/")
.getResult();
String url = abfsHttpOperation.getSignatureMaskedUrl();
String encodedUrl = abfsHttpOperation.getSignatureMaskedEncodedUrl();
int startIndex;
for (String qpKey : OID_KEYS) {
startIndex = url.indexOf(qpKey);
if (startIndex != -1) {
Assertions.assertThat(url.substring(startIndex + qpKey.length() + 5))
.describedAs("OID query param should be masked in url")
.startsWith("XXXX");
}
startIndex = encodedUrl.indexOf(qpKey);
if (startIndex != -1) {
Assertions
.assertThat(encodedUrl.substring(startIndex + qpKey.length() + 7))
.describedAs("OID query param should be masked in encoded url")
.startsWith("XXXX");
}
}
}

@Test
public void testSignatureMaskOnExceptionMessage() throws Exception {
intercept(IOException.class, "sig=XXXX",
Expand Down