HADOOP-18830. Cut S3 Select (#6144)

[steveloughran]

Cut out S3 Select

• leave public/unstable constants alone
• s3guard tool will fail with error
• s3afs. path capability will fail
• openFile() will fail with specific error
• s3 select doc updated
• New test: ITestSelectUnsupported verifies new failure handling above

This is the v1 SDK version of the feature.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[steveloughran]

testing s3 london -Dscale using noaa as landsat replacement.

[ERROR] Errors: 
[ERROR]   ITestDelegatedMRJob.testJobSubmissionCollectsTokens[0] » AccessDenied s3a://la...
[ERROR]   ITestDelegatedMRJob.testJobSubmissionCollectsTokens[1] » AccessDenied s3a://la...
[ERROR]   ITestDelegatedMRJob.testJobSubmissionCollectsTokens[2] » AccessDenied s3a://la...
[ERROR]   ITestRoleDelegationInFilesystem>ITestSessionDelegationInFilesystem.testDelegatedFileSystem:334->ITestSessionDelegationInFilesystem.readLandsatMetadata:601 » AccessDenied
[ERROR]   ITestSessionDelegationInFilesystem.testDelegatedFileSystem:334->readLandsatMetadata:601 » AccessDenied
[ERROR]   ITestS3AInputStreamPerformance.testRandomIONormalPolicy:494->executeRandomIO:534 » EOF
[ERROR]   ITestS3AInputStreamPerformance.testRandomIORandomPolicy:485->executeRandomIO:534 » EOF

[steveloughran]

and final failure, again from the move off landsat

[ERROR] testCacheFileExistence(org.apache.hadoop.fs.s3a.ITestS3APrefetchingCacheFiles)  Time elapsed: 1.941 s  <<< ERROR!
java.io.EOFException: Attempted to seek or read past the end of the file 16777216

If yetus is happy I will merge

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	34m 37s		branch-3.3 passed
+1 💚	compile	0m 28s		branch-3.3 passed
+1 💚	checkstyle	0m 22s		branch-3.3 passed
+1 💚	mvnsite	0m 32s		branch-3.3 passed
+1 💚	javadoc	0m 27s		branch-3.3 passed
+1 💚	spotbugs	0m 49s		branch-3.3 passed
+1 💚	shadedclient	23m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 12s	/patch-mvninstall-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	compile	0m 12s	/patch-compile-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	javac	0m 12s	/patch-compile-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 11s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 3 new + 6 unchanged - 8 fixed = 9 total (was 14)
-1 ❌	mvnsite	0m 14s	/patch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	javadoc	0m 14s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws.txt	hadoop-tools_hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
-1 ❌	spotbugs	0m 12s	/patch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	shadedclient	22m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 15s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	asflicense	0m 22s		The patch does not generate ASF License warnings.
		90m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6523/1/artifact/out/Dockerfile
GITHUB PR	#6523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint
uname	Linux e043a3bfc6e3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / f1e775b
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6523/1/testReport/
Max. process+thread count	557 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6523/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	35m 8s		branch-3.3 passed
+1 💚	compile	0m 22s		branch-3.3 passed
+1 💚	checkstyle	0m 18s		branch-3.3 passed
+1 💚	mvnsite	0m 26s		branch-3.3 passed
+1 💚	javadoc	0m 23s		branch-3.3 passed
+1 💚	spotbugs	0m 46s		branch-3.3 passed
+1 💚	shadedclient	21m 42s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	0m 20s		the patch passed
-1 ❌	javac	0m 20s	/results-compile-javac-hadoop-tools_hadoop-aws.txt	hadoop-tools_hadoop-aws generated 10 new + 23 unchanged - 2 fixed = 33 total (was 25)
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 13s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 6 unchanged - 8 fixed = 7 total (was 14)
+1 💚	mvnsite	0m 23s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed
+1 💚	spotbugs	0m 48s		the patch passed
+1 💚	shadedclient	20m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 2s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		87m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6523/2/artifact/out/Dockerfile
GITHUB PR	#6523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint
uname	Linux 6938203df900 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 38f839b
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6523/2/testReport/
Max. process+thread count	550 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6523/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

blanks:

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md:898:      SERVER_SIDE_ENCRYPTION_ALGORITHM);  

javac, use of the newly deprecated refs, plus something possibly more serious

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java:61:38:[deprecation] SelectConstants in org.apache.hadoop.fs.s3a.select has been deprecated
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java:78:45:[deprecation] SelectConstants in org.apache.hadoop.fs.s3a.select has been deprecated
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/OpenFileSupport.java:175:33:[deprecation] SelectConstants in org.apache.hadoop.fs.s3a.select has been deprecated
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/OpenFileSupport.java:177:48:[deprecation] SelectConstants in org.apache.hadoop.fs.s3a.select has been deprecated
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/OpenFileSupport.java:180:31:[deprecation] SelectConstants in org.apache.hadoop.fs.s3a.select has been deprecated
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyFromLocalOperation.java:235:16:[unchecked] unchecked method invocation: method sort in interface List is applied to given types
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestSelectUnsupported.java:31:45:[deprecation] SelectConstants in org.apache.hadoop.fs.s3a.select has been deprecated
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestSelectUnsupported.java:32:45:[deprecation] SelectConstants in org.apache.hadoop.fs.s3a.select has been deprecated
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java:49:56:[deprecation] none() in ExpectedException has been deprecated
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java:207:6:[deprecation] <T>assertThat(String,T,Matcher<? super T>) in Assert has been deprecated

• one unused import

./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java:29:import com.amazonaws.services.s3.model.AmazonS3Exception;:8: Unused import - com.amazonaws.services.s3.model.AmazonS3Exception. [UnusedImports]

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	33m 31s		branch-3.3 passed
+1 💚	compile	0m 24s		branch-3.3 passed
+1 💚	checkstyle	0m 21s		branch-3.3 passed
+1 💚	mvnsite	0m 28s		branch-3.3 passed
+1 💚	javadoc	0m 24s		branch-3.3 passed
+1 💚	spotbugs	0m 51s		branch-3.3 passed
+1 💚	shadedclient	21m 49s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	0m 21s		the patch passed
-1 ❌	javac	0m 21s	/results-compile-javac-hadoop-tools_hadoop-aws.txt	hadoop-tools_hadoop-aws generated 9 new + 23 unchanged - 2 fixed = 32 total (was 25)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		hadoop-tools/hadoop-aws: The patch generated 0 new + 6 unchanged - 8 fixed = 6 total (was 14)
+1 💚	mvnsite	0m 25s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed
+1 💚	spotbugs	0m 45s		the patch passed
+1 💚	shadedclient	20m 31s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 3s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		89m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6523/3/artifact/out/Dockerfile
GITHUB PR	#6523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint
uname	Linux 24fd4bed0c7b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / cd085ad
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6523/3/testReport/
Max. process+thread count	553 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6523/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.