Skip to content

Forward-port "HBASE-28065 Corrupt HFile data is mishandled in several cases" to master #5398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ndimiduk merged 2 commits into from
Sep 21, 2023
Merged

Forward-port "HBASE-28065 Corrupt HFile data is mishandled in several cases" to master #5398

ndimiduk merged 2 commits into from
Sep 21, 2023

Conversation

@ndimiduk
Copy link
Member

@ndimiduk ndimiduk commented Sep 12, 2023

Porting #5384 to master.

@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@ndimiduk ndimiduk mentioned this pull request Sep 13, 2023
Merged
@Apache9
Copy link
Contributor

Apache9 commented Sep 19, 2023

The newly added UT failed?

@ndimiduk
Copy link
Member Author

ndimiduk commented Sep 19, 2023

The newly added UT failed?

Yeah, strange; passes locally. Pushed a new commit to give me more information about the failure in Jenkins.

@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@ndimiduk ndimiduk force-pushed the 28065-hfile-corrupt-reads branch from 607539d to d5a6ddf Compare September 19, 2023 13:39
@ndimiduk
Copy link
Member Author

ndimiduk commented Sep 19, 2023

@Apache9 I added some comments about this test failure on #5384.

@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@ndimiduk ndimiduk added the backport This PR is a back port of some issue or issues already committed to master label Sep 20, 2023
@ndimiduk
HBASE-28065 Corrupt HFile data is mishandled in several cases
d4718d2 
* when no block size is provided and there's not a preread headerBuf, treat the value with
  caution.
* verify HBase checksums before making use of the block header.
* inline verifyOnDiskSizeMatchesHeader to keep throw/return logic in the method body.
* separate validation of onDiskSizeWithHeader as input parameter from as read from block header
* simplify branching around fetching and populating onDiskSizeWithHeader.
* inline retrieving nextOnDiskBlockSize ; add basic validation.
* whenever a read is determined to be corrupt and fallback to HDFS checksum is necessary, also
  invalidate the cached value of headerBuf.
* build out a test suite covering various forms of block header corruption, for blocks in first
  and second positions.

Signed-off-by: Bryan Beaudreault <[email protected]>
@ndimiduk
adjustment for master branch
29011bf
@ndimiduk ndimiduk force-pushed the 28065-hfile-corrupt-reads branch from d5a6ddf to 29011bf Compare September 20, 2023 11:34
@Apache-HBase
Copy link

Apache-HBase commented Sep 20, 2023

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 27s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 2m 43s master passed
+1 💚 compile 2m 31s master passed
+1 💚 checkstyle 0m 34s master passed
+1 💚 spotless 0m 41s branch has no errors when running spotless:check.
+1 💚 spotbugs 1m 26s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 43s the patch passed
+1 💚 compile 2m 29s the patch passed
+1 💚 javac 2m 29s the patch passed
-0 ⚠️ checkstyle 0m 36s hbase-server: The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 10m 17s Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚 spotless 0m 41s patch has no errors when running spotless:check.
+1 💚 spotbugs 1m 38s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 8s The patch does not generate ASF License warnings.
33m 11s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #5398
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 6b0f361769a4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 8b2ca86
Default Java Eclipse Adoptium-11.0.17+8
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count 78 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/console
versions git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Sep 20, 2023

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 31s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 2m 49s master passed
+1 💚 compile 0m 48s master passed
+1 💚 shadedjars 4m 55s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 28s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 37s the patch passed
+1 💚 compile 0m 47s the patch passed
+1 💚 javac 0m 47s the patch passed
+1 💚 shadedjars 4m 50s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 25s the patch passed
_ Other Tests _
-1 ❌ unit 230m 6s hbase-server in the patch failed.
252m 48s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #5398
Optional Tests javac javadoc unit shadedjars compile
uname Linux 008b86a4a061 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 8b2ca86
Default Java Eclipse Adoptium-11.0.17+8
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/testReport/
Max. process+thread count 4487 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Sep 20, 2023

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 30s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 2m 19s master passed
+1 💚 compile 0m 41s master passed
+1 💚 shadedjars 4m 45s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 25s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 19s the patch passed
+1 💚 compile 0m 41s the patch passed
+1 💚 javac 0m 41s the patch passed
+1 💚 shadedjars 4m 42s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 23s the patch passed
_ Other Tests _
-1 ❌ unit 243m 49s hbase-server in the patch failed.
264m 40s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #5398
Optional Tests javac javadoc unit shadedjars compile
uname Linux 2e28c4b5d75c 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 8b2ca86
Default Java Temurin-1.8.0_352-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/testReport/
Max. process+thread count 4526 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5398/4/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ndimiduk ndimiduk merged commit 20c4136 into apache:master Sep 21, 2023
@ndimiduk ndimiduk deleted the 28065-hfile-corrupt-reads branch September 21, 2023 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport This PR is a back port of some issue or issues already committed to master

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ndimiduk @Apache-HBase @Apache9