Skip to content

Backport "HBASE-29003 Proper bulk load tracking (#6506) (#6921)" to branch-2.6 #6927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
rmdmattingly merged 1 commit into from
Apr 27, 2025
Merged

Backport "HBASE-29003 Proper bulk load tracking (#6506) (#6921)" to branch-2.6 #6927

rmdmattingly merged 1 commit into from
Apr 27, 2025

Conversation

@rmdmattingly
Copy link
Contributor

@rmdmattingly rmdmattingly commented Apr 23, 2025

No description provided.

@Apache-HBase

This comment has been minimized.

Sign in to view
@rmdmattingly @DieterDP-ng
HBASE-29003 Proper bulk load tracking (#6506) (#6921)
43afb37 
Signed-off-by: Ray Mattingly <[email protected]>

The HBase backup mechanism keeps track of which HFiles
were bulk loaded, so they can be included in incremental
backups.

Before this ticket, these bulk load records were only
deleted when an incremental backup is created. This
commit adds 2 more locations:

1) after a full backup. Since a full backup already
captures all data, this meant that unnecessary HFiles
were being included in the next incremental backup.

2) after a table delete/truncate/CF-deletion. Previously,
if an HFile was loaded before a table was cleared, the next
incremental backup would effectively still include the
HFile. This lead to incorrect data being restored.

This commit also completely refactors & simplifies the
test for this functionality.

Co-authored-by: DieterDP <[email protected]>
@rmdmattingly rmdmattingly force-pushed the HBASE-29003-branch-2.6 branch from f24a09d to 43afb37 Compare April 25, 2025 15:04
@Apache-HBase
Copy link

Apache-HBase commented Apr 26, 2025

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 58s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
_ branch-2.6 Compile Tests _
+0 🆗 mvndep 0m 58s Maven dependency ordering for branch
+1 💚 mvninstall 5m 18s branch-2.6 passed
+1 💚 compile 4m 32s branch-2.6 passed
+1 💚 checkstyle 0m 45s branch-2.6 passed
+1 💚 spotbugs 2m 5s branch-2.6 passed
+1 💚 spotless 0m 53s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 12s Maven dependency ordering for patch
+1 💚 mvninstall 2m 59s the patch passed
+1 💚 compile 3m 17s the patch passed
+1 💚 javac 3m 17s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 46s the patch passed
+1 💚 spotbugs 2m 12s the patch passed
+1 💚 hadoopcheck 16m 21s Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.
+1 💚 spotless 0m 44s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 18s The patch does not generate ASF License warnings.
44m 32s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #6927
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname Linux 63b4ce09bb00 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 43afb37
Default Java Eclipse Adoptium-11.0.23+9
Max. process+thread count 78 (vs. ulimit of 30000)
modules C: hbase-server hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Apr 26, 2025

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 47s Docker mode activated.
-0 ⚠️ yetus 0m 6s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+0 🆗 mvndep 0m 10s Maven dependency ordering for branch
+1 💚 mvninstall 3m 13s branch-2.6 passed
+1 💚 compile 1m 15s branch-2.6 passed
+1 💚 javadoc 0m 41s branch-2.6 passed
+1 💚 shadedjars 6m 29s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 20s Maven dependency ordering for patch
+1 💚 mvninstall 3m 54s the patch passed
+1 💚 compile 1m 33s the patch passed
+1 💚 javac 1m 33s the patch passed
+1 💚 javadoc 0m 52s the patch passed
+1 💚 shadedjars 7m 12s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 210m 48s hbase-server in the patch passed.
+1 💚 unit 16m 6s hbase-backup in the patch passed.
258m 43s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #6927
Optional Tests javac javadoc unit compile shadedjars
uname Linux 79934f8aa11f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 43afb37
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/testReport/
Max. process+thread count 4395 (vs. ulimit of 30000)
modules C: hbase-server hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Apr 26, 2025

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 52s Docker mode activated.
-0 ⚠️ yetus 0m 6s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+0 🆗 mvndep 0m 12s Maven dependency ordering for branch
+1 💚 mvninstall 3m 36s branch-2.6 passed
+1 💚 compile 1m 25s branch-2.6 passed
+1 💚 javadoc 0m 47s branch-2.6 passed
+1 💚 shadedjars 6m 42s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 18s Maven dependency ordering for patch
+1 💚 mvninstall 3m 10s the patch passed
+1 💚 compile 1m 13s the patch passed
+1 💚 javac 1m 13s the patch passed
+1 💚 javadoc 0m 40s the patch passed
+1 💚 shadedjars 7m 16s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 218m 4s hbase-server in the patch passed.
+1 💚 unit 15m 37s hbase-backup in the patch passed.
264m 32s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #6927
Optional Tests javac javadoc unit compile shadedjars
uname Linux e38c19ed8eec 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 43afb37
Default Java Eclipse Adoptium-11.0.23+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/testReport/
Max. process+thread count 4131 (vs. ulimit of 30000)
modules C: hbase-server hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Apr 26, 2025

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 57s Docker mode activated.
-0 ⚠️ yetus 0m 8s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for branch
+1 💚 mvninstall 5m 56s branch-2.6 passed
+1 💚 compile 2m 14s branch-2.6 passed
+1 💚 javadoc 1m 31s branch-2.6 passed
+1 💚 shadedjars 10m 6s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for patch
+1 💚 mvninstall 5m 38s the patch passed
+1 💚 compile 2m 8s the patch passed
+1 💚 javac 2m 8s the patch passed
+1 💚 javadoc 1m 26s the patch passed
+1 💚 shadedjars 8m 51s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 273m 52s hbase-server in the patch passed.
+1 💚 unit 13m 36s hbase-backup in the patch passed.
332m 31s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #6927
Optional Tests javac javadoc unit compile shadedjars
uname Linux 10d1f29a68dc 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 43afb37
Default Java Temurin-1.8.0_412-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/testReport/
Max. process+thread count 4111 (vs. ulimit of 30000)
modules C: hbase-server hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@rmdmattingly rmdmattingly merged commit 18a28f8 into branch-2.6 Apr 27, 2025
1 check passed
@rmdmattingly rmdmattingly deleted the HBASE-29003-branch-2.6 branch April 27, 2025 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rmdmattingly @Apache-HBase