Skip to content

HBASE-25894 Improve the performance for region load and region count related cost functions #3276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Apache9 merged 1 commit into from
May 25, 2021
Merged

HBASE-25894 Improve the performance for region load and region count related cost functions #3276

Apache9 merged 1 commit into from
May 25, 2021

Conversation

@Apache9
Copy link
Contributor

@Apache9 Apache9 commented May 18, 2021

No description provided.

@Apache9
Copy link
Contributor Author

Apache9 commented May 18, 2021

Depends on #3274

@Apache9 Apache9 self-assigned this May 18, 2021
@Apache-HBase
Copy link

Apache-HBase commented May 18, 2021

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 28s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 24s Maven dependency ordering for branch
+1 💚 mvninstall 3m 59s master passed
+1 💚 compile 1m 22s master passed
+1 💚 shadedjars 8m 40s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 58s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 15s Maven dependency ordering for patch
+1 💚 mvninstall 3m 43s the patch passed
+1 💚 compile 1m 19s the patch passed
+1 💚 javac 1m 19s the patch passed
+1 💚 shadedjars 8m 31s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 55s the patch passed
_ Other Tests _
+1 💚 unit 0m 31s hbase-balancer in the patch passed.
-1 ❌ unit 8m 2s hbase-server in the patch failed.
40m 52s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #3276
Optional Tests javac javadoc unit shadedjars compile
uname Linux fc76517af058 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 741b4b4
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/testReport/
Max. process+thread count 1146 (vs. ulimit of 30000)
modules C: hbase-balancer hbase-server U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 18, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 1s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for branch
+1 💚 mvninstall 3m 50s master passed
+1 💚 compile 3m 44s master passed
+1 💚 checkstyle 1m 16s master passed
+1 💚 spotbugs 2m 36s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for patch
+1 💚 mvninstall 3m 36s the patch passed
+1 💚 compile 3m 38s the patch passed
+1 💚 javac 3m 38s the patch passed
+1 💚 checkstyle 0m 14s The patch passed checkstyle in hbase-balancer
+1 💚 checkstyle 1m 2s hbase-server: The patch generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 18m 23s Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚 spotbugs 2m 55s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 26s The patch does not generate ASF License warnings.
51m 26s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3276
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux 50484385897a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 741b4b4
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 96 (vs. ulimit of 30000)
modules C: hbase-balancer hbase-server U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 18, 2021

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 3m 8s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for branch
+1 💚 mvninstall 5m 17s master passed
+1 💚 compile 1m 54s master passed
+1 💚 shadedjars 9m 11s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 1m 1s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
+1 💚 mvninstall 5m 1s the patch passed
+1 💚 compile 1m 55s the patch passed
+1 💚 javac 1m 55s the patch passed
+1 💚 shadedjars 9m 42s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 1m 1s the patch passed
_ Other Tests _
+1 💚 unit 0m 33s hbase-balancer in the patch passed.
-1 ❌ unit 13m 3s hbase-server in the patch failed.
54m 0s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3276
Optional Tests javac javadoc unit shadedjars compile
uname Linux b2c8d9601be4 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 741b4b4
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/testReport/
Max. process+thread count 649 (vs. ulimit of 30000)
modules C: hbase-balancer hbase-server U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 18, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 59s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for branch
+1 💚 mvninstall 3m 34s master passed
+1 💚 compile 3m 37s master passed
+1 💚 checkstyle 1m 16s master passed
+1 💚 spotbugs 2m 33s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for patch
+1 💚 mvninstall 3m 35s the patch passed
+1 💚 compile 3m 36s the patch passed
+1 💚 javac 3m 36s the patch passed
+1 💚 checkstyle 0m 14s The patch passed checkstyle in hbase-balancer
+1 💚 checkstyle 1m 1s hbase-server: The patch generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 18m 4s Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚 spotbugs 2m 55s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 25s The patch does not generate ASF License warnings.
50m 30s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3276
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux 9d7344048438 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 741b4b4
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 96 (vs. ulimit of 30000)
modules C: hbase-balancer hbase-server U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 18, 2021

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 28s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for branch
+1 💚 mvninstall 3m 40s master passed
+1 💚 compile 1m 20s master passed
+1 💚 shadedjars 8m 11s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 52s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 17s Maven dependency ordering for patch
+1 💚 mvninstall 3m 45s the patch passed
+1 💚 compile 1m 20s the patch passed
+1 💚 javac 1m 20s the patch passed
+1 💚 shadedjars 8m 32s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 58s the patch passed
_ Other Tests _
+1 💚 unit 0m 30s hbase-balancer in the patch passed.
-1 ❌ unit 150m 53s hbase-server in the patch failed.
183m 27s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #3276
Optional Tests javac javadoc unit shadedjars compile
uname Linux 127d326e0ee5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 741b4b4
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/testReport/
Max. process+thread count 3342 (vs. ulimit of 30000)
modules C: hbase-balancer hbase-server U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 18, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 21s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 12s Maven dependency ordering for branch
+1 💚 mvninstall 5m 24s master passed
+1 💚 compile 1m 56s master passed
+1 💚 shadedjars 9m 55s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 1m 0s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for patch
+1 💚 mvninstall 5m 43s the patch passed
+1 💚 compile 1m 56s the patch passed
+1 💚 javac 1m 56s the patch passed
+1 💚 shadedjars 9m 55s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 1m 6s the patch passed
_ Other Tests _
+1 💚 unit 0m 32s hbase-balancer in the patch passed.
+1 💚 unit 224m 7s hbase-server in the patch passed.
266m 10s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3276
Optional Tests javac javadoc unit shadedjars compile
uname Linux 745eba1f14a9 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 741b4b4
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/testReport/
Max. process+thread count 2778 (vs. ulimit of 30000)
modules C: hbase-balancer hbase-server U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 19, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 29s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for branch
+1 💚 mvninstall 4m 10s master passed
+1 💚 compile 3m 39s master passed
+1 💚 checkstyle 1m 19s master passed
+1 💚 spotbugs 2m 39s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
+1 💚 mvninstall 3m 39s the patch passed
+1 💚 compile 3m 38s the patch passed
+1 💚 javac 3m 38s the patch passed
+1 💚 checkstyle 0m 13s The patch passed checkstyle in hbase-balancer
+1 💚 checkstyle 1m 3s hbase-server: The patch generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 17m 58s Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚 spotbugs 2m 57s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 27s The patch does not generate ASF License warnings.
50m 45s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3276
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux 97dda1208770 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5b99409
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 96 (vs. ulimit of 30000)
modules C: hbase-balancer hbase-server U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 19, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 28s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for branch
+1 💚 mvninstall 3m 52s master passed
+1 💚 compile 1m 18s master passed
+1 💚 shadedjars 8m 3s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 54s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for patch
+1 💚 mvninstall 3m 38s the patch passed
+1 💚 compile 1m 19s the patch passed
+1 💚 javac 1m 19s the patch passed
+1 💚 shadedjars 8m 13s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 54s the patch passed
_ Other Tests _
+1 💚 unit 0m 31s hbase-balancer in the patch passed.
+1 💚 unit 147m 52s hbase-server in the patch passed.
179m 59s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #3276
Optional Tests javac javadoc unit shadedjars compile
uname Linux ce9914c88f1f 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5b99409
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/testReport/
Max. process+thread count 3466 (vs. ulimit of 30000)
modules C: hbase-balancer hbase-server U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 19, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 3m 7s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 20s Maven dependency ordering for branch
+1 💚 mvninstall 4m 59s master passed
+1 💚 compile 1m 40s master passed
+1 💚 shadedjars 9m 10s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 1m 0s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for patch
+1 💚 mvninstall 4m 45s the patch passed
+1 💚 compile 1m 40s the patch passed
+1 💚 javac 1m 40s the patch passed
+1 💚 shadedjars 9m 6s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 58s the patch passed
_ Other Tests _
+1 💚 unit 0m 31s hbase-balancer in the patch passed.
+1 💚 unit 208m 58s hbase-server in the patch passed.
248m 23s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3276
Optional Tests javac javadoc unit shadedjars compile
uname Linux a891de2b9390 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5b99409
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/testReport/
Max. process+thread count 2430 (vs. ulimit of 30000)
modules C: hbase-balancer hbase-server U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Copy link
Contributor Author

Apache9 commented May 20, 2021

Wrote a simple test to verify the performance improvement

/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.hadoop.hbase.master.balancer;

import java.util.List;
import java.util.Map;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;

public class StohasticBalancerPerformanceEvaluation extends StochasticBalancerTestBase {

  private void runBalancer(int numNodes, int numRegions, int numRegionsPerServer, int numTables,
    int replication) {
    Map<ServerName, List<RegionInfo>> serverMap =
      createServerMap(numNodes, numRegions, numRegionsPerServer, replication, numTables);
    Map<TableName, Map<ServerName, List<RegionInfo>>> LoadOfAllTable =
      (Map) mockClusterServersWithTables(serverMap);
    loadBalancer.balanceCluster(LoadOfAllTable);
  }

  public void run() throws Exception {
    beforeAllTests();
    loadBalancer.setRackManager(null);
    int numNodes = 1000;
    int numRegions = 100000; // 100 regions per RS
    int numRegionsPerServer = 80; // all servers except one
    int numTables = 100;
    int replication = 1;
    for (int i = 0; i < 3; i++) {
      runBalancer(numNodes, numRegions, numRegionsPerServer, numTables, replication);
    }
    long startTimeNs = System.nanoTime();
    runBalancer(numNodes, numRegions, numRegionsPerServer, numTables, replication);
    long costNs = System.nanoTime() - startTimeNs;
    System.out.println(String.format("Total time: %.3f secs", (double) costNs / 1000000000L));
  }

  public static void main(String[] args) throws Exception {
    new StohasticBalancerPerformanceEvaluation().run();
  }
}

After applied HBASE-25873, the result is
Total time: 51.990 secs
And after applied the patch here, the result is
Total time: 32.307 secs

About 37.8% down of the execution time.

@Apache-HBase
Copy link

Apache-HBase commented May 24, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 30s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 4m 3s master passed
+1 💚 compile 3m 14s master passed
+1 💚 checkstyle 1m 5s master passed
+1 💚 spotbugs 2m 6s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 59s the patch passed
+1 💚 compile 3m 29s the patch passed
+1 💚 javac 3m 29s the patch passed
+1 💚 checkstyle 1m 6s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 20m 5s Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚 spotbugs 2m 31s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 16s The patch does not generate ASF License warnings.
51m 46s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3276
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux e00516147d41 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / b02c810
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 96 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 24, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 32s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 54s master passed
+1 💚 compile 1m 2s master passed
+1 💚 shadedjars 8m 39s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 39s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 48s the patch passed
+1 💚 compile 1m 1s the patch passed
+1 💚 javac 1m 1s the patch passed
+1 💚 shadedjars 8m 27s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 40s the patch passed
_ Other Tests _
+1 💚 unit 150m 57s hbase-server in the patch passed.
181m 57s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #3276
Optional Tests javac javadoc unit shadedjars compile
uname Linux 6022185d17d4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / b02c810
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/testReport/
Max. process+thread count 3359 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented May 24, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 21s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 5m 19s master passed
+1 💚 compile 1m 23s master passed
+1 💚 shadedjars 9m 37s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 42s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 45s the patch passed
+1 💚 compile 1m 19s the patch passed
+1 💚 javac 1m 19s the patch passed
+1 💚 shadedjars 9m 0s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 41s the patch passed
_ Other Tests _
+1 💚 unit 218m 41s hbase-server in the patch passed.
254m 50s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3276
Optional Tests javac javadoc unit shadedjars compile
uname Linux 099027cdce98 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / b02c810
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/testReport/
Max. process+thread count 2569 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9 Apache9 merged commit 6a77872 into apache:master May 25, 2021
Apache9 added a commit that referenced this pull request May 25, 2021
@Apache9
HBASE-25894 Improve the performance for region load and region count …
2cb6cc8 
…related cost functions (#3276)

Signed-off-by: Yi Mei <[email protected]>
ndimiduk
ndimiduk reviewed Jun 1, 2021
View reviewed changes
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
}
}

void setCosts(Consumer<double[]> consumer) {
Copy link
Member

@ndimiduk ndimiduk Jun 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: having a setFoo method that is not a simple POJO field assignment is surprising!

Copy link
Contributor Author

@Apache9 Apache9 Jun 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any suggestion on the naming? fillCosts?

Copy link
Member

@ndimiduk ndimiduk Jun 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not off the top of my head, no. This interface for mutability by an external actor is a little strange. Maybe applyCostsConsumer ?

Copy link
Contributor Author

@Apache9 Apache9 Jun 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Let file another issue to land this naming change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ndimiduk ndimiduk ndimiduk left review comments

@mymeiyi mymeiyi mymeiyi approved these changes

Assignees

@Apache9 Apache9

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Apache9 @Apache-HBase @ndimiduk @mymeiyi