HBASE-27355 Separate meta read requests from master and client #7261

frostruan · 2025-09-01T07:51:04Z

No description provided.

Copilot

Pull Request Overview

This PR separates meta read requests from master and client by introducing different QoS (Quality of Service) levels for internal vs client meta operations. The change allows internal meta read requests to be processed in separate queues from client requests to improve system performance and isolation.

Key changes:

Introduces INTERNAL_READ_QOS constant for internal meta read operations
Updates MetaRWQueueRpcExecutor to dispatch internal reads to scan queues and client reads to read queues
Sets priority for internal meta operations using the new QoS level

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
RSAnnotationReadingPriorityFunction.java	Adds public constant `INTERNAL_READ_QOS` and makes class public
MetaRWQueueRpcExecutor.java	Implements custom dispatch logic to separate internal vs client reads using QoS levels
RpcExecutor.java	Extracts call queue handler factor configuration to protected method
RWQueueRpcExecutor.java	Adds getter method for number of scan queues
MetaTableAccessor.java	Sets internal read priority on meta operations and fixes logging issues
TestSimpleRpcScheduler.java	Updates test mocks to include priority headers and fixes configuration keys

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-02T02:50:17Z

.../src/main/java/org/apache/hadoop/hbase/regionserver/RSAnnotationReadingPriorityFunction.java

+  // QOS for internal meta read requests
+  public static int INTERNAL_READ_QOS = 250;


The INTERNAL_READ_QOS field should be declared as final since it represents a constant value that should not be modified after initialization.

Suggested change

// QOS for internal meta read requests

public static int INTERNAL_READ_QOS = 250;

public static final int INTERNAL_READ_QOS = 250;

frostruan · 2025-09-03T10:02:38Z

The failed unit tests looks unrelated. Add a new commit to run tests again.

frostruan · 2025-09-04T03:03:42Z

Some unit tests failed again. This is a little weird, let me check the reason.

mnpoonia

Much needed functionality.

frostruan · 2025-09-09T05:33:23Z

Still working on finding out why this PR makes some tests flaky .....

Apache9

Finally we got a clean run. But let's make sure there is no other problems...

The default config for running UTs usually uses small values, for example, hbase.regionserver.metahandler.count is 3 instead of 20, will this cause problems with the new code?

@frostruan Thanks.

Apache9 · 2025-09-13T12:03:39Z

hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetaRWQueueRpcExecutor.java

+
+  @Override
+  protected float getCallQueueHandlerFactor(Configuration conf) {
+    return conf.getFloat(META_CALL_QUEUE_HANDLER_FACTOR_CONF_KEY, 0.5f);


Mind explaining why here we use 0.5 instead of 0.1 for default value?

My initial thinking was that because we always cache meta table in memory, meta handlers would have lower latency than regular handlers. So, a larger handler factor would make meta handlers more efficient compared to regular handlers because it would reducce thread synchronization cost when consuming queues. I haven't tested this, it is just speculation.

frostruan · 2025-09-13T14:02:06Z

Finally we got a clean run. But let's make sure there is no other problems...

The default config for running UTs usually uses small values, for example, hbase.regionserver.metahandler.count is 3 instead of 20, will this cause problems with the new code?

@frostruan Thanks.

Thanks for reminding duo @Apache9 , it's possible, let me check again. But I just can't figure it out, this PR only changes the queue to which a request should be dispatched. Why does it break TestTableSnapshotScannerWithSFT and TestTableSnapshotScanner for 3 times ? These two don't seem to be related, and I can't reproduce it locally.

Apache9 · 2025-09-13T14:27:17Z

Finally we got a clean run. But let's make sure there is no other problems...
The default config for running UTs usually uses small values, for example, hbase.regionserver.metahandler.count is 3 instead of 20, will this cause problems with the new code?
@frostruan Thanks.

Thanks for reminding duo @Apache9 , it's possible, let me check again. But I just can't figure it out, this PR only changes the queue to which a request should be dispatched. Why does it break TestTableSnapshotScannerWithSFT and TestTableSnapshotScanner for 3 times ? These two don't seem to be related, and I can't reproduce it locally.

If meta handler is not correctly set, any tests can hang since the request to meta may be delayed or blocked, I guess. Or maybe it is just some machine issues...

frostruan · 2025-09-13T15:33:45Z

final boolean toScanQueue = getNumScanQueues() > 0 && level == INTERNAL_READ_QOS;
if number of scan queues is 0, the meta request will be dispatched to read queue, and for RWQueueRpcExecutor we get 1 write queue and 1 read queue at least, so it's not likely meta request will be blocked...

Apache-HBase · 2025-09-14T14:42:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	3m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	5m 46s		master passed
+1 💚	compile	4m 11s		master passed
+1 💚	checkstyle	0m 43s		master passed
+1 💚	spotbugs	1m 57s		master passed
+1 💚	spotless	0m 55s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 11s		the patch passed
+1 💚	compile	4m 0s		the patch passed
+1 💚	javac	4m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 45s		the patch passed
+1 💚	spotbugs	2m 3s		the patch passed
+1 💚	hadoopcheck	17m 29s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 57s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 14s		The patch does not generate ASF License warnings.
		56m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7261/7/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7261
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 1e310d9281c2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2ca67fb`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7261/7/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-09-14T17:54:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	3m 30s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	5m 38s		master passed
+1 💚	compile	1m 22s		master passed
+1 💚	javadoc	0m 35s		master passed
+1 💚	shadedjars	7m 11s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 10s		the patch passed
+1 💚	compile	1m 11s		the patch passed
+1 💚	javac	1m 11s		the patch passed
+1 💚	javadoc	0m 34s		the patch passed
+1 💚	shadedjars	7m 5s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	213m 2s		hbase-server in the patch passed.
		248m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7261/7/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7261
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux b64ed3ebd2c5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2ca67fb`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7261/7/testReport/
Max. process+thread count	4759 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7261/7/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Co-authored-by: huiruan <[email protected]> Signed-off-by: Duo Zhang <[email protected]> Reviewed-by: Aman Poonia <[email protected]> (cherry picked from commit 89416ce)

HBASE-27355 Separate meta read requests from master and client

72918b8

frostruan mentioned this pull request Sep 1, 2025

HBASE-27355 Separate meta read requests from master and client #4801

Closed