HBASE-23205 Correctly update the position of WALs currently being replicated #980

JeongDaeKim · 2020-01-03T09:35:05Z

https://issues.apache.org/jira/browse/HBASE-23205

related PR : #944

…licated

Apache-HBase · 2020-01-03T12:15:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ branch-1.4 Compile Tests _
+1 💚	mvninstall	8m 7s	branch-1.4 passed
+1 💚	compile	0m 39s	branch-1.4 passed with JDK v1.8.0_232
+1 💚	compile	0m 45s	branch-1.4 passed with JDK v1.7.0_242
+1 💚	checkstyle	1m 35s	branch-1.4 passed
+1 💚	shadedjars	2m 47s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	branch-1.4 passed with JDK v1.8.0_232
+1 💚	javadoc	0m 40s	branch-1.4 passed with JDK v1.7.0_242
+0 🆗	spotbugs	2m 49s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 47s	branch-1.4 passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 56s	the patch passed
+1 💚	compile	0m 39s	the patch passed with JDK v1.8.0_232
+1 💚	javac	0m 39s	the patch passed
+1 💚	compile	0m 47s	the patch passed with JDK v1.7.0_242
+1 💚	javac	0m 47s	the patch passed
-1 ❌	checkstyle	1m 34s	hbase-server: The patch generated 1 new + 25 unchanged - 10 fixed = 26 total (was 35)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedjars	2m 39s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	2m 30s	Patch does not cause any errors with Hadoop 2.7.7.
+1 💚	javadoc	0m 30s	the patch passed with JDK v1.8.0_232
+1 💚	javadoc	0m 40s	the patch passed with JDK v1.7.0_242
+1 💚	findbugs	2m 51s	the patch passed
		_ Other Tests _
+1 💚	unit	118m 36s	hbase-server in the patch passed.
+1 💚	asflicense	0m 34s	The patch does not generate ASF License warnings.
		155m 34s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-980/1/artifact/out/Dockerfile
GITHUB PR	#980
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 7a7589a46ec7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-980/out/precommit/personality/provided.sh
git revision	branch-1.4 / `aaa2266`
Default Java	1.7.0_242
Multi-JDK versions	/usr/lib/jvm/zulu-8-amd64:1.8.0_232 /usr/lib/jvm/zulu-7-amd64:1.7.0_242
checkstyle	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-980/1/artifact/out/diff-checkstyle-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-980/1/testReport/
Max. process+thread count	4346 (vs. ulimit of 10000)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-980/1/console
versions	git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

saintstack

Looking good. Need to come back to this. Substantial change.

saintstack · 2020-01-03T17:31:13Z

...server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java

+  @VisibleForTesting
+  public Path getLastLoggedPath() {
+    for (ReplicationSourceShipperThread worker : workerThreads.values()) {
+      return worker.getLastLoggedPath();


Are the workerThreads sorted so we get last logged first?

Not sorted, It's only for testing in non-multi wal environment, as same as ReplicationSource.getCurrentPath().

wchevreuil

Looks in line with the changes of PR #944 (merged into branch-1). So LGTM, but let's wait for @saintstack review.

saintstack

I don't know this code well but what is here looks good to me. A few questions in the below.

saintstack · 2020-01-07T04:56:35Z

...server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java

-      long position = worker.getCurrentPosition();
-      Path currentPath = worker.getCurrentPath();
+      long position = worker.getLastLoggedPosition();
+      Path currentPath = worker.getLastLoggedPath();


This is not for test, right? We iterate the workers twice -- once to get path and then again to get position. Don't want to return a Pair with Path and offset so just iterate once?

Hmm... reading later in the patch, I see need to have them distinct. For sure we'll get the right offset for the selected path?

Yes, not for test and not the same one. each thread have it's own path and position. no need to iterate

saintstack · 2020-01-07T05:34:23Z

...server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java

-          // set "ageOfLastShippedOp" to <now> to indicate that we're current
-          metrics.setAgeOfLastShippedOp(EnvironmentEdgeManager.currentTime(), walGroupId);
-        }
+        updateLogPosition(lastReadPosition);


Will we be doing a bunch of spinning updating same value over and over w/o these checks in place that look to see if position has changed? Will we be burning CPU to no end?

It won't be updated with the same value repeatably. entryReader will put a batch only when read position is changed.

saintstack · 2020-01-07T05:34:48Z

...server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java

    }

    private void updateLogPosition(long lastReadPosition) {
-      manager.setPendingShipment(false);


Its ok to remove this setting?

We don't do pending flag anymore?

Yes, we don't need the flag anymore. it was required because not only shipper but also reader thread can update log position, but it had some issues about concurrency and high update frequency. With this PR, when wal rolled or log position changed, reader will put a batch for shipper to update log position.

JeongDaeKim · 2020-01-09T07:31:54Z

Thanks all for approvals! Could you merge this, if no more comments or questions?

wchevreuil · 2020-01-09T10:10:06Z

Thanks again for all the work here, @JeongDaeKim !

…licated (apache#980) Signed-off-by: stack <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> (cherry picked from commit 879b7ea)

HBASE-23205 Correctly update the position of WALs currently being rep…

8aa9447

…licated

saintstack reviewed Jan 3, 2020

View reviewed changes

wchevreuil approved these changes Jan 6, 2020

View reviewed changes

saintstack approved these changes Jan 7, 2020

View reviewed changes

wchevreuil merged commit 879b7ea into apache:branch-1.4 Jan 9, 2020

JeongDaeKim deleted the HBASE-23205-branch-1.4 branch January 13, 2020 01:29

HBASE-23205 Correctly update the position of WALs currently being replicated #980

HBASE-23205 Correctly update the position of WALs currently being replicated #980

Uh oh!

Conversation

JeongDaeKim commented Jan 3, 2020

Uh oh!

Apache-HBase commented Jan 3, 2020

Uh oh!

saintstack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wchevreuil left a comment

Choose a reason for hiding this comment

Uh oh!

saintstack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JeongDaeKim commented Jan 9, 2020

Uh oh!

wchevreuil commented Jan 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants