Skip to content

HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7 #1453

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
saintstack merged 3 commits into from
Apr 13, 2020
Merged

HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7 #1453

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9736753
HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7
Apr 8, 2020
ad46d05
HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7
Apr 8, 2020
148dc5f
Merge pull request #1 from apache/master
francisjjf Apr 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions hbase-resource-bundle/src/main/resources/supplemental-models.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2096,4 +2096,19 @@ Copyright (c) 2007-2017 The JRuby project
</licenses>
</project>
</supplement>
<supplement>
<project>
<groupId>jline</groupId>
<artifactId>jline</artifactId>
<name>JLine</name>
<version>2.11</version>
<licenses>
<license>
<name>BSD License</name>
<url>http://www.opensource.org/licenses/bsd-license.php</url>
<distribution>repo</distribution>
</license>
</licenses>
</project>
</supplement>
</supplementalDataModels>
4 changes: 2 additions & 2 deletions hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,8 +329,8 @@ AssignmentManager assignmentManager = master.getAssignmentManager();
</tr>
<tr>
<td>ZooKeeper Client Version</td>
<td><% org.apache.zookeeper.Version.getVersion() %>, revision=<% org.apache.zookeeper.Version.getRevision() %></td>
<td>ZooKeeper client version and revision</td>
<td><% org.apache.zookeeper.Version.getVersion() %>, revision=<% org.apache.zookeeper.Version.getRevisionHash() %></td>
<td>ZooKeeper client version and revision hash</td>
</tr>
<tr>
<td>ZooKeeper Client Compiled</td>
Expand Down
4 changes: 2 additions & 2 deletions hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,8 +197,8 @@ org.apache.hadoop.hbase.zookeeper.MasterAddressTracker;
</tr>
<tr>
<td>ZooKeeper Client Version</td>
<td><% org.apache.zookeeper.Version.getVersion() %>, revision=<% org.apache.zookeeper.Version.getRevision() %></td>
<td>ZooKeeper client version and revision</td>
<td><% org.apache.zookeeper.Version.getVersion() %>, revision=<% org.apache.zookeeper.Version.getRevisionHash() %></td>
<td>ZooKeeper client version and revision hash</td>
</tr>
<tr>
<td>ZooKeeper Client Compiled</td>
Expand Down
3 changes: 3 additions & 0 deletions ...ver/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionMoveAndAbandon.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,9 @@ public void test() throws Exception {
cluster.killRegionServer(rs2.getServerName());
UTIL.waitFor(30_000, () -> rs2.isStopped() && !rs2.isAlive());
UTIL.waitFor(30_000, () -> rs1.isStopped() && !rs1.isAlive());
// make sure none of regionserver threads is alive.
UTIL.waitFor(30_000, () ->
UTIL.getMiniHBaseCluster().getLiveRegionServerThreads().isEmpty());
// Start up everything again
Copy link
Contributor Author

@francisjjf francisjjf Apr 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a rs all down check here to make the testcase stable.

LOG.info("Starting cluster");
UTIL.getMiniHBaseCluster().startMaster();
Expand Down
4 changes: 2 additions & 2 deletions hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/HQuorumPeer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
import java.io.PrintWriter;
import java.net.InetAddress;
import java.net.NetworkInterface;
import java.net.UnknownHostException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Enumeration;
Expand All @@ -42,6 +41,7 @@
import org.apache.yetus.audience.InterfaceStability;
import org.apache.zookeeper.server.ServerConfig;
import org.apache.zookeeper.server.ZooKeeperServerMain;
import org.apache.zookeeper.server.admin.AdminServer;
import org.apache.zookeeper.server.quorum.QuorumPeerConfig;
import org.apache.zookeeper.server.quorum.QuorumPeerMain;
import org.slf4j.Logger;
Expand Down Expand Up @@ -87,7 +87,7 @@ public static void main(String[] args) {
}

private static void runZKServer(QuorumPeerConfig zkConfig)
throws UnknownHostException, IOException {
throws IOException, AdminServer.AdminServerException {
if (zkConfig.isDistributed()) {
QuorumPeerMain qp = new QuorumPeerMain();
qp.runFromConfig(zkConfig);
Expand Down
9 changes: 5 additions & 4 deletions hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Threads;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.zookeeper.common.X509Exception;
import org.apache.zookeeper.server.NIOServerCnxnFactory;
import org.apache.zookeeper.server.ZooKeeperServer;
import org.apache.zookeeper.server.persistence.FileTxnLog;
Expand Down Expand Up @@ -421,8 +422,8 @@ private static boolean waitForServerDown(int port, long timeout) throws IOExcept
long start = System.currentTimeMillis();
while (true) {
try {
send4LetterWord(HOST, port, "stat", (int)timeout);
} catch (IOException e) {
send4LetterWord(HOST, port, "stat", false, (int)timeout);
} catch (IOException | X509Exception.SSLContextException e) {
return true;
}

Expand All @@ -444,7 +445,7 @@ private static boolean waitForServerUp(int port, long timeout) throws IOExceptio
long start = System.currentTimeMillis();
while (true) {
try {
String result = send4LetterWord(HOST, port, "stat", (int)timeout);
String result = send4LetterWord(HOST, port, "stat", false, (int)timeout);
if (result.startsWith("Zookeeper version:") && !result.contains("READ-ONLY")) {
return true;
} else {
Expand All @@ -453,7 +454,7 @@ private static boolean waitForServerUp(int port, long timeout) throws IOExceptio
} catch (ConnectException e) {
// ignore as this is expected, do not log stacktrace
LOG.info("{}:{} not up: {}", HOST, port, e.toString());
} catch (IOException e) {
} catch (IOException | X509Exception.SSLContextException e) {
// ignore as this is expected
LOG.info("{}:{} not up", HOST, port, e);
}
Expand Down
6 changes: 3 additions & 3 deletions hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKMainServer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HBaseInterfaceAudience;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.ZooKeeperMain;
import org.apache.zookeeper.cli.CliException;


/**
Expand Down Expand Up @@ -57,11 +57,11 @@ public HACK_UNTIL_ZOOKEEPER_1897_ZooKeeperMain(String[] args)

/**
* Run the command-line args passed. Calls System.exit when done.
* @throws KeeperException if an unexpected ZooKeeper exception happens
* @throws IOException in case of a network failure
* @throws InterruptedException if the ZooKeeper client closes
* @throws CliException if the ZooKeeper exception happens in cli command
*/
void runCmdLine() throws KeeperException, IOException, InterruptedException {
void runCmdLine() throws IOException, InterruptedException, CliException {
processCmd(this.cl);
System.exit(0);
}
Expand Down
4 changes: 4 additions & 0 deletions hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKWatcher.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -560,6 +560,10 @@ private void connectionEvent(WatchedEvent event) {
LOG.debug(prefix("Received Disconnected from ZooKeeper, ignoring"));
break;

case Closed:
LOG.debug(prefix("ZooKeeper client closed, ignoring"));
break;

case Expired:
String msg = prefix(this.identifier + " received expired from " +
"ZooKeeper, aborting");
Expand Down
5 changes: 2 additions & 3 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1530,9 +1530,8 @@
<protobuf.plugin.version>0.6.1</protobuf.plugin.version>
<thrift.path>thrift</thrift.path>
<thrift.version>0.12.0</thrift.version>
<zookeeper.version>3.4.14</zookeeper.version>
<!-- What ZooKeeper 3.4.x depends on and nothing more -->
<jline.version>0.9.94</jline.version>
<zookeeper.version>3.5.7</zookeeper.version>
Copy link
Contributor

@symat symat Apr 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also update the jline version? I think it was added because of zookeeper, see https://issues.apache.org/jira/browse/HBASE-20108

zookeeper 3.5.7 is using jline 2.11

however, if we increment the JLine version, then the assembly:single maven target will fail, as the license part in JLine pom changed a bit (they named it 'The BSD License' instead of 'BSD License'). To make it work, we need to add a new supplement to supplemental-models.xml like:

  <supplement>
    <project>
      <groupId>jline</groupId>
      <artifactId>jline</artifactId>
      <name>JLine</name>
      <version>2.11</version>
      <licenses>
        <license>
          <name>BSD License</name>
          <url>http://www.opensource.org/licenses/bsd-license.php</url>
          <distribution>repo</distribution>
        </license>
      </licenses>
    </project>
  </supplement>

<jline.version>2.11</jline.version>
<slf4j.version>1.7.30</slf4j.version>
<clover.version>4.0.3</clover.version>
<jamon-runtime.version>2.4.1</jamon-runtime.version>
Expand Down