Skip to content

HADOOP-15984. Update jersey from 1.19 to 2.x. #7019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
slfan1989 merged 17 commits into from
Jan 18, 2025
Merged

HADOOP-15984. Update jersey from 1.19 to 2.x. #7019

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
92160ab
HADOOP-15984. Update jersey from 1.19 to 2.x.
slfan1989 Jan 3, 2025
fc8f03e
HADOOP-15984. Improve Some Code.
slfan1989 Jan 4, 2025
fff1b6a
HADOOP-15984. Improve Some Code.
slfan1989 Jan 5, 2025
3237736
HADOOP-15984. Improve Some Code.
slfan1989 Jan 5, 2025
31d28a7
HADOOP-15984. Fix CheckStyle.
slfan1989 Jan 6, 2025
9601cb9
HADOOP-15984. Improve Some Code.
slfan1989 Jan 6, 2025
c1eca7e
HADOOP-15984. Improve Some Code.
slfan1989 Jan 9, 2025
96a29fc
Merge branch 'apache:trunk' into HADOOP-15984
slfan1989 Jan 9, 2025
f8530cb
HADOOP-15984. Improve Some Code.
slfan1989 Jan 10, 2025
c2484ae
HADOOP-15984. Improve Some Code.
slfan1989 Jan 11, 2025
290cae1
HADOOP-15984. Improve Some Code.
slfan1989 Jan 12, 2025
ffd73e4
HADOOP-15984. Improve Some Code.
slfan1989 Jan 12, 2025
dde4dee
HADOOP-15984. Improve Some Code.
slfan1989 Jan 13, 2025
0dee8ac
HADOOP-15984. Improve Some Code.
slfan1989 Jan 14, 2025
842886f
HADOOP-15984. Improve Some Code.
slfan1989 Jan 16, 2025
29fdc9c
HADOOP-15984. Improve Some Code.
slfan1989 Jan 16, 2025
8d636ce
HADOOP-15984. Revert Jenkinsfile.
slfan1989 Jan 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
22 changes: 15 additions & 7 deletions LICENSE-binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -498,12 +498,16 @@ org.slf4j:slf4j-reload4j:1.7.36
CDDL 1.1 + GPLv2 with classpath exception
-----------------------------------------

com.github.pjfanning:jersey-json:1.22.0
com.sun.jersey:jersey-client:1.19.4
com.sun.jersey:jersey-core:1.19.4
com.sun.jersey:jersey-guice:1.19.4
com.sun.jersey:jersey-server:1.19.4
com.sun.jersey:jersey-servlet:1.19.4
org.glassfish.jersey.core:jersey-common:2.46
Copy link
Member

@pjfanning pjfanning Jan 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like the wrong license group

https://mvnrepository.com/artifact/org.glassfish.jersey.core/jersey-common/2.46

jersey 2 is licensed under Apache License (very important) and 3 other licenses.

<licenses>
<license>
<name>EPL 2.0</name>
<url>http://www.eclipse.org/legal/epl-2.0</url>
<distribution>repo</distribution>
<comments>Except for Guava, JSR-166 files, Dropwizard Monitoring inspired classes, ASM and Jackson JAX-RS Providers. See also https://github.com/eclipse-ee4j/jersey/blob/master/NOTICE.md</comments>
</license>
<license>
<name>The GNU General Public License (GPL), Version 2, With Classpath Exception</name>
<url>https://www.gnu.org/software/classpath/license.html</url>
<distribution>repo</distribution>
<comments>Except for Guava, and JSR-166 files. See also https://github.com/eclipse-ee4j/jersey/blob/master/NOTICE.md</comments>
</license>
<license>
<name>Apache License, 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.html</url>
<distribution>repo</distribution>
<comments>Google Guava @ org.glassfish.jersey.internal.guava</comments>
</license>
<license>
<name>Public Domain</name>
<url>https://creativecommons.org/publicdomain/zero/1.0/</url>
<distribution>repo</distribution>
<comments>JSR-166 Extension to JEP 266 @ org.glassfish.jersey.internal.jsr166</comments>
</license>
</licenses>

Copy link
Contributor Author

@slfan1989 slfan1989 Jan 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for your reply! I will improve it.

Copy link
Member

@pjfanning pjfanning Jan 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if jersey 2 is dual licensed. Reading its license and notice makes it look like it is EPL-2.0 licensed by with some parts licensed under 1 or more of the other 3 licenses.

Copy link
Contributor Author

@slfan1989 slfan1989 Jan 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can wait for input from the other members, as I'm also unsure about the best way to improve it. Once we have everyone's feedback, we can decide on the changes together. It's a good sign that we've already started reviewing this PR.

Copy link
Contributor

@steveloughran steveloughran Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm in the "merge this and then deal with the details" mode right now. Get this in and then we can start the real burndown for java17.

Copy link
Contributor Author

@slfan1989 slfan1989 Jan 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@steveloughran Thank you for the acknowledgment! If there are no further comments, I plan to merge this PR in two days. After multiple rounds of testing, everything is performing as expected. I’ve personally debugged each unit test improvement to ensure they work in most scenarios. Of course, there may still be some occasional issues with individual tests (though the likelihood is low), and we can continue refining them in the future.

Copy link
Contributor

@Hexiaoqiao Hexiaoqiao Jan 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is OK to located this package to Apache License or MIT if it is dual licensed IMO, if we could not make sure, maybe refer Spark https://github.com/apache/spark/blob/master/LICENSE-binary and write it under Apache License will not be wrong. Thanks.

Copy link
Contributor Author

@slfan1989 slfan1989 Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Hexiaoqiao @steveloughran In the latest commit, I have rolled back the Jenkinsfile commit, which may cause the next build to not complete successfully, as the current CI cannot compile code with more than 10,000 lines. However, this is expected, so please don't worry.

org.glassfish.jersey.core:jersey-server:2.46
org.glassfish.jersey.inject:jersey-hk2:2.46
org.glassfish.jersey.core:jersey-client:2.46
org.glassfish.jersey.test-framework:jersey-test-framework-core:2.46
org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-grizzly2:2.46
org.glassfish.jersey.containers:jersey-container-servlet:2.46
org.glassfish.jersey.containers:jersey-container-servlet-core:2.46
org.glassfish.jersey.media:jersey-media-json-jettison:2.46
org.glassfish.jersey.media:jersey-media-jaxb:2.46
com.sun.xml.bind:jaxb-impl:2.2.3-1
javax.annotation:javax.annotation-api:1.3.2
javax.cache:cache-api:1.1.1
Expand All @@ -512,7 +516,11 @@ javax.servlet.jsp:jsp-api:2.1
javax.websocket:javax.websocket-api:1.0
javax.ws.rs:jsr311-api:1.1.1
javax.xml.bind:jaxb-api:2.2.11

jakarta.ws.rs-api:jakarta.ws.rs-api:2.1.6
jakarta.servlet.jsp:jakarta.servlet.jsp-api:2.3.6
jakarta.servlet:jakarta.servlet-api:4.0.4
jakarta.xml.bind:jakarta.xml.bind-api:2.3.3
net.jodah:failsafe:2.4.4

Eclipse Public License 1.0
--------------------------
Expand Down
4 changes: 2 additions & 2 deletions dev-support/Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ pipeline {
environment {
YETUS='yetus'
// Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
YETUS_VERSION='rel/0.14.0'
YETUS_VERSION='a7d29a6a72750a0c5c39512f33945e773e69303e'
}

parameters {
Expand All @@ -71,7 +71,7 @@ pipeline {
checkout([
$class: 'GitSCM',
branches: [[name: "${env.YETUS_VERSION}"]],
userRemoteConfigs: [[ url: 'https://github.com/apache/yetus.git']]]
userRemoteConfigs: [[ url: 'https://github.com/ayushtkn/yetus.git']]]
Copy link
Contributor

@Hexiaoqiao Hexiaoqiao Jan 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is better to link Hadoop repo rather than Ayushtkn's personal repo.

Copy link
Contributor Author

@slfan1989 slfan1989 Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Hexiaoqiao Thank you very much for your message! I have carefully checked the agreement for the new package we introduced, as follows:

Apache License Version 2.0

org.glassfish.jersey.core:jersey-common:2.46
org.glassfish.jersey.core:jersey-server:2.46
org.glassfish.jersey.inject:jersey-hk2:2.46
org.glassfish.jersey.core:jersey-client:2.46
org.glassfish.jersey.test-framework:jersey-test-framework-core:2.46
org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-grizzly2:2.46
org.glassfish.jersey.containers:jersey-container-servlet:2.46
org.glassfish.jersey.containers:jersey-container-servlet-core:2.46
org.glassfish.jersey.media:jersey-media-json-jettison:2.46
org.glassfish.jersey.media:jersey-media-jaxb:2.46
net.jodah:failsafe:2.4.4

Eclipse Distribution License (EDL) 1.0

jakarta.xml.bind:jakarta.xml.bind-api:2.3.3

Eclipse Public License (EPL) 2.0

jakarta.ws.rs-api:jakarta.ws.rs-api:2.1.6
jakarta.servlet.jsp:jakarta.servlet.jsp-api:2.3.6
jakarta.servlet:jakarta.servlet-api:4.0.4

cc: @pjfanning

)
}
}
Expand Down
2 changes: 2 additions & 0 deletions hadoop-client-modules/hadoop-client-check-invariants/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@
<exclude>ch.qos.reload4j:reload4j</exclude>
<!-- Leave javax annotations we need exposed -->
<exclude>com.google.code.findbugs:jsr305</exclude>
<exclude>jakarta.annotation:jakarta.annotation-api</exclude>
<exclude>org.glassfish.jersey:*</exclude>
<!-- Leave bouncycastle unshaded because it's signed with a special Oracle certificate so it can be a custom JCE security provider -->
<exclude>org.bouncycastle:*</exclude>
<!-- Leave snappy that includes native methods which cannot be relocated. -->
Expand Down
6 changes: 5 additions & 1 deletion ...es/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,11 @@ allowed_expr+="|^krb5_udp-template.conf$"
allowed_expr+="|^jetty-dir.css$"
# Snappy java is native library. We cannot relocate it to under org/apache/hadoop.
allowed_expr+="|^org/xerial/"

# Comes from jersey, not sure if relocatable.
allowed_expr+="|^jersey"
# Comes from jakarta, not sure if relocatable.
allowed_expr+="|^javax"
allowed_expr+="|^javassist"
allowed_expr+=")"
declare -i bad_artifacts=0
declare -a bad_contents
Expand Down
1 change: 1 addition & 0 deletions hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@
<!-- Leave snappy that includes native methods which cannot be relocated. -->
<exclude>org.xerial.snappy:*</exclude>
<exclude>org.ehcache:*</exclude>
<exclude>org.glassfish.jersey:*</exclude>
</excludes>
</banTransitiveDependencies>
<banDuplicateClasses>
Expand Down
10 changes: 2 additions & 8 deletions hadoop-client-modules/hadoop-client-integration-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,14 +86,8 @@
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.activation</groupId>
<artifactId>activation</artifactId>
<version>1.1.1</version>
<groupId>jakarta.xml.bind</groupId>
<artifactId>jakarta.xml.bind-api</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
Expand Down
106 changes: 19 additions & 87 deletions hadoop-client-modules/hadoop-client-minicluster/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,10 @@
<groupId>org.ow2.asm</groupId>
<artifactId>asm-commons</artifactId>
</exclusion>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm-commons</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- Add optional runtime dependency on the in-development timeline server module
Expand Down Expand Up @@ -415,72 +419,20 @@
<!-- Skip commons-logging:commons-logging-api because it looks like nothing actually included it -->
<!-- Skip jetty-util because it's in client -->
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-common</artifactId>
<optional>true</optional>
<exclusions>
<exclusion>
<groupId>javax.ws.rs</groupId>
<artifactId>jsr311-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-client</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.github.pjfanning</groupId>
<artifactId>jersey-json</artifactId>
<optional>true</optional>
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>jackson-jaxrs-json-provider</artifactId>
</exclusion>
<exclusion>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jettison</groupId>
<artifactId>jettison</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-server</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-servlet</artifactId>
<optional>true</optional>
<exclusions>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.enterprise</groupId>
<artifactId>cdi-api</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.cal10n</groupId>
<artifactId>cal10n-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- skip org.apache.avro:avro-ipc because it doesn't look like hadoop-common actually uses it -->
<dependency>
<groupId>net.sf.kosmosfs</groupId>
Expand Down Expand Up @@ -624,23 +576,16 @@
</exclusions>
</dependency>
<dependency>
<groupId>com.sun.jersey.jersey-test-framework</groupId>
<artifactId>jersey-test-framework-grizzly2</artifactId>
<groupId>org.glassfish.jersey.test-framework</groupId>
<artifactId>jersey-test-framework-core</artifactId>
<optional>true</optional>
<exclusions>
<!-- excluding because client already has the tomcat version -->
<exclusion>
<groupId>org.glassfish</groupId>
<artifactId>javax.servlet</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- skipping jersey-server because it's above -->
<dependency>
<groupId>com.sun.jersey.contribs</groupId>
<artifactId>jersey-guice</artifactId>
<groupId>org.glassfish.jersey.test-framework.providers</groupId>
<artifactId>jersey-test-framework-provider-grizzly2</artifactId>
<optional>true</optional>
</dependency>
<!-- skipping jersey-server because it's above -->
<!-- skipping guice-servlet because it's above -->
<!-- skipping avro because it is in client via hadoop-common -->
<!-- skipping jersey-core because it's above -->
Expand Down Expand Up @@ -709,6 +654,10 @@
<exclude>org.bouncycastle:*</exclude>
<!-- Leave snappy that includes native methods which cannot be relocated. -->
<exclude>org.xerial.snappy:*</exclude>
<exclude>org.glassfish.jersey:*</exclude>
<exclude>org.hamcrest:*</exclude>
<exclude>aopalliance:*</exclude>
<exclude>javassist:*</exclude>
</excludes>
</artifactSet>
<filters>
Expand All @@ -729,24 +678,6 @@
<!-- Since runtime has classes for these jars, we exclude them.
We still want the java services api files, since those were excluded in runtime
-->
<filter>
<artifact>com.sun.jersey:jersey-client</artifact>
<excludes>
<exclude>**/*.class</exclude>
</excludes>
</filter>
<filter>
<artifact>com.sun.jersey:jersey-core</artifact>
<excludes>
<exclude>**/*.class</exclude>
</excludes>
</filter>
<filter>
<artifact>com.sun.jersey:jersey-servlet</artifact>
<excludes>
<exclude>**/*.class</exclude>
</excludes>
</filter>
<filter>
<artifact>org.apache.hadoop:hadoop-mapreduce-client-jobclient:*</artifact>
<excludes>
Expand All @@ -760,6 +691,7 @@
<excludes>
<exclude>META-INF/versions/9/module-info.class</exclude>
<exclude>META-INF/versions/11/module-info.class</exclude>
<exclude>META-INF/versions/9/javax/xml/bind/ModuleUtil.class</exclude>
</excludes>
</filter>

Expand Down
26 changes: 19 additions & 7 deletions hadoop-client-modules/hadoop-client-runtime/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,24 @@
<exclude>org.xerial.snappy:*</exclude>
<!-- leave out kotlin classes -->
<exclude>org.jetbrains.kotlin:*</exclude>
Copy link
Contributor

@steveloughran steveloughran Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do these no longer need to be cut?

Copy link
Contributor Author

@slfan1989 slfan1989 Jan 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have completed the fixes in the latest version.

<exclude>org.glassfish.jersey.test-framework:*</exclude>
<exclude>org.glassfish.jersey.media:*</exclude>
<exclude>org.glassfish.jersey.containers:*</exclude>
<exclude>org.glassfish.jersey.test-framework.providers:*</exclude>
<exclude>org.glassfish.hk2:*</exclude>
<exclude>org.glassfish.jersey.inject:*</exclude>
<exclude>org.glassfish.grizzly:*</exclude>
<exclude>org.glassfish.jersey.core:*</exclude>
<exclude>org.glassfish.hk2.external:*</exclude>
<exclude>org.glassfish.jaxb:*</exclude>
<exclude>jakarta.ws.rs:*</exclude>
<exclude>jakarta.annotation:*</exclude>
<exclude>jakarta.validation:*</exclude>
<exclude>jakarta.servlet:*</exclude>
<exclude>javax.annotation:*</exclude>
<exclude>org.hamcrest:*</exclude>
<exclude>aopalliance:*</exclude>
<exclude>javassist:*</exclude>
</excludes>
</artifactSet>
<filters>
Expand All @@ -184,13 +202,6 @@
<exclude>org/apache/jasper/compiler/Localizer.class</exclude>
</excludes>
</filter>
<!-- We rely on jersey for our web interfaces. We want to use its java services stuff only internal to jersey -->
<filter>
<artifact>com.sun.jersey:*</artifact>
<excludes>
<exclude>META-INF/services/javax.*</exclude>
</excludes>
</filter>
<filter>
<!-- skip french localization -->
<artifact>org.apache.commons:commons-math3</artifact>
Expand Down Expand Up @@ -247,6 +258,7 @@
<exclude>META-INF/versions/9/module-info.class</exclude>
<exclude>META-INF/versions/11/module-info.class</exclude>
<exclude>META-INF/versions/18/module-info.class</exclude>
<exclude>META-INF/versions/9/javax/xml/bind/ModuleUtil.class</exclude>
</excludes>
</filter>

Expand Down
Loading
Loading