HDFS-16665. Fix duplicate sources for HDFS test #4573

GauthamBanasandra · 2022-07-17T16:50:03Z

Description of PR

The library target hdfspp_test_shim_static is built using the following sources, which causes duplicate symbols to be defined -

hdfs_shim.c

${LIBHDFSPP_BINDING_C}/hdfs.cc

hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/CMakeLists.txt

Line 153 in 8774f17

    
           add_library(hdfspp_test_shim_static STATIC $<TARGET_OBJECTS:x_platform_obj> hdfs_shim.c libhdfs_wrapper.c libhdfspp_wrapper.cc ${LIBHDFSPP_BINDING_C}/hdfs.cc)

And fails the compilation -

H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj) : error LNK2005: hdfsAllowSnapshot already defined in hdfspp_test_shim_static.lib(hdfs.obj) 
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj) : error LNK2005: hdfsAvailable already defined in hdfspp_test_shim_static.lib(hdfs.obj) 
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj) : error LNK2005: hdfsBuilderConfSetStr already defined in hdfspp_test_shim_static.lib(hdfs.obj) 
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj) : error LNK2005: hdfsBuilderConnect already defined in hdfspp_test_shim_static.lib(hdfs.obj)
Duplicate symbols defined by hdfs_shim.c - https://github.com/apache/hadoop/blob/440f4c2b28515d2007b81ac00b549bbf14fa9f64/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c#L583-L585.

Adding the source ${LIBHDFSPP_BINDING_C}/hdfs.cc is redundant here since this file is transitively included in hdfs_shim.c through libhdfspp_wrapper.h -

hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c

Line 20 in 440f4c2

#include "libhdfspp_wrapper.h"

Thus, we need to exclude ${LIBHDFSPP_BINDING_C}/hdfs.cc to fix this issue.

How was this patch tested?

Tested this locally by building on my Windows system.
Hadoop Jenkins CI validation.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

* The library target hdfspp_test_shim_static is built using the following sources, which causes duplicate symbols to be defined - - hdfs_shim.c - ${LIBHDFSPP_BINDING_C}/hdfs.cc * ${LIBHDFSPP_BINDING_C}/hdfs.cc is redundant and removing this fixes the issue.

hadoop-yetus · 2022-07-17T20:54:11Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	40m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 17s		trunk passed
+1 💚	compile	4m 2s		trunk passed
+1 💚	mvnsite	0m 49s		trunk passed
+1 💚	shadedclient	62m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	3m 44s		the patch passed
+1 💚	cc	3m 44s		the patch passed
+1 💚	javac	3m 44s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	mvnsite	0m 28s		the patch passed
+1 💚	shadedclient	19m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	112m 27s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 54s		The patch does not generate ASF License warnings.
		242m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/artifact/out/Dockerfile
GITHUB PR	#4573
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell detsecrets
uname	Linux 845690b8b35a 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cd9094d`
Default Java	Red Hat, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/testReport/
Max. process+thread count	555 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-18T00:15:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	24m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	22m 26s		trunk passed
+1 💚	compile	4m 16s		trunk passed
+1 💚	mvnsite	1m 3s		trunk passed
+1 💚	shadedclient	47m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s		the patch passed
+1 💚	compile	3m 37s		the patch passed
+1 💚	cc	3m 37s		the patch passed
+1 💚	javac	3m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 36s		the patch passed
+1 💚	shadedclient	19m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	101m 54s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		201m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/artifact/out/Dockerfile
GITHUB PR	#4573
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell detsecrets
uname	Linux f836d92bb685 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cd9094d`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/testReport/
Max. process+thread count	701 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-18T03:28:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	25m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 58s		trunk passed
+1 💚	compile	3m 31s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	shadedclient	55m 7s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 22s		the patch passed
+1 💚	compile	3m 14s		the patch passed
+1 💚	cc	3m 14s		the patch passed
+1 💚	javac	3m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 26s		the patch passed
+1 💚	shadedclient	25m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	78m 21s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 48s		The patch does not generate ASF License warnings.
		191m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/artifact/out/Dockerfile
GITHUB PR	#4573
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell detsecrets
uname	Linux 2b3ab75d5860 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cd9094d`
Default Java	Debian-11.0.15+10-post-Debian-1deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/testReport/
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-18T06:37:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	21m 27s		trunk passed
+1 💚	compile	4m 4s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	4m 6s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	shadedclient	50m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	3m 43s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	cc	3m 43s		the patch passed
+1 💚	javac	3m 43s		the patch passed
+1 💚	compile	3m 45s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	cc	3m 45s		the patch passed
+1 💚	javac	3m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 27s		the patch passed
+1 💚	shadedclient	19m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	106m 28s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		188m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/artifact/out/Dockerfile
GITHUB PR	#4573
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell detsecrets
uname	Linux 6e018b4716e2 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cd9094d`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/testReport/
Max. process+thread count	619 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

* The library target hdfspp_test_shim_static is built using the following sources, which causes duplicate symbols to be defined - - hdfs_shim.c - ${LIBHDFSPP_BINDING_C}/hdfs.cc * ${LIBHDFSPP_BINDING_C}/hdfs.cc is redundant and removing this fixes the issue.

GauthamBanasandra self-assigned this Jul 17, 2022

GauthamBanasandra added the HDFS label Jul 17, 2022

GauthamBanasandra marked this pull request as ready for review July 18, 2022 09:19

GauthamBanasandra requested a review from goiri July 18, 2022 09:19

goiri approved these changes Jul 18, 2022

View reviewed changes

GauthamBanasandra merged commit 6415eb0 into apache:trunk Jul 19, 2022

GauthamBanasandra deleted the fix-dups branch July 19, 2022 16:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16665. Fix duplicate sources for HDFS test #4573

HDFS-16665. Fix duplicate sources for HDFS test #4573

Uh oh!

GauthamBanasandra commented Jul 17, 2022

Uh oh!

hadoop-yetus commented Jul 17, 2022

Uh oh!

hadoop-yetus commented Jul 18, 2022

Uh oh!

hadoop-yetus commented Jul 18, 2022

Uh oh!

hadoop-yetus commented Jul 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDFS-16665. Fix duplicate sources for HDFS test #4573

HDFS-16665. Fix duplicate sources for HDFS test #4573

Uh oh!

Conversation

GauthamBanasandra commented Jul 17, 2022

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Jul 17, 2022

Uh oh!

hadoop-yetus commented Jul 18, 2022

Uh oh!

hadoop-yetus commented Jul 18, 2022

Uh oh!

hadoop-yetus commented Jul 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants