New mirror-target-platform goal and source artifacts download issue

[mdaloia]

Hi,

I'm trying to use the new mirror-target-platform goal added in 4.0.5 to mirror a custom Target Platform definition in a .target file (as using the goal mirror with a target platform seems to not be working as expected. See this discussion #1554)

To start, the only way I found to make this goal work without having to create a feature is to set eclipse-target-definition as <packaging> in pom.xml

Having said that, the issue I encounter now is that it doesn't download the source of the corresponding units (unless the source unit is included explicitly by some feature). I have set includeSource="true" in the <location> node of the .target file and I also added <targetDefinitionIncludeSource>force</targetDefinitionIncludeSource> to the target platform configuration.

When it starts the execution I see it says include source mode=ignore (see below) even when I have set it as force.

[INFO] Scanning for projects...
[INFO] Tycho Version:  4.0.8 (d07e53b9e5b0b4c40ee529fb831732f631e4e3ef)
[INFO] Tycho Mode:     project
[INFO] Tycho Builder:  maven
[INFO] Build Threads:  1
[INFO] Using tycho.disableP2Mirrors to disable P2 mirrors is deprecated, use the property eclipse.p2.mirrors instead, see https://tycho.eclipseprojects.io/doc/4.0.8/SystemProperties.html for details.
[INFO]
[INFO] ----------------------< test.tycho:mirror-target >----------------------
[INFO] Building mirror-target 1.0.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] ---------------------[ eclipse-target-definition ]----------------------
[INFO] Resolving target definition file:/Users/mdaloia/tycho-issue-report/custom.target for environments=[macosx/cocoa/x86_64], include source mode=ignore, referenced repository mode =ignore, execution environment=StandardEEResolutionHints [executionEnvironment=OSGi profile 'JavaSE-17' { source level: 17, target level: 17}], remote p2 repository options=org.eclipse.tycho.p2maven.DefaultProvisioningAgent@19d13d04
[INFO] ### Using TychoRepositoryTransport for remote P2 access ###
[INFO]     Cache location:         /Users/mdaloia/.m2/repository/.cache/tycho
[INFO]     Transport mode:         online
[INFO]     Http Transport type:    Java11Client
[INFO]     Update mode:            cache first
[INFO]     Minimum cache duration: 60 minutes
[INFO]       (you can configure this with -Dtycho.p2.transport.min-cache-minutes=<desired minimum cache duration>)
[INFO] Downloading from p2: https://download.eclipse.org/releases/2024-03/202403131000/p2.index
[INFO] Downloaded from p2: https://download.eclipse.org/releases/2024-03/202403131000/p2.index (140 bytes at 0 bytes/s)
[INFO]
[INFO] --- clean:3.2.0:clean (default-clean) @ mirror-target ---
[INFO] Deleting /Users/mdaloia/tycho-issue-report/target
[INFO]
[INFO] --- target-platform-configuration:4.0.8:mirror-target-platform (mirror-target) @ mirror-target ---
[INFO] Mirroring 3 unit(s) from the target platform, this can take a while ...
[INFO] Downloading from p2: https://download.eclipse.org/releases/2024-03/202403131000/p2.index
[INFO] Downloaded from p2: https://download.eclipse.org/releases/2024-03/202403131000/p2.index (140 bytes at 9 KB/s)
[INFO]
[INFO] --- tycho-packaging:4.0.8:package-target-definition (default-package-target-definition) @ mirror-target ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16.650 s
[INFO] Finished at: 2024-06-07T10:53:15-03:00
[INFO] ------------------------------------------------------------------------

I debugged it a little bit and found this:
https://github.com/eclipse-tycho/tycho/blob/tycho-4.0.8/tycho-core/src/main/java/org/eclipse/tycho/p2resolver/P2DependencyResolver.java#L230-L231

Find a reproducer example at the end.

Question

Is it possible to have a minimal configuration in order to mirror a target platform?

Example

pom.xml

<?xml version="1.0" encoding="UTF-8" ?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<modelVersion>4.0.0</modelVersion>

	<groupId>test.tycho</groupId>
	<artifactId>mirror-target</artifactId>
	<version>1.0.0-SNAPSHOT</version>
	<packaging>eclipse-target-definition</packaging>

	<properties>
		<tycho-version>4.0.8</tycho-version>
	</properties>

	<build>
		<plugins>
			<plugin>
				<groupId>org.eclipse.tycho</groupId>
				<artifactId>tycho-maven-plugin</artifactId>
				<version>${tycho-version}</version>
				<extensions>true</extensions>
			</plugin>

			<plugin>
				<groupId>org.eclipse.tycho</groupId>
				<artifactId>target-platform-configuration</artifactId>
				<version>${tycho-version}</version>
				<executions>
					<execution>
						<id>mirror-target</id>
						<phase>compile</phase>
						<goals>
							<goal>mirror-target-platform</goal>
						</goals>
					</execution>
				</executions>
				<configuration>
					<target>
						<file>custom.target</file>
					</target>
					<targetDefinitionIncludeSource>force</targetDefinitionIncludeSource>
				</configuration>
			</plugin>
		</plugins>
	</build>
</project>

custom.target

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?pde version="3.8"?>

<target name="test">

	<locations>
		<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
			<unit id="org.apache.commons.lang3" version="3.14.0"/>

			<repository location="https://download.eclipse.org/releases/2024-03/202403131000/"/>
		</location>
	</locations>

</target>

[laeubi]

The mojo is quite new (if not to say: experimental :-) ) so its possibly room for improvements.

i think as a first step you should use Tycho 4.0.8 (what it the latest release), then we can see how to enhance this, in general sources are always handled a little bit different.

[mdaloia]

@laeubi Thanks for the quick reply... I'm already using 4.0.8, as you can see on the supplied execution log and also the pom.xml file.

Does it sounds like a bug? I can create an issue if so.

[laeubi]

Upps.. I was confused because you mentioned Tycho 4.0.5 earlier. I would not call it a bug but maybe a missing feature :-)

Looking at the code it seems reasonable to at least use IncludeSourceMode.consider I can't really remember why I initially set it to ignore, so if you like you can propose a patch (maybe including a testcase to).

[hieplq]

hi @laeubi
To include the source artifact in the mirrored target repository, I conducted a test on this GitHub repository: https://github.com/hieplq/vn.hieplq.tycho.target.mirror.sourceissue

you can follow it by run

git clone git@github.com:hieplq/vn.hieplq.tycho.target.mirror.sourceissue.git
cd vn.hieplq.tycho.target.mirror.sourceissue/com.codeandme.tycho.releng
mvn verify
ls -1 ../com.codeandme.tycho.releng.targetplatform/target/target-platform-repository/plugins

ouput (Some bundle include the source bundle while others do not)

org.apache.commons.lang_2.6.0.jar
org.atmosphere.runtime_2.7.2.v202109010034.jar
org.atmosphere.runtime.source_2.7.2.v202109010034.jar
org.eclipse.jetty.server_12.0.14.jar
org.eclipse.jetty.server.source_12.0.14.jar
org.eclipse.text_3.13.100.v20230801-1334.jar
org.opentest4j_1.3.0.jar
org.opentest4j.source_1.3.0.jar
wrapped.org.apache.lucene.lucene-core_8.4.1.jar

this's my conclude:

1. artifact on p2 repository created using org.eclipse.equinox.p2.publisher.FeaturesAndBundlesPublisher are automatically included source in the mirror
2. artifact on every p2 can explicitly include source by use unit id="org.opentest4j.source" version="1.3.0"/>
3. artifact that source has OSGi Bundle Manifest Headers on maven can explicitly include by use <classifier>sources</classifier>
4. source artifact without OSGi Bundle Manifest Headers on maven can't add to mirror