Feature Request: Fine-Grained Content Filtering for Dependencies

[shunping]

What feature do you want to see added?

We are currently facing an issue where our project (JAR A) depends on a shaded JAR (JAR B). This shaded JAR B includes an older, vulnerable version of a library (Package C), including its META-INF files.

Our project A also brings in a newer, non-vulnerable version of Package C as a transitive dependency. However, during the final shading process for JAR A, the older META-INF files from Package C, which are bundled inside the shaded JAR B, are being included. This is causing our vulnerability scanner to flag the final artifact as having a known vulnerability, even though the newer classes of Package C are correctly used at runtime.

We need a mechanism to precisely exclude the problematic META-INF directory of the old Package C that resides within the shaded JAR B.

---

We would like the Gradle Shadow plugin to implement a feature for fine-grained content filtering on a per-dependency basis. This functionality is already available in the Apache Maven Shade Plugin (https://maven.apache.org/plugins/maven-shade-plugin/examples/includes-excludes.html) and is extremely powerful for resolving these types of conflicts.

Here is an example from their documentation of how they achieve this:

<filters>
  <filter>
    <artifact>junit:junit</artifact>
    <includes>
      <include>junit/framework/**</include>
    </includes>
    <excludes>
      <exclude>org/junit/runners/**</exclude>
    </excludes>
  </filter>
  <filter>
    <artifact>*:*</artifact>
    <excludes>
      <exclude>META-INF/*.SF</exclude>
      <exclude>META-INF/*.DSA</exclude>
      <exclude>META-INF/*.RSA</exclude>
    </excludes>
  </filter>
</filters>

For our specific use case in gradle, it might look something like this:

// Hypothetical DSL
shadowJar {
  dependencies {
    // ... other configurations
    include(dependency('group:shaded-jar-B')) {
      exclude('META-INF/maven/vulnerable/package-c/**')
    }
}

This would instruct the Shadow plugin to remove all files matching the exclude pattern, but only from the contents of the shaded-jar-B dependency before merging them into the final uber JAR.

Thanks for your consideration!

Upstream changes

n/a

Are you interested in contributing this feature?

Yes if the proposal is sound.

[Goooler]

If I understand correctly, you want to preserve/include the specific resources from the project but from the dependencies?

[shunping]

That's correct. Basically we need filters (include/exclude) that can be applied to the content/resource within the dependency.

[Goooler]

There is a built-in resource transformer

shadow/src/main/kotlin/com/github/jengelman/gradle/plugins/shadow/transformers/PreserveFirstFoundResourceTransformer.kt

Line 24 in 400c601

public open class PreserveFirstFoundResourceTransformer @Inject constructor(

You can use it with setting duplicatesStrategy = DuplicatesStrategy.INCLUDE.

Or try out filesMatching to override the strategy to EXCLUDE for specific resources.

See https://gradleup.com/shadow/configuration/merging/#handling-duplicates-strategy

[shunping]

Thanks for the suggestion @Goooler. I tried setting duplicatesStrategy as follows but it didn't work for me. Please correct me if my approach is not right.

Here is the code I used to reproduce.

== build.gradle ==

plugins {
    id 'java'
    id 'application'
    id 'com.github.johnrengelman.shadow' version '7.1.2'
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.apache.hbase:hbase-shaded-client:2.5.3-hadoop3'
    implementation 'org.apache.hadoop:hadoop-common:3.4.1'
    testImplementation 'junit:junit:4.13.2'
}

application {
    mainClassName = 'com.example.HelloWorld'
}

shadowJar {
    duplicatesStrategy = DuplicatesStrategy.INCLUDE
}

== HellowWorld.java ==

package com.example;

import org.apache.hadoop.util.VersionInfo;
import org.apache.commons.compress.compressors.CompressorStreamFactory;

public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hadoop version: " + VersionInfo.getVersion());
        Package p = CompressorStreamFactory.class.getPackage();
        System.out.println("commons-compress version: " + p.getImplementationVersion());
    }
}

Running the above code with gradle run, the output will be something like the following. This suggests that the runtime commons-compress version is 1.26.1.

Hadoop version: 3.2.4
commons-compress version: 1.26.1

However, if we create the shadow jar with gradle shadowjar, unzip it and check inside, we can see the commons-compress version from in META-INF/maven/org.apache.commons/commons-compress/pom.properties is 1.21.1. This old pom.properties file comes from the shaded jar dependency org.apache.hbase:hbase-shaded-client:2.5.3-hadoop3, rather than the transitive dependency.

I also tried changing duplicatesStrategy to other values like EXCLUDE, FAIL, but none of them changes the outcome.

[Goooler]

https://github.com/GradleUp/shadow?tab=readme-ov-file#shadow

[shunping]

Thanks for the guidance, @Goooler! It seems this functionality was indeed added in a more recent version.

I've confirmed that the solution works. The key was to use filesMatching or eachFile to apply a duplicatesStrategy to the specific conflicting file path, which effectively filters the contents from the problematic shaded JAR.

For the benefit of anyone who finds this issue in the future, here is the final, working build.gradle file:

plugins {
    id 'java'
    id 'application'
    id 'com.gradleup.shadow' version '9.0.2'
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.apache.hbase:hbase-shaded-client:2.5.3-hadoop3'
    implementation 'org.apache.commons:commons-compress:1.26.1'
    testImplementation 'junit:junit:4.13.2'
}

application {
    mainClass.set("com.example.HelloWorld")
}

shadowJar {
  // Use eachFile or filesMatching with duplicatesStrategy to exclude
  // "META-INF/maven/org.apache.commons/commons-compress/pom.properties" from
  // the shaded jar org.apache.hbase:hbase-shaded-client:2.5.3-hadoop3.
  //
  // eachFile {
  //  duplicatesStrategy = DuplicatesStrategy.INCLUDE
  // }
  filesMatching('META-INF/maven/org.apache.commons/commons-compress/pom.properties') {
    duplicatesStrategy = DuplicatesStrategy.INCLUDE
  }
  manifest {
    attributes "Main-Class": application.mainClass.get()
  }
}

I think it would be useful to add some simple examples of using filesMatching or eachFile in https://gradleup.com/shadow/configuration/merging/ too.

---

One last question: if we have multiple dependent jars that have the same resource but different versions, how can we tell or control which one is first and which one is last so that we can use either INCLUDE or EXCLUDE duplicateStrategy to include the correct version into the final shaded jar? @Goooler

[Goooler]

Cannot guarantee the orders. Needs

• Add original artifact name to TransformerContext #546
• Add original artifact name to FileTreeElement #913

[shunping]

Looks like if gradle/gradle#34493 is accepted and implemented, the original feature request here on per-dependency filtering could be implemented.

[Goooler]

Not yet. I'll track the updates from there.