[Fix][FTP]Fix 'file_filterpattern' cannot filter directories #10049

LiJie20190102 · 2025-11-09T11:58:35Z

Purpose of this pull request

fix #10012. I previously submitted a PR, which is #10014 , but I didn't handle it properly, so I resubmitted it

Does this PR introduce any user-facing change?

How was this patch tested?

If the path is /data/setunnel, and the file structure example is:

/data/seatunnel/20241001/report.txt
/data/seatunnel/20241007/abch202410.csv
/data/seatunnel/20241002/abcg202410.csv
/data/seatunnel/20241005/old_data.csv
/data/seatunnel/20241012/logo.png

If you only want to filter based on file names, simply write the regular file names; If you want to filter based on the file directory at the same time, the expression needs to start with path.

Example 1: Match all .txt files，Regular Expression:

.*.txt

The result of this example matching is:

/data/seatunnel/20241001/report.txt

Example 2: Match third level folders starting with 202410 and files ending with .csv, the Regular Expression:

/data/seatunnel/202410\d*/.*.csv

The result of this example matching is:

/data/seatunnel/20241007/abch202410.csv
/data/seatunnel/20241002/abcg202410.csv
/data/seatunnel/20241005/old_data.csv

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

LiJie20190102 · 2025-11-09T12:02:54Z

@davidzollo Could you please help me review the code again

davidzollo

+1 if CI passed

zhangshenghang · 2025-11-11T01:57:39Z

docs/en/connector-v2/source/CosFile.md

 There are some examples.

-File Structure Example:
+If the `path` is `/data/setunnel`, and the file structure example is:


Suggested change

If the `path` is `/data/setunnel`, and the file structure example is:

If the `path` is `/data/seatunnel`, and the file structure example is:

zhangshenghang · 2025-11-11T01:57:59Z

docs/en/connector-v2/source/FtpFile.md

 There are some examples.

-File Structure Example:
+If the `path` is `/data/setunnel`, and the file structure example is:


zhangshenghang · 2025-11-11T01:58:06Z

docs/en/connector-v2/source/HdfsFile.md

 There are some examples.

-File Structure Example:
+If the `path` is `/data/setunnel`, and the file structure example is:


zhangshenghang · 2025-11-11T02:04:33Z

.../java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/AbstractReadStrategy.java

+            if (pattern.pattern().startsWith(path)) {
+                // filter based on the file directory at the same time
+                return pattern.matcher(fileStatus.getPath().getName()).matches();
+            }
+            // filter based on file names
+            return pattern.matcher(fileStatus.getPath().toUri().getPath()).matches();


Is the logic reversed?

pattern.pattern().startsWith(path) = true ，Should we filter directories or files?

Thank you for your review. I have made improvements and the fileStatus in this method will only be files, not directories

zhangshenghang · 2025-11-11T02:08:36Z

thanks @LiJie20190102
It is also necessary to ensure that CI can run normally and pass.

LiJie20190102 · 2025-11-11T06:15:35Z

thanks @LiJie20190102 It is also necessary to ensure that CI can run normally and pass.

Can you help me check why it failed? I am referring to 'org.apache.seatunnel.connectors.seatunnel.file.writer.XmlReadStrategyTest', which can pass

dybyte · 2025-11-11T07:55:07Z

...c/test/java/org/apache/seatunnel/connectors/seatunnel/file/reader/FileFilterPatternTest.java

+    @Test
+    public void testJsonFilterPatternWithFilePath() throws URISyntaxException, IOException {


Can you help me check why it failed? I am referring to 'org.apache.seatunnel.connectors.seatunnel.file.writer.XmlReadStrategyTest', which can pass

What do you think about using the annotation @DisabledOnOs(OS.WINDOWS) here? cc @davidzollo @zhangshenghang

Can you help me check why it failed? I am referring to 'org.apache.seatunnel.connectors.seatunnel.file.writer.XmlReadStrategyTest', which can pass

What do you think about using the annotation @DisabledOnOs(OS.WINDOWS) here? cc @davidzollo @zhangshenghang

@dybyte I agree with your idea, @LiJie20190102 but can you provide a screenshot of the successful run under Windows ?

@davidzollo @zhangshenghang If I provide a locally successful snapshot, does it not require adding e2e?

dybyte · 2025-11-11T07:55:26Z

...c/test/java/org/apache/seatunnel/connectors/seatunnel/file/reader/FileFilterPatternTest.java

+    @Test
+    public void testJsonFilterPatternWithFileName() throws URISyntaxException, IOException {


davidzollo · 2025-11-11T13:41:06Z

Considering that this PR affects lots of file connectors. Please add related e2e test, thx

LiJie20190102 · 2025-11-15T06:37:51Z

Considering that this PR affects lots of file connectors. Please add related e2e test, thx

e2e has already been added. @davidzollo @zhangshenghang

zhangshenghang · 2025-11-17T13:02:20Z

.../java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/AbstractReadStrategy.java

                // skip hidden tmp directory, such as .hive-staging_hive
                if (!fileStatus.getPath().getName().startsWith(".")) {
-                    fileNames.addAll(getFileNamesByPath(fileStatus.getPath().toString()));
+                    fileNames.addAll(getFileNamesByPath(fileStatus.getPath().toUri().getPath()));


Will there be any problem with this change? If the file path is this: s3a://my-bucket/data/2025/11/report.csv, is there a problem with it?

configuration file：

Debugging results：

I think the logic is correct, what do you think?

configuration file： Debugging results： I think the logic is correct, what do you think?

What was the previous result like? Before the modification, s3a:/ was not removed; after the modification, it is removed. Will there be any other impacts?

Sorry, after careful consideration, this is the area where I adapted the code previously modified, and this code can be left unchanged

above

corgy-w · 2025-11-17T13:14:36Z

...connector-file-s3-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/s3/S3FileIT.java

                Assertions.assertEquals(0, extraCommands.getExitCode());
            };

+    @TestTemplate


The current S3 tests are disabled. Could you please modify the changes to enable the S3 tests as well? It should be similar to how MinIO supports the S3 protocol.

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

a5822ec

github-actions bot added document connectors-v2 file labels Nov 9, 2025

lijie added 4 commits November 10, 2025 09:04

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

bbdd493

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

46a2787

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

2e352bc

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

f64eede

davidzollo previously approved these changes Nov 10, 2025

View reviewed changes

github-actions bot added approved reviewed labels Nov 10, 2025

zhangshenghang reviewed Nov 11, 2025

View reviewed changes

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

f1aff6e

LiJie20190102 dismissed davidzollo’s stale review via f1aff6e November 11, 2025 04:29

github-actions bot removed approved reviewed labels Nov 11, 2025

dybyte reviewed Nov 11, 2025

View reviewed changes

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

0bd638d

github-actions bot added the e2e label Nov 12, 2025

lijie added 4 commits November 13, 2025 09:16

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

b70136c

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

768c0be

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

75c81f2

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

3028d99

lijie added 4 commits November 14, 2025 15:36

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

6d6995a

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

15e15e2

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

9928688

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

a589d60

corgy-w previously approved these changes Nov 17, 2025

View reviewed changes

github-actions bot added approved reviewed labels Nov 17, 2025

corgy-w requested a review from zhangshenghang November 17, 2025 12:43

zhangshenghang reviewed Nov 17, 2025

View reviewed changes

github-actions bot removed approved reviewed labels Nov 17, 2025

corgy-w reviewed Nov 17, 2025

View reviewed changes

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories

d96a651

	If the `path` is `/data/setunnel`, and the file structure example is:
	If the `path` is `/data/seatunnel`, and the file structure example is:

		@Test
		public void testJsonFilterPatternWithFilePath() throws URISyntaxException, IOException {

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories #10049

Are you sure you want to change the base?

[Fix][FTP]Fix 'file_filterpattern' cannot filter directories #10049

Conversation

LiJie20190102 commented Nov 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

LiJie20190102 commented Nov 9, 2025

Uh oh!

davidzollo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangshenghang Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangshenghang commented Nov 11, 2025

Uh oh!

LiJie20190102 commented Nov 11, 2025

Uh oh!

dybyte Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidzollo commented Nov 11, 2025

Uh oh!

LiJie20190102 commented Nov 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

LiJie20190102 commented Nov 9, 2025 •

edited

Loading

zhangshenghang Nov 11, 2025 •

edited

Loading

dybyte Nov 11, 2025 •

edited

Loading