Commit dbb8143
[MINOR][SS][DOC] Added missing config
## What changes were proposed in this pull request?
added the missing config for structured streaming when using file source.
from the code we have
```
/**
* Maximum age of a file that can be found in this directory, before it is ignored. For the
* first batch all files will be considered valid. If `latestFirst` is set to `true` and
* `maxFilesPerTrigger` is set, then this parameter will be ignored, because old files that are
* valid, and should be processed, may be ignored. Please refer to SPARK-19813 for details.
*
* The max age is specified with respect to the timestamp of the latest file, and not the
* timestamp of the current system. That this means if the last file has timestamp 1000, and the
* current system time is 2000, and max age is 200, the system will purge files older than
* 800 (rather than 1800) from the internal state.
*
* Default to a week.
*/
val maxFileAgeMs: Long =
Utils.timeStringAsMs(parameters.getOrElse("maxFileAge", "7d"))
```
which is not documented.
also the file processing order was not mentioned but in the code we specifically select the file list based on file mtime:
```scala
private val fileSortOrder = if (sourceOptions.latestFirst) {
logWarning(
"""'latestFirst' is true. New files will be processed first, which may affect the watermark
|value. In addition, 'maxFileAge' will be ignored.""".stripMargin)
implicitly[Ordering[Long]].reverse
} else {
implicitly[Ordering[Long]]
}
val files = allFiles.sortBy(_.getModificationTime)(fileSortOrder).map { status =>
(status.getPath.toUri.toString, status.getModificationTime)
}
```
---------

---------

Closes #24548 from linehrr/master.
Lead-authored-by: ryne.yang <[email protected]>
Co-authored-by: Ryne Yang <[email protected]>
Co-authored-by: linehrr <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>maxFileAge in file streaming source1 parent bcd3b61 commit dbb8143
1 file changed
Lines changed: 3 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
510 | 510 | | |
511 | 511 | | |
512 | 512 | | |
513 | | - | |
514 | | - | |
| 513 | + | |
515 | 514 | | |
516 | 515 | | |
517 | 516 | | |
| |||
541 | 540 | | |
542 | 541 | | |
543 | 542 | | |
| 543 | + | |
| 544 | + | |
544 | 545 | | |
545 | 546 | | |
546 | 547 | | |
| |||
0 commit comments