Clone files on Darwin rather than copying them#2823
Merged
Conversation
rnro
force-pushed
the
copy_file_on_darwin_use_clone
branch
from
5901179 to
adeb5b6
Compare
glbrntt
requested changes
Aug 2, 2024
Contributor
glbrntt
left a comment
glbrntt
left a comment
There was a problem hiding this comment.
Looks great, thanks for fixing this Rick. The actual fix looks good but I left a couple of nitty things to fix up.
Comment on lines
+368
to
+369
| let opt = try? receiver.getOption(.socketOption(.so_rcvbuf)).wait() | ||
| let receiveBufferSize = Int(opt ?? 8192) |
Contributor
There was a problem hiding this comment.
Can you do this in separate PR?
| state: nil, | ||
| flags: flags | ||
| ).mapError { errno in | ||
| FileSystemError.fcopyfile( |
Contributor
There was a problem hiding this comment.
Can you update this helper? One of the things it'll do is attach a SyscallError to the FileSystemError which will include the name of the syscall, now it will incorrectly be fcopyfile.
Where it doesn't make sense to duplicate the helper I've tended to just pass in the name of the syscall: FileSystemError.copyfile("copyfile", ... )
| Can't copy file from '\(sourcePath)' to '\(destinationPath)', the destination \ | ||
| path already exists. | ||
| """ | ||
| case .fileExists: |
Contributor
There was a problem hiding this comment.
Can you update the error tests? They check that errnos get mapped to an appropriate file system error code.
|
|
||
| #if canImport(Darwin) | ||
| /// copyfile(3): Copy a file from one file to another. | ||
| /// copyfile(3): Copy a file from one file to another. (fcopyfile) |
Contributor
There was a problem hiding this comment.
nit
Suggested change
| /// copyfile(3): Copy a file from one file to another. (fcopyfile) | |
| /// fcopyfile(3): Copy a file from one file to another. |
|
|
||
| #if canImport(Darwin) | ||
| @_spi(Testing) | ||
| public static func copyfile( |
Contributor
There was a problem hiding this comment.
Can you add a test for this please?
?
Motivation: Copying regular files on Darwin was taking more time than it needed to because it was manually opening files and copying over bites rather than making use of the system's ability to create CoW clones. Modifications: The primary change is to switch `copyItem` to be backed by Darwin's `copyfile` method rather than `fcopyfile`. `fcopyfile` ignores the flag we were passing in to indicate that the file should be cloned if possible, whereas `copyfile` is a higher-level API which takes this into account. This change exposes a `copyfile` wrapper method as a new public function. This change also adds a workaround to a seeming compiler bug on channel options blocking building tests on more recent Xcodes. Result: Copying files on Darwin will be faster.
rnro
force-pushed
the
copy_file_on_darwin_use_clone
branch
from
aa683ab to
6eb5833
Compare
cgrindel-self-hosted-renovate Bot
referenced
this pull request
in cgrindel/rules_swift_package_manager
Aug 19, 2024
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [apple/swift-nio](https://togithub.com/apple/swift-nio) | minor | `2.68.0` -> `2.70.0` | --- ### Release Notes <details> <summary>apple/swift-nio (apple/swift-nio)</summary> ### [`v2.70.0`](https://togithub.com/apple/swift-nio/releases/tag/2.70.0): SwiftNIO 2.70.0 [Compare Source](https://togithub.com/apple/swift-nio/compare/2.69.0...2.70.0) <!-- Release notes generated using configuration in .github/release.yml at main --> #### What's Changed ##### SemVer Minor - `FileSystem.copyItem` can parallelise directory copy by [@​UncleMattHope](https://togithub.com/UncleMattHope) in [https://github.com/apple/swift-nio/pull/2806](https://togithub.com/apple/swift-nio/pull/2806) - `ChannelOption`: Allow types to be accessed with leading dot syntax by [@​ayush1794](https://togithub.com/ayush1794) in [https://github.com/apple/swift-nio/pull/2816](https://togithub.com/apple/swift-nio/pull/2816) - Make `EventLoopPromise` conform to Equatable by [@​gjcairo](https://togithub.com/gjcairo) in [https://github.com/apple/swift-nio/pull/2714](https://togithub.com/apple/swift-nio/pull/2714) - Provide a default `CopyStrategy` overload for copyItem. by [@​UncleMattHope](https://togithub.com/UncleMattHope) in [https://github.com/apple/swift-nio/pull/2818](https://togithub.com/apple/swift-nio/pull/2818) ##### SemVer Patch - Better align shutdown semantics of testing event loops by [@​simonjbeaumont](https://togithub.com/simonjbeaumont) in [https://github.com/apple/swift-nio/pull/2800](https://togithub.com/apple/swift-nio/pull/2800) - Clone files on Darwin rather than copying them by [@​rnro](https://togithub.com/rnro) in [https://github.com/apple/swift-nio/pull/2823](https://togithub.com/apple/swift-nio/pull/2823) ##### Other Changes - Fix compose file used in update-benchmark-thresholds script by [@​simonjbeaumont](https://togithub.com/simonjbeaumont) in [https://github.com/apple/swift-nio/pull/2808](https://togithub.com/apple/swift-nio/pull/2808) - Remove advice to generate linux tests. by [@​PeterAdams-A](https://togithub.com/PeterAdams-A) in [https://github.com/apple/swift-nio/pull/2807](https://togithub.com/apple/swift-nio/pull/2807) - Make `testInstantTCPConnectionResetThrowsError` more reliable by [@​hamzahrmalik](https://togithub.com/hamzahrmalik) in [https://github.com/apple/swift-nio/pull/2810](https://togithub.com/apple/swift-nio/pull/2810) - \[CI] Add `shellcheck` and fix up warnings by [@​FranzBusch](https://togithub.com/FranzBusch) in [https://github.com/apple/swift-nio/pull/2809](https://togithub.com/apple/swift-nio/pull/2809) - \[CI] Fix docs check by [@​FranzBusch](https://togithub.com/FranzBusch) in [https://github.com/apple/swift-nio/pull/2811](https://togithub.com/apple/swift-nio/pull/2811) - \[CI] Add Swift 6 language mode workflow by [@​FranzBusch](https://togithub.com/FranzBusch) in [https://github.com/apple/swift-nio/pull/2812](https://togithub.com/apple/swift-nio/pull/2812) - Fix test compilation on non-macOS Darwin platforms by [@​simonjbeaumont](https://togithub.com/simonjbeaumont) in [https://github.com/apple/swift-nio/pull/2817](https://togithub.com/apple/swift-nio/pull/2817) - Add `.index-build` to `.gitignore` by [@​MaxDesiatov](https://togithub.com/MaxDesiatov) in [https://github.com/apple/swift-nio/pull/2819](https://togithub.com/apple/swift-nio/pull/2819) - \[CI] Add action and workflow to check for semver label by [@​FranzBusch](https://togithub.com/FranzBusch) in [https://github.com/apple/swift-nio/pull/2814](https://togithub.com/apple/swift-nio/pull/2814) - Update repository docs for swift-version support and recent CI check changes by [@​UncleMattHope](https://togithub.com/UncleMattHope) in [https://github.com/apple/swift-nio/pull/2815](https://togithub.com/apple/swift-nio/pull/2815) - Fix failing build for test by [@​ayush1794](https://togithub.com/ayush1794) in [https://github.com/apple/swift-nio/pull/2824](https://togithub.com/apple/swift-nio/pull/2824) - Fix typo in comment in `WebSocketErrorCodes.swift` by [@​valeriyvan](https://togithub.com/valeriyvan) in [https://github.com/apple/swift-nio/pull/2604](https://togithub.com/apple/swift-nio/pull/2604) - \[CI] Add a scheduled workflow for tests and benchmarks by [@​FranzBusch](https://togithub.com/FranzBusch) in [https://github.com/apple/swift-nio/pull/2822](https://togithub.com/apple/swift-nio/pull/2822) - \[CI] Fix label check by [@​FranzBusch](https://togithub.com/FranzBusch) in [https://github.com/apple/swift-nio/pull/2827](https://togithub.com/apple/swift-nio/pull/2827) #### New Contributors - [@​UncleMattHope](https://togithub.com/UncleMattHope) made their first contribution in [https://github.com/apple/swift-nio/pull/2806](https://togithub.com/apple/swift-nio/pull/2806) - [@​ayush1794](https://togithub.com/ayush1794) made their first contribution in [https://github.com/apple/swift-nio/pull/2816](https://togithub.com/apple/swift-nio/pull/2816) - [@​valeriyvan](https://togithub.com/valeriyvan) made their first contribution in [https://github.com/apple/swift-nio/pull/2604](https://togithub.com/apple/swift-nio/pull/2604) **Full Changelog**: apple/swift-nio@2.69.0...2.70.0 ### [`v2.69.0`](https://togithub.com/apple/swift-nio/releases/tag/2.69.0): SwiftNIO 2.69.0 [Compare Source](https://togithub.com/apple/swift-nio/compare/2.68.0...2.69.0) <!-- Release notes generated using configuration in .github/release.yml at main --> #### What's Changed ##### SemVer Minor - Add manual control to `NIOLockedValueBox` by [@​glbrntt](https://togithub.com/glbrntt) in [https://github.com/apple/swift-nio/pull/2786](https://togithub.com/apple/swift-nio/pull/2786) - ChannelHandler: provide static `(un)wrap(In|Out)bound(In|Out)` by [@​weissi](https://togithub.com/weissi) in [https://github.com/apple/swift-nio/pull/2791](https://togithub.com/apple/swift-nio/pull/2791) ##### SemVer Patch - Pre-box some errors to reduce allocations by [@​glbrntt](https://togithub.com/glbrntt) in [https://github.com/apple/swift-nio/pull/2765](https://togithub.com/apple/swift-nio/pull/2765) - Allow in-place mutation of `NIOLoopBoundBox.value` by [@​dnadoba](https://togithub.com/dnadoba) in [https://github.com/apple/swift-nio/pull/2771](https://togithub.com/apple/swift-nio/pull/2771) - Avoid creating a yield ID counter per async writer by [@​glbrntt](https://togithub.com/glbrntt) in [https://github.com/apple/swift-nio/pull/2768](https://togithub.com/apple/swift-nio/pull/2768) - Combine the two `NIOAsyncChannel` channel handlers by [@​glbrntt](https://togithub.com/glbrntt) in [https://github.com/apple/swift-nio/pull/2779](https://togithub.com/apple/swift-nio/pull/2779) - Use the new Android overlay and Bionic module from Swift 6 by [@​finagolfin](https://togithub.com/finagolfin) in [https://github.com/apple/swift-nio/pull/2784](https://togithub.com/apple/swift-nio/pull/2784) - Change `unsafeDownCast` to `as!` by [@​FranzBusch](https://togithub.com/FranzBusch) in [https://github.com/apple/swift-nio/pull/2802](https://togithub.com/apple/swift-nio/pull/2802) ##### Other Changes - CI migration to GitHub Action by [@​FranzBusch](https://togithub.com/FranzBusch) in ([https://github.com/apple/swift-nio/pull/2760](https://togithub.com/apple/swift-nio/pull/2760) [https://github.com/apple/swift-nio/pull/2762](https://togithub.com/apple/swift-nio/pull/2762) [https://github.com/apple/swift-nio/pull/2763](https://togithub.com/apple/swift-nio/pull/2763) [https://github.com/apple/swift-nio/pull/2764](https://togithub.com/apple/swift-nio/pull/2764) [https://github.com/apple/swift-nio/pull/2767](https://togithub.com/apple/swift-nio/pull/2767) [https://github.com/apple/swift-nio/pull/2766](https://togithub.com/apple/swift-nio/pull/2766) [https://github.com/apple/swift-nio/pull/2776](https://togithub.com/apple/swift-nio/pull/2776) [https://github.com/apple/swift-nio/pull/2780](https://togithub.com/apple/swift-nio/pull/2780) [https://github.com/apple/swift-nio/pull/2785](https://togithub.com/apple/swift-nio/pull/2785) [https://github.com/apple/swift-nio/pull/2781](https://togithub.com/apple/swift-nio/pull/2781) [https://github.com/apple/swift-nio/pull/2787](https://togithub.com/apple/swift-nio/pull/2787) [https://github.com/apple/swift-nio/pull/2783](https://togithub.com/apple/swift-nio/pull/2783) [https://github.com/apple/swift-nio/pull/2789](https://togithub.com/apple/swift-nio/pull/2789) [https://github.com/apple/swift-nio/pull/2790](https://togithub.com/apple/swift-nio/pull/2790)) - Ignore format commit from git blame by [@​FranzBusch](https://togithub.com/FranzBusch) in [https://github.com/apple/swift-nio/pull/2796](https://togithub.com/apple/swift-nio/pull/2796) [https://github.com/apple/swift-nio/pull/2797](https://togithub.com/apple/swift-nio/pull/2797) [https://github.com/apple/swift-nio/pull/2801](https://togithub.com/apple/swift-nio/pull/2801) [https://github.com/apple/swift-nio/pull/2803](https://togithub.com/apple/swift-nio/pull/2803) - Adopt swift-format by [@​FranzBusch](https://togithub.com/FranzBusch) in [https://github.com/apple/swift-nio/pull/2794](https://togithub.com/apple/swift-nio/pull/2794) - `HTTPPart` Documentation Clarification by [@​dimitribouniol](https://togithub.com/dimitribouniol) in [https://github.com/apple/swift-nio/pull/2775](https://togithub.com/apple/swift-nio/pull/2775) - Add benchmark for creating `NIOAsyncChannel` by [@​glbrntt](https://togithub.com/glbrntt) in [https://github.com/apple/swift-nio/pull/2774](https://togithub.com/apple/swift-nio/pull/2774) - Disable warnings as errors on Swift 6 and main by [@​glbrntt](https://togithub.com/glbrntt) in [https://github.com/apple/swift-nio/pull/2793](https://togithub.com/apple/swift-nio/pull/2793) **Full Changelog**: apple/swift-nio@2.68.0...2.69.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC4xOC4xIiwidXBkYXRlZEluVmVyIjoiMzguMzkuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOltdfQ==--> Co-authored-by: cgrindel-self-hosted-renovate[bot] <139595543+cgrindel-self-hosted-renovate[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
Copying regular files on Darwin was taking more time than it needed to because it was manually opening files and copying over bites rather than making use of the system's ability to create CoW clones.
Modifications:
The primary change is to switch
copyItemto be backed by Darwin'scopyfilemethod rather thanfcopyfile.fcopyfileignores the flag we were passing in to indicate that the file should be cloned if possible, whereascopyfileis a higher-level API which takes this into account.This change exposes a
copyfilewrapper method as a new public function.This change also adds a workaround to a seeming compiler bug on channel options blocking building tests on more recent Xcodes.
Result:
Copying files on Darwin will be faster.