Server hangs when running with Swift 6.1 image

[finestructure]

The server process seems to be hanging when deploying the new swift:6.1.0-jammy based base image.

Compare this launch of the 2.116.27 docker image (swift:6.0.3-jammy) with 2.117.0 (swift:6.1.0-jammy):

In both cases, the process has no configuration and should immediately raise an internal server error. However, the Swift 6.1 based image just hangs.

[finestructure]

The good news is this issue is actually reproducible on Linux and macOS. My current test matrix is as follows:

	swift:6.1-macOS arm64	swift:6.1-jammy arm64
Vapor template release		✅
spi-server debug		✅
spi-server release	❌	❌

(Blanks I haven't bothered testing, they're very likely ok.)

Something we're doing very early in the startup process is hanging the entire process in release mode. It's happening in the entry point:

@main
enum Entrypoint {
    static func main() async throws {
        print("\(#function): start")
        defer {
            print("\(#function): end")
        }
        var env = try Environment.detect()
        print("1")
        try LoggingSystem.bootstrap(from: &env)

        print("2")
        let app = try await Application.make(env)

        do {
            print("3")
            try await configure(app)
        } catch {
            app.logger.report(error: error)
            try? await app.asyncShutdown()
            throw error
        }
        try await app.execute()
        try await app.asyncShutdown()
    }
}

Running this yields:

[1552/1552] Linking Run
Build of product 'Run' complete! (244.64s)
main(): start
1
2
^C⏎

We never even get to configure. Push comes to shove, we could deploy a -c debug binary but I want to drill deeper before we do that.

[heckj]

I'm super curious where this ends up, thanks for including the detail here.

[finestructure]

In order to try and debug this further, I had to fork Vapor, which required me to drop SwiftPM as a dependency. Doing this actually fixed the hang. The Vapor fork is the latest release + a couple of print statements, so I'm operating under the assumption that the SwiftPM dependency is the problem.

I'll try to move us to release/6.0 on the main branch first, addressing the new concurrency issues along the way, and then to release/6.1 subsequently.

[finestructure]

Moving to SwiftPM release/6.1 fixes the hang in my local testing. We should be able to avoid the hang by updating the branch at the same time we move to 6.1.

[finestructure]

Ok, updating to SwiftPM release/6.1 is not sufficient. I thought I got that working on Friday but perhaps it was working only when I removed SwiftPM as a dependency entirely?

I'm now looking into dropping SwiftPM as a dependency. However, there are some issues with that.

There are two modules we need, PackageCollectionsModel and PackageCollectionsSigning.

• PackageCollectionsModel is easy. It has no upstream dependencies, it's a straight up copy into a new module.
• PackageCollectionsSigning depends on PackageCollectionsModel (not a problem) but also on the "Basics" SwiftPM module as well as "Crypto" from swift-crypto.
  • swift-crypto is not a problem, but "Basics" again has upstream dependencies:
    • SPMSQLLite3
    • swift-collections
    • swift-tools-support-core
    • swift-system

This will probably only work if we can avoid depending on "Basics".

Unfortunately, its dependency on "Basics" also means that there's no way we'll get SwiftPM to pull this into its own upstream dependency which we could import.

So the only two choices we have are:

• keep using SwiftPM as a dependency with all its downsides
• copy the two module sources and make (probably some nontrivial) changes that allow us to use them standalone, and maintain that going forward

I'll briefly explore replacing the "Basics" module. Perhaps PackageCollectionsSigning isn't using too much of it and I can copy some more bits.

This is to unblock us. I don't think we want to maintain signing code on our end in the long run.

[finestructure]

Ok, even if outlining the SwiftPM modules will work in the end (not sure) this is not something we'll want to do.

I've already collected a big number of non-trivial changes just trying to get it to compile (it doesn't yet) and now I'm seeing API changes to package collections signing, as well as concurrency warnings we'd need to fix if we keep the modules in our codebase.

I think we seriously need to consider dropping package collections support or at least signed package collections support.

[finestructure]

We definitely cannot replace SwiftPM as a dependency. Neither will the SwiftPM be able to pull this into an upstream standalone package that we could depend on, because it's using SwiftPM internals they cannot also push upstream.

Nor can we copy the modules, pretty much for the same reason. Replacing/removing those dependencies is not feasible and neither is maintaining patched versions of those modules even if we managed to do it.

[finestructure]

I'll try once more to get the release binary to run with SwiftPM release/6.1, which seems to have worked at some point last week.

[daveverwer]

I think we seriously need to consider dropping package collections support or at least signed package collections support.

I think dropping this temporarily until we get a better solution to this would probably be fine, but we should probably gather some information on how often collections are being used before we do. We don't have any stats on that already, do we? If not, we could temporarily use the Plausible back-end code that we already implemented to get a quick read on it for a few days. I expect it to be very low, but I'm guessing. We should also remember when looking at the stats that the pages are cached for 2 hours, but it would be good to see how many unique collections are being accessed for a few days.

Obviously our ability to upgrade to 6.1 is critical to our being able to get 6.1 compatibility going, which is very important, but there are plenty of articles and videos that people may see that says we have this feature, so we shouldn't disable it unless absolutely necessary.

Is the other option to make our analysis process not dependent on our base image? I know we have talked about decoupling this before and decided not to, but is that worth revisiting now? It isn't ideal that we need to adopt new versions of Swift in order to analyze packages.

[finestructure]

If we end up doing it, we should just skip the signing. That way the basic feature is still available and URLs aren't 404ing even though the produced collection is somewhat flawed.

As to the base image, we used to ship the base image with a different compiler version for analysis but that was a whole different can of worms which I'd rather we didn't go back to if it can be avoided at all.

[daveverwer]

As to the base image, we used to ship the base image with a different compiler version for analysis but that was a whole different can of worms which I'd rather we didn't go back to if it can be avoided at all.

I know this isn't the place to get into this in detail, but my thought was that the analysis process could still live in the base image on whatever that version of Swift is, but the call to run swift package dump-package would run via docker using the latest Swift image. I know that's running docker inside docker, but I believe that works?

[finestructure]

That could theoretically be made to work but it'd be much more complicated than installing an additional toolchain into the base image like we used to.

Not only would the docker image need to contain docker so we could launch the command, we'd also have to make sure the volume map is set up in a way that the input files are visible to the additional docker container. It's something we do in the builder and it's very complicated. The benefit there is that we run multiple Swift versions and don't have to have to update the base image with each one when any change, and that's worth it.

[finestructure]

I have verified that it's feasible to drop SwiftPM as a dependency and only pull in PackageCollectionsModel and PackageIdentity from the PackageModel module*. It requires dropping signed package collections but the release binary works and all tests pass with those changes:

https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/actions/runs/14467163569/job/40572136308?pr=3761

I'll still want to verify if there's not another way to preserve functionality before we go through with this.

The adopted source files aren't quite drop-ins without changes, so it's not an ideal scenario even if we disregard the loss of signed package collections.

*) PackageIdentity was a SwiftPM module dependency I hadn't listed above because it had been drowned out in the compiler errors.