Powershell never exits on Windows and the build hangs forever

[wilg]

I don't know if it something with my configuration of a local runner, but something is preventing the Docker container from terminating. I think it is that Powershell never exits after completing the build script, even when explicitly exited. I've been taking some stabs in the dark on my fork, but haven't had any success here. https://github.com/wilg/unity-builder

[webbertakken]

Would you mind attaching your workflow file, so we can see your configuration?
Could you also tell us what kind of runner you're using?

[wilg]

I'm using self-hosted/local runners, one on Mac, one on Windows. My idea is to have the Mac build Mac IL2CPP and have the Windows machine build Windows IL2CPP and Linux IL2CPP.

The Mac build works properly (though I haven't gotten Steam uploads working yet because of game-ci/steam-deploy#35).

The Windows machine is Windows 11 Pro. I installed Docker via Docker Desktop, maybe that's at issue.

The Windows build fully completes, but the Docker instance hangs. It looks like the Powershell script completes (no code is called after the manual exit I added). However I see that it's possible for Powershell to hang even in this case. https://stackoverflow.com/questions/2041799/powershell-script-gets-stuck-doesnt-exit-when-called-from-batch-file I tried various suggested wacky ways of exiting Powershell, but it didn't work. Maybe I missed one.

I have some puzzling configuration issues with memory with docker on Windows, documented here, but manually setting memory/cpu flags got it working. #432

If I manually terminate the container from Docker Desktop, docker run exits (though with a crazy status code of what appears to be INT_MAX) and the build proceeds as normal. The player is fully built and uploaded.

name: Build Players
on:
  workflow_dispatch:
    inputs:
      build_mac:
        description: "Build for macOS"
        required: true
        type: boolean
      build_windows:
        description: "Build for Windows"
        required: true
        type: boolean
      build_linux:
        description: "Build for Linux"
        required: true
        type: boolean
env:
  NODE_TLS_REJECT_UNAUTHORIZED: 0
jobs:
  windows_host:
    if: inputs.build_linux || inputs.build_windows
    name: Build for ${{ matrix.platform[1] }} on ${{ matrix.platform[0] }}
    runs-on: [self-hosted, "${{ matrix.platform[0] }}"]
    strategy:
      fail-fast: false
      matrix:
        unityVersion:
          - 2021.3.9f1
        platform:
          - [Windows, StandaloneWindows64]
    steps:
      - uses: actions/checkout@v3
        name: Git Checkout + LFS
        with:
          lfs: true
          fetch-depth: 0
          clean: false

      - uses: wilg/unity-builder@main
        name: Build Unity player
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
          UNITY_BUILDER_DOCKER_ARGS: "-m 28g --cpus 10 --gpus all"
        with:
          unityVersion: ${{ matrix.unityVersion }}
          targetPlatform: ${{ matrix.platform[1] }}
          buildName: Gumshoe
          allowDirtyBuild: true
          # customParameters: "-myParameter myValue -myBoolean -ThirdParameter andItsValue"

      - uses: actions/upload-artifact@v3
        name: Upload builds artifact
        with:
          name: ${{ matrix.platform[1] }}-build
          path: build/${{ matrix.platform[1] }}
          if-no-files-found: error

  mac_host:
    if: inputs.build_mac
    name: Build for ${{ matrix.platform[1] }} on ${{ matrix.platform[0] }}
    runs-on: [self-hosted, "${{ matrix.platform[0] }}"]
    strategy:
      fail-fast: false
      matrix:
        unityVersion:
          - 2021.3.9f1
        platform:
          - [macOS, StandaloneOSX]
    steps:
      - uses: actions/checkout@v3
        name: Git Checkout + LFS
        with:
          lfs: true
          fetch-depth: 0
          clean: false

      - uses: wilg/unity-builder@main
        id: build
        name: Build Unity player
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
        with:
          unityVersion: ${{ matrix.unityVersion }}
          targetPlatform: ${{ matrix.platform[1] }}
          buildName: Gumshoe
          allowDirtyBuild: true
          # customParameters: "-myParameter myValue -myBoolean -ThirdParameter andItsValue"

      - uses: wilg/steam-deploy@main
        env:
          STEAM_HOME: ~/Library/Application Support/Steam/
        with:
          username: ${{ secrets.STEAM_USERNAME }}
          password: ${{ secrets.STEAM_PASSWORD }}
          configVdf: ${{ secrets.STEAM_CONFIG_VDF}}
          ssfnFileName: ${{ secrets.STEAM_SSFN_FILE_NAME }}
          ssfnFileContents: ${{ secrets.STEAM_SSFN_FILE_CONTENTS }}
          appId: 2154550
          buildDescription: ${{ steps.build.outputs.buildVersion}}
          rootPath: build
          depot2Path: StandaloneOSX
          releaseBranch: beta

      # - name: Zip build
      #   run: |
      #     ditto -c -k --keepParent --sequesterRsrc build/${{ matrix.platform[1] }}/Gumshoe.app build/${{ matrix.platform[1] }}/artifacts/Gumshoe.app.zip

      # - uses: actions/upload-artifact@v3
      #   name: Upload builds artifact
      #   with:
      #     name: ${{ matrix.platform[1] }}-build-zip
      #     path: build/${{ matrix.platform[1] }}/artifacts
      #     if-no-files-found: error

[wilg]

I had to set allowDirtyBuild: true because the build process seems to create an Editor folder in my assets and copy the build scripts in, which causes uncommitted changes in Git. Not sure how this is intended to work normally.

[webbertakken]

I had to set allowDirtyBuild: true because the build process seems to create an Editor folder in my assets and copy the build scripts in, which causes uncommitted changes in Git. Not sure how this is intended to work normally.

This is added after the check for a dirty branch (or at least supposed to). For linux this works as expected. On windows it should to but I haven't verified this. It is possible for self-hosted runners that don't wipe the workspace to get this error on subsequent runs as well.

[wilg]

Ah, that must be the issue. I don't wipe the workspace (which makes everything super fast!) I guess I can just add it to .gitignore.

[wilg]

Hmmm... I replaced the build script with

Write-Output "Exit 1"

Exit 1

Write-Output "THIS IS AN IMPOSSIBLE LINE"

And it properly exits and continues to the next step...

[wilg]

Narrowed it to

& "c:\steps\build.ps1"

in entrypoint.ps1 causing the hang. so something that happens in the build script is preventing exit. perhaps a process started there somehow never closes

[wilg]

Actually, it's worse. It seems nondeterministic potentially.

[wilg]

I can't say it was quick for me to figure it out, but it looks like removing this line fixes the issue:

# Register the Visual Studio installation so Unity can find it
regsvr32 C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll

[webbertakken]

Perhaps it's not being mounted in the docker container. When I tested this locally it worked as expected though.

The exit part is definitely something we need to add, indeed.

[ManuelRauber]

Is there something I can do to help figure out this issue?

I've noticed it when I've switched from 2022.2.5f1 to 2022.2.14f1 (commit: BoundfoxStudios/fairy-tale-defender@79d1cec) the build for Windows tend to hang so GitHub aborts them after 6 hours.

Example build that hang: https://github.com/BoundfoxStudios/fairy-tale-defender/actions/runs/4811774943
That one was working: https://github.com/BoundfoxStudios/fairy-tale-defender/actions/runs/4811536351

The one that hang just added one line of code: BoundfoxStudios/fairy-tale-defender@88332be

So it's not the matter of the code or something.

Also, as @wilg mentioned before it seems totally random.

[davidmfinol]

I've been seeing this issue with my windows builds.
I normally just re-run the job after it times out, but this can be pretty annoying. One time I had to retry ~5 times before it finally succeeded.

[szunami]

I've been getting hit with this as well :(

For my builds, I notice that when they hang there isn't output for a long period of time. I wonder if something like inotifywait could be used to kill jobs when this is detected (as a workaround, not a real fix)

Looks like there is a windows port of inotifywait, not sure how mature it is https://github.com/thekid/inotify-win