How should I publish artifacts?

[blackears]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

Misc

Discussion Details

I'm trying to put together a set of workflows to automate packaging a Python for Blender extension. I've created separte workflows for Windows, Mac and Linux. At the moment, I'm able to generate the artifacts and upload them. They all require downloading and attaching python wheels to the distribution, which makes the artifact file sizes quite large. (The Linux artifact is huge at ~3GB. The windows one is ~300Mb and the MacOS is ~100Mb).

I'd like to be able to attach them to my releases to that users can download them. One issue is that the release page does not allow you to attach huge files to it, so the giant Linux artifact is rejected. I was reading that you could use LFS to get around this, and was wondering if I could just copy my artifacts into an extensions directory with ".zip" added to LFS and then somehow reference that in the release?

The other issue is that workflow that uploads the artifact wraps it in a zip file. However, the Blender tool that I'm using to create the extensions outputs a single zip file already. I'd rather not have the artifact uploader zip and already zipped file and instead just copy all the output files from my workflows to some directory where I can store them and then reference them for my releases. Is there some way to alter these scripts to do that?

This is what my MacOS workflow looks like. The workflows for Windows and Linux are pretty similar:

name: 🍎 MacIntosh Builds
on: [ workflow_call, workflow_dispatch ]

jobs:
  build:
    name: 🍎 MacIntosh ${{ matrix.arch }}
    runs-on: ${{ matrix.runs_on }}
    strategy:
      fail-fast: false
      matrix:
        include:
          - arch: x64
            runs_on: macos-13
            py_arch: x64
            blender_version: "1.1.1"
            blender_arch: x64
            blender_platform: macos-x64
          - arch: arm64
            runs_on: macos-latest
            py_arch: arm64
            blender_version: "1.1.1"
            blender_arch: arm64
            blender_platform: macos-arm64

    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          submodules: recursive
        
      - name: Setup Python 3.x
        uses: actions/setup-python@v5
        with:
          python-version: 3.x
          architecture: ${{ matrix.py_arch }}  # x64 or arm64

      - name: Download Wheels
        run: |
          mkdir source/wheels
          # Optional, fail fast if no native wheel exists for this arch
          pip wheel --prefer-binary -w source/wheels allosaurus

      - name: Create manifest
        run: |
          cd source
          (cat ../blender_manifest_template_mac.toml; echo "wheels = ["; ls -1 wheels | sed 's/\(^.*$\)/  "\.\/wheels\/\1",/'; echo "]") > blender_manifest.toml

      - name: Setup Blender
        run: |
          cd ${{ github.workspace }}
          BLENDER_VERSION="4.5.2"
          BLENDER_VOLUME="blender-${BLENDER_VERSION}-macos-${{ matrix.arch }}"
          BLENDER_ARCHIVE="${BLENDER_VOLUME}.dmg"
          BLENDER_URL="https://download.blender.org/release/Blender4.5/${BLENDER_ARCHIVE}"
          curl -L ${BLENDER_URL} -o ${BLENDER_ARCHIVE}
          hdiutil attach ${BLENDER_ARCHIVE}
          echo "--ls /Volumes/Blender/Blender.app/Contents/MacOS--"
          ls /Volumes/Blender/Blender.app/Contents/MacOS

      - name: Build Extension
        run: |
          cd ${{ github.workspace }}
          mkdir extension
          "/Volumes/Blender/Blender.app/Contents/MacOS/Blender" --command extension build --source-dir source --output-filepath "extension/parrotLipsync-${{ matrix.blender_version }}-macos-${{ matrix.arch }}.zip"

      - name: Upload Package
        uses: actions/upload-artifact@v4
        with:
          include-hidden-files: true
          name: parrotLipsync-macos-${{ matrix.arch }}
          path: |
            ${{ github.workspace }}/extension/

[Krishnazzz]

When publishing large artifacts in GitHub Actions, here are professional best practices and key points:
Uploading Artifacts:

Use actions/upload-artifact@v4 to upload files or directories as workflow artifacts.

You can specify a file, directory, or pattern for uploading.

By default, this action compresses the files into a zip. For large files that do not compress well, set compression-level: 0 for faster uploads.

Avoid uploading whole folders containing already zipped files to prevent double compression.

Upload individual pre-zipped files directly by specifying their exact path to avoid the extra zip layer.

Example:

- uses: actions/upload-artifact@v4
  with:
    name: artifact-name
    path: path/to/your/prezipped-file.zip
    compression-level: 0

Artifact Size Limits and Storage:

A single artifact can be up to 5 GB in size.

Repository storage limits apply (2 GB per repository by default); enterprise accounts have their own quotas.

For very large files (like your 3 GB Linux artifact), if it exceeds the release upload limits (about 2 GB max per release asset), consider:

Splitting large artifacts into smaller parts.

Uploading artifacts to external storage services like AWS S3 or Azure Blob Storage during workflows and referencing them in release notes.

GitHub Releases and Artifacts:

Artifacts uploaded via the Actions workflow are separate from GitHub Releases.

To attach artifacts directly to GitHub Releases, use actions/upload-release-asset or marketplace tools like softprops/action-gh-release.

Release assets have their own size limits, so very large binaries might still need external hosting.

LFS and Artifacts:

Git LFS is for versioning large files in git repositories, not for artifact or release asset storage.

LFS cannot be used to bypass release asset size limits or to directly store workflow artifacts.

Workflow Example for Uploading a Single Zip Artifact:

- name: Upload Package
  uses: actions/upload-artifact@v4
  with:
    name: your-artifact-name
    path: ${{ github.workspace }}/extension/your-extension.zip
    compression-level: 0

Uploading to Releases:

- name: Upload Release Asset
  uses: softprops/action-gh-release@v1
  with:
    files: extension/your-extension.zip
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Summary:

Upload your pre-zipped extension files directly to avoid double zipping.

For very large files, split uploads or use external storage linked in your releases.

Use dedicated release asset upload actions to attach files to GitHub Releases.

Git LFS is not suitable for storing release artifacts or large workflow outputs.

This approach helps you manage artifact size, prevents upload rejections, and offers users easy access to your Blender extensions professionally and efficiently.

References: GitHub Docs(actions/upload-artifact), GitHub Release Uploads, community best practices.

[syedlabib4]

For large artifacts in GitHub Actions, here are some approaches you can take:

Use GitHub LFS – You can store large files (like your Linux artifact) in LFS, then reference them in releases. Just commit the files to LFS with .gitattributes configured, and your release can link to them.

Skip additional zipping – Instead of using the default actions/upload-artifact which zips files automatically, you can:

Use actions/upload-release-asset to directly attach existing zip files to a release

[blackears]

I'm trying to upload my artifact like this now, but it's still wrapping my zip file inside another zip file.

      - name: Upload Package
        uses: actions/upload-artifact@v4
        with:
          name: parrotLipsync-${{ matrix.blender_version }}-macos-${{ matrix.arch }}
          path: "extension/parrotLipsync-${{ matrix.blender_version }}-macos-${{ matrix.arch }}.zip"
          compression-level: 0

Also, the actions/upload-release-asset seems to depend on actions/create-release@v1 already being called. If I do create a release, what happens if I run this target multiple times? Does it overwrite the previous release?

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬