-
Notifications
You must be signed in to change notification settings - Fork 265
chore: Update for 0.3.0 release, prepare for 0.4.0 development #970
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 18 commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
5a84b35
Generate changelog for 0.3.0 release
andygrove ec13218
change maven version from 0.3.0-SNAPSHOT to 0.3.0
andygrove 56764d2
update version in diffs
andygrove 3783faa
Merge remote-tracking branch 'public/changelog-0.3.0' into branch-0.3
andygrove 7c68b2d
update scripts
andygrove 56e6db6
update docs
andygrove 4a03bb9
Merge remote-tracking branch 'apache/main' into post-0.3.0-fixes
andygrove c6d6081
prepare for 0.4.0
andygrove 167701f
prepare for 0.4.0
andygrove f9b4684
prepare for 0.4.0
andygrove dd773b4
update CI
andygrove 29f1ec0
improve release instructions
andygrove f9e00f1
more release note edits + formatting
andygrove 93a8523
github repo release
andygrove 39eaca3
remove GH_TOKEN references
andygrove 76fb39f
fix an error in the release docs
andygrove 70f98b9
fix maven url
andygrove 01552e8
fix maven url
andygrove 0f781af
Merge remote-tracking branch 'apache/main' into post-0.3.0-fixes
andygrove eae3cb4
regenerate docs
andygrove File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,9 +17,9 @@ specific language governing permissions and limitations | |
| under the License. | ||
| --> | ||
|
|
||
| # Aapche DataFusion Comet: Source Release Process | ||
| # Apache DataFusion Comet: Release Process | ||
|
|
||
| This documentation is for creating an official source release of Apache DataFusion Comet. | ||
| This documentation explains the release process for Apache DataFusion Comet. | ||
|
|
||
| ## Creating the Release Candidate | ||
|
|
||
|
|
@@ -49,12 +49,18 @@ git checkout -b branch-0.1 | |
| git push apache branch-0.1 | ||
| ``` | ||
|
|
||
| Create and merge a PR against the release branch to update the Maven version from `0.3.0-SNAPSHOT` to `0.1.0` | ||
| Update the `pom.xml` files in the release branch to update the Maven version from `0.1.0-SNAPSHOT` to `0.1.0`. | ||
|
|
||
| There is no need to update the Rust crate versions because they will already be `0.1.0`. | ||
|
|
||
| ### Update Version in main | ||
|
|
||
| Create a PR against the main branch to update the Rust crate version to `0.2.0` and the Maven version to `0.2.0-SNAPSHOT`. | ||
| The Spark diffs also need updating. | ||
| Create a PR against the main branch to prepare for developing the next release: | ||
|
|
||
| - Update the Rust crate version to `0.2.0`. | ||
| - Update the Maven version to `0.2.0-SNAPSHOT` (both in the `pom.xml` files and also in the diff files | ||
| under `dev/diffs`). | ||
| - Update the CI scripts under the `.github` directory. | ||
|
|
||
| ### Generate the Change Log | ||
|
|
||
|
|
@@ -81,26 +87,29 @@ python3 generate-changelog.py 0.0.0 HEAD 0.1.0 > ../changelog/0.1.0.md | |
| Create a PR against the _main_ branch to add this change log and once this is approved and merged, cherry-pick the | ||
| commit into the release branch. | ||
|
|
||
| ### Build the jars | ||
| ### Build the jars | ||
|
|
||
| #### Setup to do the build | ||
| The build process requires Docker. Download the latest Docker Desktop from https://www.docker.com/products/docker-desktop/. | ||
| If you have multiple docker contexts running switch to the context of the Docker Desktop. For example - | ||
|
|
||
| ```shell | ||
| The build process requires Docker. Download the latest Docker Desktop from https://www.docker.com/products/docker-desktop/. | ||
| If you have multiple docker contexts running switch to the context of the Docker Desktop. For example - | ||
|
|
||
| ```shell | ||
| $ docker context ls | ||
| NAME DESCRIPTION DOCKER ENDPOINT ERROR | ||
| default Current DOCKER_HOST based configuration unix:///var/run/docker.sock | ||
| desktop-linux Docker Desktop unix:///Users/parth/.docker/run/docker.sock | ||
| my_custom_context * tcp://192.168.64.2:2376 | ||
|
|
||
| $ docker context use desktop-linux | ||
| ``` | ||
| ``` | ||
|
|
||
| #### Run the build script | ||
| The `build-release-comet.sh` script will create a docker image for each architecture and use the image | ||
|
|
||
| The `build-release-comet.sh` script will create a docker image for each architecture and use the image | ||
| to build the platform specific binaries. These builder images are created every time this script is run. | ||
| The script optionally allows overriding of the repository and branch to build the binaries from (Note that | ||
| the local git repo is not used in the building of the binaries, but it is used to build the final uber jar). | ||
| The script optionally allows overriding of the repository and branch to build the binaries from (Note that | ||
| the local git repo is not used in the building of the binaries, but it is used to build the final uber jar). | ||
|
|
||
| ```shell | ||
| Usage: build-release-comet.sh [options] | ||
|
|
@@ -122,8 +131,10 @@ cd dev/release && ./build-release-comet.sh && cd ../.. | |
| ``` | ||
|
|
||
| #### Build output | ||
| The build output is installed to a temporary local maven repository. The build script will print the name of the repository | ||
| location at the end. This location will be required at the time of deploying the artifacts to a staging repository | ||
|
|
||
| The build output is installed to a temporary local maven repository. The build script will print the name of the | ||
| repository location at the end. This location will be required at the time of deploying the artifacts to a staging | ||
| repository | ||
|
|
||
| ### Tag the Release Candidate | ||
|
|
||
|
|
@@ -137,27 +148,28 @@ git tag 0.1.0-rc1 | |
| git push apache 0.1.0-rc1 | ||
| ``` | ||
|
|
||
| Note that pushing a release candidate tag will trigger a GitHub workflow that will build a Docker image and publish | ||
| it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet | ||
|
|
||
| ## Publishing the Release Candidate | ||
|
|
||
| This part of the process can mostly only be performed by a PMC member. | ||
|
|
||
| ### Create the Release Candidate Tarball | ||
|
|
||
| Run the create-tarball script on the release candidate tag (`0.1.0-rc1`) to create the source tarball and upload it to the dev subversion repository | ||
|
|
||
| ```shell | ||
| GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 0.1.0 1 | ||
| ``` | ||
|
|
||
| ### Publish the maven artifacts | ||
|
|
||
| #### Setup maven | ||
|
|
||
| ##### One time project setup | ||
|
|
||
| Setting up your project in the ASF Nexus Repository from here: https://infra.apache.org/publishing-maven-artifacts.html | ||
|
|
||
| ##### Release Manager Setup | ||
| Set up your development environment from here: https://infra.apache.org/publishing-maven-artifacts.html | ||
|
|
||
| Set up your development environment from here: https://infra.apache.org/publishing-maven-artifacts.html | ||
|
|
||
| ##### Build and publish a release candidate to nexus. | ||
| The script `publish-to-maven.sh` will publish the artifacts created by the `build-release-comet.sh` script. | ||
|
|
||
| The script `publish-to-maven.sh` will publish the artifacts created by the `build-release-comet.sh` script. | ||
| The artifacts will be signed using the gpg key of the release manager and uploaded to the maven staging repository. | ||
|
|
||
| Note: This script needs `xmllint` to be installed. On MacOS xmllint is available by default. | ||
|
|
@@ -183,7 +195,8 @@ GPG_KEY - GPG key used to sign release artifacts | |
| GPG_PASSPHRASE - Passphrase for GPG key | ||
| ``` | ||
|
|
||
| example | ||
| example | ||
|
|
||
| ```shell | ||
| /comet:$./dev/release/publish-to-maven.sh -u release_manager_asf_id -r /tmp/comet-staging-repo-VsYOX | ||
| ASF Password : | ||
|
|
@@ -193,23 +206,56 @@ Creating Nexus staging repository | |
| ... | ||
| ``` | ||
|
|
||
| In the Nexus repository UI (https://repository.apache.org/) locate and verify the artifacts in | ||
| In the Nexus repository UI (https://repository.apache.org/) locate and verify the artifacts in | ||
| staging (https://central.sonatype.org/publish/release/#locate-and-examine-your-staging-repository). | ||
|
|
||
| If the artifacts appear to be correct, then close and release the repository so it is made visible. | ||
| If the artifacts appear to be correct, then close and release the repository so it is made visible (this should | ||
| actually happen automatically when running the script). | ||
|
|
||
| ### Create the Release Candidate Tarball | ||
|
|
||
| Run the create-tarball script on the release candidate tag (`0.1.0-rc1`) to create the source tarball and upload it to | ||
| the dev subversion repository | ||
|
|
||
| ```shell | ||
| ./dev/release/create-tarball.sh 0.1.0 1 | ||
| ``` | ||
|
|
||
| This will generate an email template for starting the vote. | ||
|
|
||
| ### Start an Email Voting Thread | ||
|
|
||
| Send the email that is generated in the previous step to `[email protected]`. | ||
|
|
||
| ### Publish the Release Tarball | ||
| ## Publishing Binary Releases | ||
|
|
||
| Once the vote passes, we can publish the source and binary releases. | ||
|
|
||
| Once the vote passes, run the release-tarball script to move the tarball to the release subversion repository. | ||
| ### Publishing Source Tarball | ||
|
|
||
| Run the release-tarball script to move the tarball to the release subversion repository. | ||
|
|
||
| ```shell | ||
| ./dev/release/create-tarball.sh 0.1.0 1 | ||
| ./dev/release/release-tarball.sh 0.1.0 1 | ||
| ``` | ||
|
|
||
| ### Create a release in the GitHub repository | ||
|
|
||
| Go to https://github.com/apache/datafusion-comet/releases and create a release for the release tag, and paste the | ||
| changelog in the description. | ||
|
|
||
| ### Publishing Maven Artifacts | ||
|
|
||
| Promote the Maven artifacts from staging to production by visiting https://repository.apache.org/#stagingRepositories | ||
| and selecting the staging repository and then clicking the "release" button. | ||
|
|
||
| ### Publishing Crates | ||
|
|
||
| Publish the `datafusion-comet-spark-expr` crate to crates.io so that other Rust projects can leverage the | ||
| Spark-compatible operators and expressions outside of Spark. | ||
|
|
||
| ### Push a release tag to the repo | ||
|
|
||
| Push a release tag (`0.1.0`) to the `apache` repository. | ||
|
|
||
| ```shell | ||
|
|
@@ -219,6 +265,9 @@ git tag 0.1.0 | |
| git push apache 0.1.0 | ||
| ``` | ||
|
|
||
| Note that pushing a release tag will trigger a GitHub workflow that will build a Docker image and publish | ||
| it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet | ||
|
|
||
| Reply to the vote thread to close the vote and announce the release. | ||
|
|
||
| ## Post Release Admin | ||
|
|
@@ -260,20 +309,9 @@ svn ls https://dist.apache.org/repos/dist/release/datafusion | grep comet | |
| Delete a release: | ||
|
|
||
| ```shell | ||
| svn delete -m "delete old DataFusion Comet release" https://dist.apache.org/repos/dist/release/datafusion-comet/datafusion-comet-0.0.0 | ||
| svn delete -m "delete old DataFusion Comet release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-comet-0.0.0 | ||
| ``` | ||
|
|
||
| ## Publishing Binary Releases | ||
|
|
||
| ### Publishing JAR Files to Maven | ||
|
|
||
| Once the vote has passed, promote the staged release candidate to production in the Nexus repository UI (https://repository.apache.org/). | ||
|
|
||
| ### Publishing to crates.io | ||
|
|
||
| We may choose to publish the `datafusion-comet` to crates.io so that other Rust projects can leverage the | ||
| Spark-compatible operators and expressions outside of Spark. | ||
|
|
||
| ## Post Release Activities | ||
|
|
||
| Writing a blog post about the release is a great way to generate more interest in the project. We typically create a | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.