Skip to content

[SPARK-31167][BUILD] Refactor how we track Python test/build dependencies #27928

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nchammas wants to merge 18 commits into from
Closed

[SPARK-31167][BUILD] Refactor how we track Python test/build dependencies #27928

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
7a20608
missing test dependencies
nchammas Mar 14, 2020
7273851
add docs requirements file
nchammas Mar 16, 2020
0a9cbbc
reference new docs requirements file
nchammas Mar 16, 2020
9c3a7af
pin doc gem requirements in github workflow
nchammas Mar 16, 2020
de9fbbd
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
nchammas Mar 17, 2020
1086d81
merge docs reqs into general dev reqs
nchammas Mar 17, 2020
cd05e25
merged reqs
nchammas Mar 17, 2020
4326933
open python reqs to latest patch version
nchammas Mar 20, 2020
75682a6
Unidecode 04 -> 4
nchammas Mar 20, 2020
bb83262
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
nchammas Apr 1, 2020
839669d
add requirements and pinned requirements
nchammas Apr 1, 2020
473c85d
requirements.txt -> requirements-pinned.txt
nchammas Apr 1, 2020
4b540a9
run pip-tools from within dev/
nchammas Apr 1, 2020
b088d15
add dev readme
nchammas Apr 1, 2020
9704c60
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
nchammas Apr 17, 2020
7df0040
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
nchammas May 4, 2020
bae2c0c
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
nchammas Jun 1, 2020
e21d97d
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
nchammas Jul 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ jobs:
run: ./dev/lint-java
- name: Python
run: |
pip install flake8 sphinx numpy
pip install -r ./dev/requirements.txt
./dev/lint-python
- name: License
run: ./dev/check-license
Expand Down Expand Up @@ -147,8 +147,8 @@ jobs:
sudo apt-get install -y r-base r-base-dev libcurl4-openssl-dev pandoc
- name: Install packages
run: |
pip install sphinx mkdocs numpy
gem install jekyll jekyll-redirect-from rouge
pip install -r ./dev/requirements.txt
gem install jekyll:4.0.0 jekyll-redirect-from:0.16.0 rouge:3.15.0
sudo Rscript -e "install.packages(c('curl', 'xml2', 'httr', 'devtools', 'testthat', 'knitr', 'rmarkdown', 'roxygen2', 'e1071', 'survival'), repos='https://cloud.r-project.org/')"
- name: Run jekyll build
run: |
Expand Down
3 changes: 3 additions & 0 deletions dev/create-release/do-release-docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@

set -e
SELF=$(cd $(dirname $0) && pwd)
SPARK_ROOT="$SELF/../.."
. "$SELF/release-util.sh"

function usage {
Expand Down Expand Up @@ -91,6 +92,8 @@ for f in "$SELF"/*; do
fi
done

cp "$SPARK_ROOT/dev/requirements.txt" "$WORKDIR/dev-requirements.txt"

GPG_KEY_FILE="$WORKDIR/gpg.key"
fcreate_secure "$GPG_KEY_FILE"
$GPG --export-secret-key --armor "$GPG_KEY" > "$GPG_KEY_FILE"
Expand Down
4 changes: 2 additions & 2 deletions dev/create-release/spark-rm/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ ENV DEBCONF_NONINTERACTIVE_SEEN true
# These arguments are just for reuse and not really meant to be customized.
ARG APT_INSTALL="apt-get install --no-install-recommends -y"

ARG PIP_PKGS="sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1"
COPY ./dev-requirements.txt /dev-requirements.txt
ARG GEM_PKGS="jekyll:4.0.0 jekyll-redirect-from:0.16.0 rouge:3.15.0"

# Install extra needed repos and refresh.
Expand Down Expand Up @@ -61,7 +61,7 @@ RUN pyenv global 3.7.6
RUN python --version
RUN pip install --upgrade pip
RUN pip --version
RUN pip install $PIP_PKGS
RUN pip install -r dev-requirements.txt

ENV PATH "$PATH:/root/.rbenv/bin:/root/.rbenv/shims"
RUN curl -fsSL https://github.com/rbenv/rbenv-installer/raw/108c12307621a0aa06f19799641848dde1987deb/bin/rbenv-installer | bash
Expand Down
9 changes: 7 additions & 2 deletions dev/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
flake8==3.5.0
flake8==3.7.*
Copy link
Contributor Author

@nchammas nchammas Mar 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I still think we should pin every version here, perhaps this approach is a compromise we can agree on.

==3.7.* means pip will install the latest bugfix release on 3.7. If you already have any 3.7 version installed, even if it's not the latest one, pip will consider the requirement satisfied and won't do anything. To force pip to upgrade to the latest bugfix release of 3.7 when you already have a compatible version installed, simply request it via pip install --upgrade.

Copy link
Member

@HyukjinKwon HyukjinKwon Mar 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can assume these dependencies follow SemVer, it we should better use wildcards on minor versions ...

Copy link
Contributor Author

@nchammas nchammas Mar 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mentioned it elsewhere but I'll mention it again here: Linters like flake8 and pycodestyle introduce new checks in minor/feature releases. There is very high chance that every new check they introduce will flag new problems and fail the build.

In fact, we saw exactly that behavior with pydocstyle just before we removed it. And I experienced this with pycodestyle in Flintrock before pinning the version.

I don't understand the point of waiting for the build to break before pinning or severely limiting the versions for libraries like these.

jira==1.0.3
pycodestyle==2.5.*
PyGithub==1.26.0
Unidecode==0.04.19
sphinx

# docs
mkdocs==1.0.*
numpy==1.18.*
sphinx==2.3.*
2 changes: 1 addition & 1 deletion docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
To generate API docs for any language, you'll need to install these libraries:

```sh
pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
pip install -r ./dev/requirements.txt
```

## Generating the Documentation HTML
Expand Down