Skip to content

chore: Upgrade to later version of gcp-uploader and protobuf #1741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lqiu96 merged 20 commits into from
Jan 19, 2023
Merged

chore: Upgrade to later version of gcp-uploader and protobuf #1741

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
8c2bf24
chore: Upgrade to later version of gcp-uploader and protobuf
lqiu96 Dec 20, 2022
2609959
chore: Add secretstorage version
lqiu96 Dec 21, 2022
a78206c
chore: Use python 3.9.13
lqiu96 Dec 21, 2022
e0b8be5
Merge branch 'master' into master-update_synthtool_requirements
lqiu96 Dec 21, 2022
234f014
chore: Unpin all python versions
lqiu96 Dec 21, 2022
a56d844
chore: Rerun pipcompile tool
lqiu96 Dec 21, 2022
07f4ecb
chore: Rerun pipcompile tool with --allow-unsafe
lqiu96 Dec 21, 2022
09fa74d
chore: Add --require-hashes
lqiu96 Dec 21, 2022
bd22b68
chore: Update requirements file
lqiu96 Dec 21, 2022
9fe9009
chore: Remove --require-hashes
lqiu96 Dec 21, 2022
629f92d
chore: Print python version
lqiu96 Dec 29, 2022
90bff51
chore: Add new generated hashes for requirements.txt
lqiu96 Dec 29, 2022
20e588d
chore: Remove python version check in cloud build
lqiu96 Dec 29, 2022
477f6ca
chore: Allow synthtool to update python dependencies
lqiu96 Dec 29, 2022
9bdb449
chore: Add typing-extensions into requirements
lqiu96 Jan 3, 2023
9f2e13f
chore: Unpin docker requirements.in file
lqiu96 Jan 4, 2023
30ab721
chore: Add java synthtool validation
lqiu96 Jan 12, 2023
0c657ea
chore: Add check to import java synthtool python package
lqiu96 Jan 12, 2023
f61cfeb
Merge branch 'master' into master-update_synthtool_requirements
lqiu96 Jan 12, 2023
e0e5463
Merge branch 'master' into master-update_synthtool_requirements
lqiu96 Jan 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 16 additions & 5 deletions docker/owlbot/java/container_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,26 @@
# See the License for the specific language governing permissions and
# limitations under the License.

schemaVersion: 1.0.0
schemaVersion: 2.0.0
commandTests:
- name: "version"
command: ["java", "-version"]
command: "java"
args: ["-version"]
# java -version outputs to stderr...
expectedError: ["(java|openjdk) version \"1.8.*\""]
- name: "formatter"
command: ["java", "-jar", "/owlbot/google-java-format.jar", "--version"]
command: "java"
args: ["-jar", "/owlbot/google-java-format.jar", "--version"]
expectedError: ["google-java-format: Version 1.7"]
- name: "python"
command: ["python", "--version"]
expectedOutput: ["Python 3.6.1"]
command: "python"
args: ["--version"]
expectedOutput: ["Python 3.9.13"]
Copy link
Member

@suztomo suztomo Jan 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lqiu96 Would you add an assertion that this python module synthtool.languages.java is available? I think it would look like this:

python3 -c 'import synthtool.languages.java; print("import successs")'

An example failure would look like this:

suztomo@suztomo:~/java-storage$ docker run --user "$(id -u):$(id -g)" --entrypoint /bin/bash --rm -it  a8e0ab6765a4
I have no name!@752c3d65c2dd:/workspace$ python3 -c 'import synthtool.languages.java; print("import successs")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'synthtool.languages'
I have no name!@752c3d65c2dd:/workspace$

It gives more confidence that a change for OwlBot Postprocessor works fine in owlbot.py (example location)

Copy link
Member Author

@lqiu96 lqiu96 Jan 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I can add that validation in

Copy link
Member Author

@lqiu96 lqiu96 Jan 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the new validation in as java synthtool validation which runs the commands above.

FYI, couldn't find the docs for schemaVersion 1.0.0 and the docs on https://github.com/GoogleContainerTools/container-structure-test were showing docs for 2.0.0, so I used that for local testing

Copy link
Member

@suztomo suztomo Jan 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you.

- name: "java synthtool validation"
command: "python"
# Use YAML List Style for this command
# Having to use backslash for quotes (\") was resulting in an invalid test
args:
- -c
- 'import synthtool.languages.java; print("import success")'
expectedOutput: ["import success"]
22 changes: 11 additions & 11 deletions docker/owlbot/java/src/requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
attrs==22.1.0
click==7.1.2
jinja2==3.0.3
lxml==4.6.5
typing==3.7.4.3
markupsafe==2.0.1
colorlog==6.7.0
protobuf==3.19.5
watchdog==2.1.9
requests==2.27.1
pyyaml==6.0
attrs
click
jinja2
lxml
typing
markupsafe
colorlog
protobuf
watchdog
requests
pyyaml
4 changes: 2 additions & 2 deletions docker/owlbot/java/src/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
# pip-compile --allow-unsafe --generate-hashes requirements.in
#
Expand Down
2 changes: 1 addition & 1 deletion renovate.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"extends": [
"schedule:weekly"
],
"ignorePaths": ["*/**", "synthtool/gcp/templates/java_library/.kokoro/requirements.txt", "synthtool/docker/owlbot/java/src/requirements.txt"],
"ignorePaths": ["*/**"],
"internalChecksFilter": "strict",
"stabilityDays": 30,
"timezone": "America/Los_Angeles",
Expand Down
40 changes: 6 additions & 34 deletions synthtool/gcp/templates/java_library/.kokoro/requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,34 +1,6 @@
gcp-docuploader==0.6.3
google-crc32c==1.3.0
googleapis-common-protos==1.56.3
gcp-releasetool==1.9.1
cryptography==38.0.3
cachetools==4.2.4
cffi==1.15.1
jeepney==0.7.1
jinja2==3.0.3
markupsafe==2.0.1
keyring==23.4.1
packaging==21.3
protobuf==3.19.5
pyjwt==2.4.0
pyparsing==3.0.9
pycparser==2.21
pyperclip==1.8.2
python-dateutil==2.8.2
requests==2.27.1
certifi==2022.12.7
importlib-metadata==4.8.3
zipp==3.6.0
google_api_core==2.8.2
google-cloud-storage==2.0.0
google-resumable-media==2.3.3
google-cloud-core==2.3.1
typing-extensions==4.1.1
urllib3==1.26.12
zipp==3.6.0
rsa==4.9
six==1.16.0
attrs==22.1.0
google-auth==2.14.1
idna==3.4
gcp-docuploader
gcp-releasetool
wheel
setuptools
typing-extensions
click<8.1.0
Loading