Add PreSignURL method#255
Open
EngHabu wants to merge 36 commits into
Open
Conversation
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Closed
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Rename module to flyteorg/stow
Support ContentMD5 as an optional param
Signed-off-by: Sean Lin <sean@union.ai>
Upgrade dependencies
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Use latest google cloud lib
Signed-off-by: Babis Kiosidis <ckiosidis@gmail.com>
combine write and update to one request
Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com>
Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com>
Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com>
Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com>
Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com>
Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com>
Signed-off-by: Gopal K. Vashishtha <15335863+gvashishtha@users.noreply.github.com>
* adding baseUrl to get signed URl Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com> * default correctly Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com> * adding base url in Containers Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com> * renaming Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com> * Revert "renaming" This reverts commit 05c7cb3. Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com> * renaming to non-exported function Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com> --------- Signed-off-by: Gopal K. Vashishtha <gvashishtha@anduril.com>
* A refactor of the `azure` module to support authenticating via either Azure AD or shared keys. This required a larger-than-expected series of changes because the Azure SDK now very different than the one stow used before. This is an early commit to get changes out there for discussion. Included here is: - An upgrade of the Azure SDK used for the project (needed to support AD auth). This, unfortunately also brought in a whole series if indirect upgrades so testing will be needed. - Support for authenticating via either Azure AD or Shared keys - Removal of several azure config values (ConfigAPIVersion, ConfigUseHTTPS) and addition of ConfigUploadConcurrency. - Removal of the multi-part upload code for Azure, this is now included in the SDK - Addition of new presigned url tests into the general stow test/ module. - Addition of a new terraform module to stand up an azure storage account for testing. Signed-off-by: Terence Kent <terence.kent@mcg.com> * fix misspellings (doh) Signed-off-by: Terence Kent <terence.kent@mcg.com> * Update azure/config.go Accepting len(var)==0 vs var=="" Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: Terence Kent <terence.kent@mcg.com> * Update azure/config.go Accepting improved error message Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: Terence Kent <terence.kent@mcg.com> * Changes before moving PR from draft. No impact to tests or functionality at this point: - Removed all inline context.Background() calls when using Azure SDK methods. Now set an explicit `ctx` variable at the top of each function. - Added comments around the clock skew code, and moved all clock skews to 15 minutes (the azure-recommended value). Also left a comment explaining the decision. - Made the use of removed config values an error state for validation. Added a test to ensure this happens. - Moved some errant logging in the `checkMetadata` test method to only show up in inequality, to keep test output clean. Signed-off-by: Terence Kent <terence.kent@mcg.com> * Missed that the previous version of the Azure implementation silently ignored failures due to creating an already-existing container! The code from the old SDK to ignore "already exists" was preserved, but not working with the new SDK. - Updated the check to work with the new SDK. - Added a general test to confirm this behavior. Note: It's debatable if this should be the default behavior for all stow implementations. However, it seems like whatever the desired behavior is, it should be consistent across all implementations. Signed-off-by: Terence Kent <terence.kent@mcg.com> --------- Signed-off-by: Terence Kent <terence.kent@mcg.com> Co-authored-by: Terence Kent <terence.kent@mcg.com> Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
* Add extra header to signed url Signed-off-by: Kevin Su <pingsutw@gmail.com> * make metadata optional Signed-off-by: Kevin Su <pingsutw@gmail.com> * nit Signed-off-by: Kevin Su <pingsutw@gmail.com> * Add metadata to PresignRequestParams Signed-off-by: Kevin Su <pingsutw@gmail.com> * lint Signed-off-by: Kevin Su <pingsutw@gmail.com> * lint Signed-off-by: Kevin Su <pingsutw@gmail.com> --------- Signed-off-by: Kevin Su <pingsutw@gmail.com>
Signed-off-by: Kevin Su <pingsutw@gmail.com>
Signed-off-by: Kevin Su <pingsutw@gmail.com>
* Update azure sdk / mod tidy - Resolves vuln CVE-2024-35255 Signed-off-by: ddl-ebrown <ethan.brown@dominodatalab.com> * Fix test compile errors - These were caught by go vet. How does CI even run? Signed-off-by: ddl-ebrown <ethan.brown@dominodatalab.com> --------- Signed-off-by: ddl-ebrown <ethan.brown@dominodatalab.com>
Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
* Introduce new config disable_force_path_style for S3 Signed-off-by: Ranjit Kumar Parvathaneni <prk.bits@gmail.com> Signed-off-by: ranjit-parva <ranjit.parva@mistral.ai> * Adding debug logging to s3 client Signed-off-by: ranjit-parva <ranjit.parva@mistral.ai> * Avoiding any additional config Signed-off-by: ranjit-parva <ranjit.parva@mistral.ai> * Based on config Signed-off-by: ranjit-parva <ranjit.parva@mistral.ai> --------- Signed-off-by: Ranjit Kumar Parvathaneni <prk.bits@gmail.com> Signed-off-by: ranjit-parva <ranjit.parva@mistral.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add PreSignURL API on container interface and implement it for S3, Azure and Google.
fixes #213