add azure storage support #923
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Keeping in mind that I'm not anywhere near proficient in rust, I'm starting to look into arroyo as a possible streaming engine, but noticed it lacks azure support. This PR attempts to fix that.
Should be fairly self explanatory as the
object_storecrate includes an azure builder. Since Azure has a ton of different authentication methods, I'm opting to only offer it through env vars parsed automatically by theobject_storebuilder (see below)How I tested these changes
Unit tests, compiled image and ran locally against an azure account, tested both writing to azure with the filesystem sink, as well as the backend checkpointing.
To note
The delta sink is untested; I expect it would work based on the underlying filesystem implementation, but can't be sure as I don't have a delta setup. The docs on
storage_provider.get_backing_store()confused me a little, as typically azure would contain both the container and the storage account, but theobject_storecrate also has a few examples that include a URI following the format I passed.