GH-32123: [R] Expose azure blob filesystem

[marberts]

Rationale for this change

This PR adds support for Azure. The Arrow R package already has support for AWS and GCS, and the Arrow C++ library has had support for Azure for a couple years now. Support for Azure is already available in pyarrow.

This would close #32123.

What changes are included in this PR?

• A new class AzureFileSystem that's analogous to S3FileSystem/GcsFileSystem, along with a helper function az_container() that's analogous to s3_bucket()/gcs_bucket().

• Updates to src/filesystem.cpp to interact with the machinery in arrow/filesystem/azurefs.h.

• Updates to the configuration and build scripts to support building with support for Azure.

• Updates to the vignettes on cloud storage, installation, and developer setup.

Are these changes tested?

Yes. See tests/testthat/test-azure.R.

Are there any user-facing changes?

Yes. There is a new function az_container(), serving the analogous role to s3_bucket()/gcs_bucket(), along with an R6 class AzureFileSystem, again serving the same role as S3FileSystem and GcsFileSystem. There is also a function arrow_with_azure() to indicate if Arrow was built with support for Azure.

• GitHub Issue: [R] Expose Azure Blob Storage filesystem #32123

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #32123 has been automatically assigned in GitHub to PR creator.

[marberts]

The CI is failing because the Azure C++ SDK depends on libxml2.

• Update tools/nixlibs.R to check for libxml2 in with_cloud_support(), turning off the Azure feature flag if libxml2 isn't found.
  • What about configure and configure.win?
  • Is with_cloud_support() the only function that needs changing in tools/nixlibs.R?
• Update the install.Rmd and developers/setup.Rmd vignettes to note that libxml2 is a dependency of Azure (same as with curl and openssl).
• Update DESCRIPTION to include libxml2 in SystemRequirements.

[marberts]

It looks like the failing CI job is from a warning in R CMD check about a non-API call to a C function, due to all the changes in R-devel. I don't think this is due to our PR.

At this point, I'm fairly confident that we've implemented the Azure features correctly. The only thing I'm not sure about is the setup in configure, configure.win, and tools/nixlibs.R. The Azure C++ SDK used by Arrow requires libxml2, which is a new system dependency for this package. I don't fully understand how these configuration scripts work, so I wouldn't be surprised if they're not updated correctly to reflect the new system dependency.

@jonkeane, @thisisnic, @assignUser Please let me or @Collinbrown95 know if there are any changes needed to this PR.

[thisisnic]

Thanks for the PR @marberts - if you rebase from the main branch now, the non-API call stuff should be resolved.

Your best bet for the configure etc stuff is looking at the history of that file or older PRs to see how this kind of thing was handled there.

This PR where the GCS bindings were added might help: #13404, but let us know if you have any questions!

[thisisnic]

We'll also need additional CI jobs to test this - the PR I linked to above contains examples of what we added there.

[marberts]

Great, thanks @thisisnic!

[marberts]

To do list.

• Rebase against main to fix failing CI job.
• Review configure, configure.win and tools/nixlibs.R against changes in ARROW-16510: [R] Add bindings for GCS filesystem #13404. It's possible this will reveal some other changes that are needed.
• Add CI job to test changes. See again ARROW-16510: [R] Add bindings for GCS filesystem #13404.

[Collinbrown95]

@github-actions crossbow submit -g r

[github-actions]

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/23576717645

[thisisnic]

@github-actions crossbow submit -g r

[github-actions]

Revision: 4da7280

Submitted crossbow builds: ursacomputing/crossbow @ actions-92325d6543

Task	Status
r-binary-packages
r-recheck-most
test-r-alpine-linux-cran
test-r-arrow-backwards-compatibility
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-fedora-clang
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-offline-maximal
test-r-ubuntu-22.04
test-r-versions

[marberts]

@github-actions crossbow submit -g r

[github-actions]

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/23698292050

[thisisnic]

Some of those CI failures are from things which have been fixed on main - you'll need to rebase your branch.

[marberts]

It seems I made a mess of the rebase, although I'm not entirely sure how 😞.

[Collinbrown95]

Hi @thisisnic (CC: @marberts ),

Here's a quick update of where things are at.

Right now the AMD64 Windows C++ RTools 40 ucrt64 job is failing due to the various Azure sdk dependencies (azure-core-cpp, azure-identity-cpp, azure-storage-blobs-cpp, azure-storage-common-cpp, and azure-storage-files-datalake-cpp). Specifically, these packages do not seem to be available on MSYS2 repositories (i.e., we can't pacman install them), and we haven't been able to install these packages with vcpkg using either x64-mingw-dynamic or x64-mingw-static triplets.

Our understanding is that the Arrow C++ code in the Arrow R package must be built with the MinGW toolchain (due to Rtools), so the Azure SDKs we depend on must also be built with MinGW. To the best of our understanding, the only way to do this is to build the azure-sdk-for-cpp components we need from source with MinGW.

We were able to build azure-core-cpp, azure-storage-blobs-cpp, azure-storage-common-cpp, and azure-storage-files-datalake-cpp successfully with MinGW. Unfortunately, we were not able to get azure-identity-cpp to build as it seems to depend heavily on the Windows Implementation Library (WIL) which lacks stable support for MinGW.

What this means for this PR

Our understanding is that this azure-identity-cpp limitation means that either (1) certain Azure Identity features won't work on the Windows build of Arrow R or (2) Azure should be disabled on the Windows build of Arrow R to avoid confusion.

Before proceeding on this, we wanted to check if this limitation is a showstopper for this PR, and if not, do you want us to proceed by modifying the Windows build to ignore Azure identity, or should we just set -DARROW_AZURE=OFF on the Windows build?

Alternatively, if you think there's a way to get azure-identity-cpp build working with MinGW, we would be happy to accept any advice on this.

Thanks in advance!

[thisisnic]

Thanks for persisting with this, I see a lot of rounds of iteration there!

It's a shame it's such a pain to support on Windows, but I think there's value including it even if it can only work on MacOS and Linux.

I'm not an expert in this stuff, and so I did a 🤖 analysis, which I'll paste below so I can be transparent about the underlying assumptions on which I'm basing my opinions, in case any are incorrect.

tl;dr:

should we just set -DARROW_AZURE=OFF on the Windows build?

Yes, I think so, but would like to hear from @jonkeane on this as well.

---

The Core Problem

R packages on Windows must use MinGW (via Rtools) - this is an R/CRAN requirement. However, azure-identity-cpp depends on WIL (Windows Implementation Library), which explicitly does not support MinGW due to MSVC-specific compiler features.

PyArrow works because it uses Visual Studio + vcpkg, not MinGW.

What azure-identity-cpp provides

Looking at azurefs.cc, the auth methods split into two groups:

Don't require azure-identity-cpp (only need azure-storage-*-cpp):

• Anonymous access
• Account key auth (uses StorageSharedKeyCredential)
• SAS token auth (just appends token to URL)

Require azure-identity-cpp (use Azure::Identity::* classes):

• DefaultAzureCredential, ClientSecretCredential, ManagedIdentityCredential, AzureCliCredential, WorkloadIdentityCredential, EnvironmentCredential

Current Arrow C++ constraints

• FindAzure.cmake requires all 5 Azure SDK packages including azure-identity-cpp
• AZURE_SDK_LINK_LIBRARIES always includes Azure::azure-identity
• azurefs.cc unconditionally includes <azure/identity.hpp>

Options

1. -DARROW_AZURE=OFF on Windows R builds - simplest, no Azure on Windows R at all

2. Make azure-identity-cpp optional in Arrow C++ - would require:

   • CMake changes to make azure-identity-cpp optional in FindAzure.cmake
   • Conditional linking in ThirdpartyToolchain.cmake
   • #ifdef guards or Status::NotImplemented() returns for identity-based auth in azurefs.cc
   • This would allow account key + SAS auth on Windows R, just not the token-based methods
   • Moderate complexity (~100-200 lines), but needs C++ maintainer buy-in

3. Wait for MinGW/WIL compatibility - unlikely given WIL maintainers closed the issue citing lack of expertise