403 Error when reading S3 URLs with special characters

[TomAugspurger]

This snippet gets a 403 error when reading an S3 URL with = in the object key:

import boto3
import os
import kvikio
import rmm

client = boto3.client(
    's3',
    aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'],
    aws_session_token=os.environ['AWS_SESSION_TOKEN'],
    region_name=os.environ['AWS_DEFAULT_REGION']
)

bucket = ...

for key in ["test/date-2025-09-15", "test/date=2025-09-15"]:
    client.put_object(Bucket=bucket, Key=key, Body=b'Hello, world!')
    client.head_object(Bucket=bucket, Key=key)

    print("boto3 done", key)

    buf = rmm.DeviceBuffer(size=13)
    f = kvikio.RemoteFile.open_s3(bucket, key)
    f.read(buf)

    print("kvikio done", key)

Which outputs

boto3 done test/date-2025-09-15
kvikio done test/date-2025-09-15
boto3 done test/date=2025-09-15
Traceback (most recent call last):
  File "/home/coder/kvikio/debug.py", line 23, in <module>
    f = kvikio.RemoteFile.open_s3(bucket, key)
  File "/home/coder/kvikio/python/kvikio/kvikio/remote_file.py", line 146, in open_s3
    _get_remote_module().RemoteFile.open_s3(bucket_name, object_name, nbytes)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "kvikio/_lib/remote_handle.pyx", line 178, in kvikio._lib.remote_handle.RemoteFile.open_s3
  File "kvikio/_lib/remote_handle.pyx", line 135, in kvikio._lib.remote_handle.RemoteFile._from_endpoint
RuntimeError: KvikIO failure at: /home/coder/kvikio/cpp/src/shim/libcurl.cpp:183: curl_easy_perform() error (The requested URL returned error: 403)

This is similar to curl/curl#13754, which is fixed in recent versions of curl. It's possible that this is fixed if we use a newer version of libcurl.

[TomAugspurger]

Some more debugging: Add

import logging

boto3.set_stream_logger('', logging.DEBUG)

to that script and compare the canonical request from boto3 and kvikio (with tokens redacted). boto3 outputs:

HEAD
/test/date%3D2025-09-15

host:kvikiobench-33622.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20250916T174425Z
x-amz-security-token:[...]

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

while kvikio outputs

HEAD
/test/date=2025-09-15

host:kvikiobench-33622.s3.us-east-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20250916T174425Z
x-amz-security-token:[...]

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

so the URL in the request is the only thing that differs (aside from x-amz-date, which I think depends on the current time). boto3 urllencodes the url, while kvikio doesn't.

[TomAugspurger]

As a workaround for now, users can quote the URL themselves. In python, urllib.parse.quote handles that:

for key in ["test/date-2025-09-15", "test/date=2025-09-15"]:
    client.put_object(Bucket=bucket, Key=key, Body=b'Hello, world!')
    client.head_object(Bucket=bucket, Key=key)

    print("boto3 done", key)

    buf = rmm.DeviceBuffer(size=13)
    f = kvikio.RemoteFile.open_s3(bucket, urllib.parse.quote(key))
    f.read(buf)

    print("kvikio done", key)

[kingcrimsontianyu]

To summarize my findings, using curl 8.15.0 compiled from source:

S3 type	Use %3D in path	Use = in path
Private (require credentials)	✅	❌
Public	✅	✅
Presigned	✅	✅

So the problem described in curl/curl#13754 seems to still exist.

[TomAugspurger]

@kingcrimsontianyu do you have a sense for who is responsible for quoting the URL when using libcurl and CURLOPT_AWS_SIGV4? I can't tell whether CURLOPT_URL needs to be encoded by the user / library, or whether curl handles that for you.

[kingcrimsontianyu]

I didn't find any encoding requirement from CURLOPT_AWS_SIGV4's documentation, so I'm inclined to think that this is an unsolved issue in libcurl (at least a missing documentation).

[kingcrimsontianyu]

After all, I believe we could (if agreed upon by the team) provide simple C++ workaround specific to the S3 private access, such as unconditionally encoding the path component, or regex-replacing the sub-delimiters with their percent-encoded strings.

[TomAugspurger]

provide simple C++ workaround specific to the S3 private access, such as unconditionally encoding the path component, or regex-replacing the sub-delimiters with their percent-encoded strings.

IMO, that's worth doing. We found that cudf.read_parquet("s3://<bucket>/<key-with-=>.parquet") currently fails regardless of whether the user calls urllib.parse.quote on the object key. If you don't quote it, you hit this issue in kvikio. If you do quote it, the read_parquet fails somewhere in pyarrow since its used to read some things before kvikio, and that ends up double-encoding the URL :/

[kingcrimsontianyu]

Thanks for the info. I'll come up with a fix early next week (for 25.12).