implement ContentServerDataStore for azure blob storage too #1234

xelat09 · 2025-01-05T20:36:48Z

Hi & Happy new year

Implement ContentServerDataStore for azure blob storage too, tests with azurite+azure blob are ok.
Incomplete range requests (Range: bytes=100- ) are not supported, hope that's ok

Regards

Acconut

Thank you very much for this PR, it's greatly appreciated! I had a brief first look and left a few comments. Let me know what you think!

pkg/azurestore/azureservice.go

…more headers and Range without rangeEnd

xelat09 · 2025-01-12T20:06:35Z

yeah, as suggested, azure supports all those headers too. please have a look, i gave it a new try. tests with azurite do work..

do you see anything else missing?

Acconut

Thank you very much for the quick updates! The handler is the request headers looks much better yet. Could you also have a look at the failing tests at https://github.com/tus/tusd/actions/runs/12736636713/job/35496766413?pr=1234? Talking about tests, it would be great to have some tests for ServeContent in azurestore as well. For inspiration you can use s3store's tests: https://github.com/tus/tusd/blob/main/pkg/s3store/serve_content_test.go

pkg/azurestore/azureservice.go

…ally called.

Acconut

Hello there, I just had another look over this PR. Apologies for missing it earlier. The PR seems mostly finished. I think the handling of Range header could be improved a bit and then there is still this comment: https://github.com/tus/tusd/pull/1234/files#r1905384839

Let me know if you are still interested in completing this PR. If not, I can consider pulling it across the finish line.

Acconut · 2025-05-21T09:32:17Z

pkg/azurestore/azureservice.go

+
+	if val := r.Header.Get("Range"); val != "" {
+		// zero value count indicates from the offset to the resource's end, suffix-length is not required
+		input.Range = azblob.HTTPRange{Offset: 0, Count: 0}


Does Azure support fetching multiple ranges in a single request? I.e. using Range: <unit>=<range-start>-<range-end>, …, <range-startN>-<range-endN> (from https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Range). S3 doesn't so the s3store doesn't support it. We can do the same here, but I was just wondering.

Hello, thank you for getting back :)
Response headers are forwarded now.
Regarding Range Header handling, what is missing, give me some hints?

Regarding multiple range in a single request, that is not supported. see also https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/[email protected]/internal/exported#HTTPRange

Regarding multiple range in a single request, that is not supported. see also https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/[email protected]/internal/exported#HTTPRange

Ok, I see. Can you please add a comment mentioning that Azure doesn't support multiple ranges and thus azurestore falls back to fetching the full resource, just like s3store does?

Response headers are forwarded now.

Great, thanks you!

Acconut · 2025-05-21T09:33:39Z

pkg/azurestore/azureservice.go

+		input.Range = azblob.HTTPRange{Offset: 0, Count: 0}
+		bytesEnd := 0
+		if _, err := fmt.Sscanf(val, "bytes=%d-%d", &input.Range.Offset, &bytesEnd); err != nil {
+			if _, err := fmt.Sscanf(val, "bytes=%d-", &input.Range.Offset); err != nil {


This doesn't seem to be handling requests where only the last bytes are requested using Range: <unit>=-<suffix-length>. Is that supported by Azure?

that isn't supported either. how should this be handled? throw an error/map to a different httprange?

It would be nice, if you could pull it across the finish line, I'm willing to do the testing :)

Ok, I see that the Azure SDK doesn't support this natively: https://github.com/Azure/azure-sdk-for-go/blob/f29a1d52c4e6894a536b0da18ac4399692e02c4c/sdk/storage/azblob/internal/exported/exported.go#L23

However, I think the corresponding AzUpload might have the total size of the object already in its info object. So we could calculate the corresponding offset based on the size as offset = size - suffix-length (and maybe -1 depending on how the bytes are counted, not sure).

I don't want to drag this on too long, but if there is an easy way to support fetching trailing bytes, I think it would be great to have it :)

quality-leftovers · 2026-01-14T11:43:19Z

pkg/azurestore/azureservice.go

+	}
+	result, err := blockBlob.BlobClient.DownloadStream(ctx, downloadOptions)
+	if err != nil {
+		return err


We've done some tests using this branch, so first I'd like to thank you for your work :).

The only issue we found is that when a client does a request with a range above the total content length he'll receive an 500 error response instead of 416 because of the error being returned here.

I think it would be better to return the 416 response here, including the content-range header (example value: bytes */64978)

if err != nil { // copy header "Content-Range", "X-Ms-Error-Code", "Date", "X-Ms-Request-Id"; Body empty; StatusCode+Status if present in error response var azureError *azcore.ResponseError if errors.As(err, &azureError) { if azureError.StatusCode == http.StatusRequestedRangeNotSatisfiable { if azureError.RawResponse != nil { if val := azureError.RawResponse.Header.Get("Content-Range"); val != "" { w.Header().Set("Content-Range", val) } if val := azureError.RawResponse.Header.Get("X-Ms-Error-Code"); val != "" { w.Header().Set("X-Ms-Error-Code", val) } if val := azureError.RawResponse.Header.Get("X-Ms-Request-Id"); val != "" { w.Header().Set("X-Ms-Request-Id", val) } if val := azureError.RawResponse.Header.Get("Date"); val != "" { w.Header().Set("Date", val) } } w.WriteHeader(azureError.StatusCode) return nil } }

Also missing http.StatusPreconditionFailed in case of If-None-Match header is used.

Might want to include http.StatusNotFound, though that should normally not be hit since GetInfo in unroute_handler will fail before reaching ServerContent. File might no longer exist later on though.

So updated condition would be:

// ... if http.StatusRequestedRangeNotSatisfiable == azureError.StatusCode || http.StatusPreconditionFailed == azureError.StatusCode || http.StatusNotFound == azureError.StatusCode { // ...

quality-leftovers · 2026-01-19T12:16:31Z

pkg/azurestore/azureservice.go

+		// Use 206 Partial Content for range requests
+		statusCode = http.StatusPartialContent
+	} else if result.ContentLength != nil && *result.ContentLength == 0 {
+		statusCode = http.StatusNoContent


When ETag is matched the response should be 304 StatusNotModified. The following allows doing so use the returned error header. Sadly the SDK does NOT expose the status code / actual typed error.

} else if result.ContentLength != nil && *result.ContentLength == 0 { if result.ErrorCode != nil && *result.ErrorCode == string(bloberror.ConditionNotMet) && downloadOptions != nil && downloadOptions.AccessConditions != nil && downloadOptions.AccessConditions.ModifiedAccessConditions != nil && downloadOptions.AccessConditions.ModifiedAccessConditions.IfNoneMatch != nil { // If the client sent an If-None-Match header and we get an X-Ms-Error-Code "ConditionNotMet", return 304 Not Modified statusCode = http.StatusNotModified } else { statusCode = http.StatusNoContent } }

Acconut requested changes Jan 7, 2025

View reviewed changes

pkg/azurestore/azureservice.go Outdated Show resolved Hide resolved

pkg/azurestore/azureservice.go Show resolved Hide resolved

pkg/azurestore/azureservice.go Outdated Show resolved Hide resolved

xelat09 closed this Jan 12, 2025

xelat09 force-pushed the main branch from 2c0e5df to a50bc42 Compare January 12, 2025 16:43

implement ContentServerDataStore for azure blob storage too, support …

607aafc

…more headers and Range without rangeEnd

xelat09 reopened this Jan 12, 2025

Acconut requested changes Jan 14, 2025

View reviewed changes

pkg/azurestore/azureservice.go Outdated Show resolved Hide resolved

pkg/azurestore/azureservice.go Outdated Show resolved Hide resolved

pkg/azurestore/azureservice.go Outdated Show resolved Hide resolved

xelat09 added 6 commits January 17, 2025 20:12

format source code

c0f70f3

return an error here so we notice it early if this method is accident…

e7d6c40

…ally called.

forward response headers

922e53b

fix test

874f79e

fix test by rerunning mockgen

7607b93

a few tests + fix, not sure if this is good enough..

5a7b497

xelat09 requested a review from Acconut January 23, 2025 12:57

xelat09 added 3 commits February 21, 2025 13:42

Merge branch 'tus:main' into main

c089bc2

Merge branch 'tus:main' into main

25280a5

Merge branch 'tus:main' into main

ba65513

Acconut requested changes May 21, 2025

View reviewed changes

xelat09 and others added 3 commits June 2, 2025 21:55

Merge branch 'tus:main' into main

e5dbeb1

just fixed docu

ea551c2

Merge branch 'tus:main' into main

1825452

quality-leftovers reviewed Jan 14, 2026

View reviewed changes

quality-leftovers reviewed Jan 19, 2026

View reviewed changes

implement ContentServerDataStore for azure blob storage too #1234

Are you sure you want to change the base?

implement ContentServerDataStore for azure blob storage too #1234

Uh oh!

Conversation

xelat09 commented Jan 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Acconut left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xelat09 commented Jan 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Acconut left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Acconut left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quality-leftovers Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xelat09 commented Jan 5, 2025 •

edited

Loading

xelat09 commented Jan 12, 2025 •

edited

Loading

quality-leftovers Jan 19, 2026 •

edited

Loading