LargeFileUploadTask does not add header Content-Length (always?)

[kekolab]

I couldn't manage to have the LargeFileUploadTask to work.
While debugging I noticed that LargeFileUploadTask.upload() does not add the Content-Length header to the request, as specified here

Expected behavior

Uploading a file

Actual behavior

Exception

Steps to reproduce the behavior

This is the code I use:

public class LargeUploadFileTaskTest {
    @Test
    public void test() throws IOException, IllegalAccessException, InvocationTargetException, NoSuchMethodException,
            InterruptedException {
        OkHttpClient.Builder clientBuilder = new OkHttpClient().newBuilder();
        String name = "theName";
        byte[] content = new byte[5];
        new Random().nextBytes(content);
        InputStream stream = new ByteArrayInputStream(content);
        long length = stream.available();
        ApacheHttpTransport transport = new ApacheHttpTransport(
                ApacheHttpTransport.newDefaultHttpClientBuilder().build());
        Credential credential = new Credential.Builder(BearerToken.authorizationHeaderAccessMethod())
                .setClientAuthentication(new ClientParametersAuthentication(CLIENT_ID, null))
                .setJsonFactory(new GsonFactory())
                .setTokenServerEncodedUrl(TOKEN_URL)
                .setTransport(transport)
                .build()
                .setAccessToken(ACCESS_TOKEN)
                .setExpirationTimeMilliseconds(EXPIRATION_MILLIS)
                .setRefreshToken(REFRESH_TOKEN);
        AuthenticationProvider authenticationProvider = new GoogleCredentialAuthenticationProvider(credential);
        GraphServiceClient client = new GraphServiceClient(authenticationProvider, clientBuilder.build());

        DriveItemUploadableProperties uploadableProperties = new DriveItemUploadableProperties();
        uploadableProperties.setName(name);
        uploadableProperties.setFileSize(length);
        uploadableProperties.getAdditionalData().put("@microsoft.graph.conflictBehavior", "fail");

        CreateUploadSessionPostRequestBody uploadSessionPostRequestBody = new CreateUploadSessionPostRequestBody();
        uploadSessionPostRequestBody.setItem(uploadableProperties);
        String driveId = client.drives().get().getValue().get(0).getId();
        UploadSession uploadSession = client.drives().byDriveId(driveId).items()
                .byDriveItemId("root:/" + name + ":")
                .createUploadSession()
                .post(uploadSessionPostRequestBody);

        LargeFileUploadTask<DriveItem> largeFileUploadTask = new LargeFileUploadTask<>(
                client.getRequestAdapter(), uploadSession,
                stream, length, DriveItem::createFromDiscriminatorValue);
        UploadResult<DriveItem> uploadResult = largeFileUploadTask.upload();
        client.drives().byDriveId(driveId).items().byDriveItemId(uploadResult.itemResponse.getId()).delete();
    }

    public static class GoogleCredentialAuthenticationProvider implements AuthenticationProvider {
        private Credential credential;

        public GoogleCredentialAuthenticationProvider(Credential credential) {
            this.credential = credential;
        }

        @Override
        public void authenticateRequest(RequestInformation request,
                Map<String, Object> additionalAuthenticationContext) {
            if (Instant.ofEpochMilli(this.credential.getExpirationTimeMilliseconds())
                    .isBefore(Instant.now().plusMillis(60000))) {
                try {
                    credential.refreshToken();
                } catch (IOException e) {
                    throw new RuntimeException("Cannot refresh token", e);
                }
            }
            request.headers.add("Authorization", "Bearer " + credential.getAccessToken());
        }
    }
}

receiving this exception:

com.microsoft.kiota.ApiException: generalException
        at com.microsoft.kiota.ApiExceptionBuilder.withMessage(ApiExceptionBuilder.java:45)
        at com.microsoft.graph.core.requests.upload.UploadResponseHandler.handleResponse(UploadResponseHandler.java:61)
        at com.microsoft.graph.core.requests.upload.UploadSliceRequestBuilder.put(UploadSliceRequestBuilder.java:69)
        at com.microsoft.graph.core.tasks.LargeFileUploadTask.uploadSlice(LargeFileUploadTask.java:207)
        at com.microsoft.graph.core.tasks.LargeFileUploadTask.upload(LargeFileUploadTask.java:131)
        at com.microsoft.graph.core.tasks.LargeFileUploadTask.upload(LargeFileUploadTask.java:111)
[...]

To understand why, I wrote this Interceptor to log the requests and responses:

public static class OkHttpRequestResponseLogger implements Interceptor {
    private boolean logResponse;
    private boolean logRequest;

    public OkHttpRequestResponseLogger(boolean logRequest, boolean logResponse) {
        this.logRequest = logRequest;
        this.logResponse = logResponse;
    }

    private void logHeaders(StringBuilder logMessage, Headers headers) {
        for (String name : headers.names())
            logMessage.append(name + " : " + headers.get(name) + System.lineSeparator());
    }

    private boolean isContentTypeText(MediaType contentType) {
        if (contentType == null)
            return false;
        String ct = contentType.toString();
        if (ct == null)
            return false;

            ct = ct.toLowerCase();
            return ct.contains("json") || ct.contains("text") || ct.contains("xml");
        }

        private byte[] logRequestBody(StringBuilder logMessage, RequestBody body) throws IOException {
            if (body == null)
                return null;
            Buffer sink = new Buffer();
            body.writeTo(sink);
            byte[] bytes = sink.readByteArray();
            logByteArray(logMessage, bytes, body.contentType());
            return bytes;
        }

        private void logByteArray(StringBuilder logMessage, byte[] bytes, MediaType contentType) {
            if (isContentTypeText(contentType)) {
                logMessage.append(new String(bytes));
            } else {
                logMessage.append(Hex.encodeHexString(bytes));
            }
            logMessage.append(System.lineSeparator());
        }

        private Request logRequest(StringBuilder logMessage, Request request) throws IOException {
            logMessage.append(request.method() + " " + request.url() + System.lineSeparator());
            logHeaders(logMessage, request.headers());
            RequestBody requestBody = request.body();
            byte[] body = logRequestBody(logMessage, requestBody);

            requestBody = requestBody != null ? RequestBody.create(body, requestBody.contentType()) : null;
            Request.Builder requestBuilder = new Request.Builder()
                    .cacheControl(request.cacheControl())
                    .headers(request.headers())
                    .method(request.method(), requestBody)
                    .url(request.url())
                    .removeHeader("Accept-Encoding"); // To avoid gzip encoding in the response
            return requestBuilder.build();
        }

        private Response logResponse(StringBuilder logMessage, Response response) throws IOException {
            logMessage.append(response.code() + " " + response.message() + System.lineSeparator());
            logHeaders(logMessage, response.headers());
            ResponseBody responseBody = response.body();
            byte[] body = logResponseBody(logMessage, responseBody);

            responseBody = responseBody != null ? ResponseBody.create(body, responseBody.contentType()) : null;
            return new Response.Builder()
                    .body(responseBody)
                    .code(response.code())
                    .handshake(response.handshake())
                    .headers(response.headers())
                    .message(response.message())
                    .networkResponse(response.networkResponse())
                    .priorResponse(response.priorResponse())
                    .protocol(response.protocol())
                    .receivedResponseAtMillis(response.receivedResponseAtMillis())
                    .request(response.request())
                    .sentRequestAtMillis(response.sentRequestAtMillis())
                    .build();
        }

        private byte[] logResponseBody(StringBuilder logMessage, ResponseBody body) throws IOException {
            if (body == null)
                return null;
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            byte[] chunk = new byte[4096];
            int bytesRead;
            InputStream bodyStream = body.byteStream();
            while ((bytesRead = bodyStream.read(chunk)) != -1)
                baos.write(chunk, 0, bytesRead);
            byte[] bytes = baos.toByteArray();
            logByteArray(logMessage, bytes, body.contentType());
            return bytes;
        }

        @Override
        public Response intercept(Chain chain) throws IOException {
            StringBuilder logMessage = new StringBuilder();
            Request request = chain.request();
            if (logRequest) {
                logMessage.append("--- REQUEST ---" + System.lineSeparator());
                request = logRequest(logMessage, chain.request());
                logMessage.append(System.lineSeparator());
            }
            Response response = chain.proceed(request);
            if (logResponse) {
                logMessage.append("--- RESPONSE ---" + System.lineSeparator());
                response = logResponse(logMessage, response);
                logMessage.append(System.lineSeparator());
            }
            if (!logMessage.isEmpty()) {
                logMessage.append(System.lineSeparator());
                System.out.println(logMessage.toString());
            }
            return response;
        }
    }

and attached it to the OkHttpClientBuilder as a Network Interceptor:

OkHttpClient.Builder clientBuilder = new OkHttpClient().newBuilder().addNetworkInterceptor(new OkHttpRequestResponseLogger(true, true));

thus obtaining the following log. Please note that:

• the log is created before removing the Accept-Encoding header; that's why it appears in the log of the Request;
• the body of the request are the 5 bytes, hex-encoded

--- REQUEST ---
PUT https://api.onedrive.com/rup/83a259e7e5f58e69/eyJSZXNvdXJjZUlEIjoiODNBMjU5RTdFNUY1OEU2OSExMDYiLCJSZWxhdGlvbnNoaXBOYW1lIjoidGhlTmFtZSJ9/4mmUna845TQ8AIQV60JuGUz85GIKZqgXPfYDiL7D0iapzCnn2QB-YMxwHYOFDv1QVUXFsKdVlVyE5xXj-IYsVbOfj6S5D1tmV9qZd3MW5uFwE/eyJuYW1lIjoidGhlTmFtZSIsImZpbGVTaXplIjo1LCJAbmFtZS5jb25mbGljdEJlaGF2aW9yIjoiZmFpbCJ9/4wK3xIQUXIHrtW7IxFslvprEAtnAPaebYdqlIn3scsCAjqBSdIybJdRQYdchnG_ClfCWoqm5odGae1AZ07i3YlZsp8u9wwq2ARxBK8lcExWtjas0mi83cqw9v1TsNZRfvNpzmYOgNQZVHcx0YtypCYDG3GErD5TQGaXStuoYnzXMr32lRBq2lckM20KSVIm05nmxLWSEOguUwe6KqqU5wZgEjm_Wk-GFY3nmTNh1pFIC9kjc-7ZgD08VDfXWzi02Sciejxo1of7_RL6slEjanaFDf6TiwJ9zCA8IkXKqz8AWGEN1a8nF1jNrwfAzD2X5HdyqrnvX4T4OzaVh2K0QkF3q5IAvQxdZLH-UjhOBvYLV6M_d1Ds5GjLDQemXSMbdxqABmnTPE-0MSbaJYzBM7NiSzeH5f_M0REgBATIHwdDvRXIgEx_qiQfwvF5OPhEU82irTNjZ30_3gBOHAg1ARdAEBD7s9eBO9uKwkYQebFAOO3SMDMyoAUpEeFglE1rbxFom8OuzJhTpv36k8Qpk6sjUR-LSfXN9pB9GNIic2loKmO5cTbVOPSXD0pZ_HhMBVU
Accept-Encoding : gzip
authorization : Bearer [...]
Connection : Keep-Alive
content-range : bytes 0-4/5
Content-Type : application/octet-stream
Host : api.onedrive.com
Transfer-Encoding : chunked
User-Agent : okhttp/4.12.0
2cad62d665

--- RESPONSE ---
400 
content-length : 116
content-type : application/json; charset=utf-8
date : Wed, 21 Feb 2024 21:58:43 GMT
ms-cv : NEC5dq6oIUC8MbCCa1aR2Q.0
p3p : CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
strict-transport-security : max-age=31536000; includeSubDomains
www-authenticate : Bearer realm="OneDriveAPI", error="invalid_token", error_description="Invalid auth token"
x-asmversion : UNKNOWN; 19.1338.129.2007
x-cache : CONFIG_NOCACHE
x-msedge-ref : Ref A: 208B0365004E4A08AE53C2BE5A3790C1 Ref B: MIL30EDGE1120 Ref C: 2024-02-21T21:58:44Z
x-msnserver : AMS0PF70708741F
x-qosstats : {"ApiId":0,"ResultType":2,"SourcePropertyId":0,"TargetPropertyId":42}
x-throwsite : 7b59.16f2
{"error":{"code":"invalidRequest","message":"Declared fragment length does not match the provided number of bytes"}}

The mistery is that if I attach the Interceptor as an Application Interceptor instead of an Network Interceptor (see here):

OkHttpClient.Builder clientBuilder = new OkHttpClient().newBuilder().addInterceptor(new OkHttpRequestResponseLogger(true, true));

the content-length header magically appears (I'd think it is added when I rebuild the request) and the response is a 201/Created:

--- REQUEST ---
PUT https://api.onedrive.com/rup/83a259e7e5f58e69/eyJSZXNvdXJjZUlEIjoiODNBMjU5RTdFNUY1OEU2OSExMDYiLCJSZWxhdGlvbnNoaXBOYW1lIjoidGhlTmFtZSJ9/4mK4M4DSp7mPr5dU_-DCeMF6SA7G6ouclrdte55WKjYIkNPDuMD8rNNnhhOGDvGlQAYi6rVIaupnGxLyewAg7t3I4TFkcaEVEmgnj8nm3V3SE/eyJuYW1lIjoidGhlTmFtZSIsImZpbGVTaXplIjo1LCJAbmFtZS5jb25mbGljdEJlaGF2aW9yIjoiZmFpbCJ9/4wEdbotO1H-HwjM1PtejvTZjnCKaMmLMZXRg-apkzp_eOA2VmfLYquwCLjhVJ2MbXHsi5g6GrK1uDSlTHsPT--cFF3SHDcCPxqAQqTgJgH6MwE9Drcm-2pmxVb9eCcXtqlhiJk7qOPzWixG8RUuKkr_KGeOwo93I4JSrO71SoQWLBzqDoNpCK-NwQ_f6M3Uuat6E9aPtBfGijlEMLlJeyGq_V3E7EFE_ORVEX11mLIQSrzqZjv6f9n_SKlWsz4qIetSBUpEjQQkTPal3mThnOVDtrksBYfH5iiWcEoZbOWs7DOM4Re6IOsCBfb5WdwYnGyHfReiDKDd2t1puWBE4NqjF8aACUCzbAyqntVz6DvYA_VAuEwTfW6HMAFzzYyXHAuWi2toceziOzxsUyR9B9E6h3twTKFJzJbOiZvcdSpCdfQ1yJmTTlrQW55Dqz4xOXDmtNc6v4e13z4-SlxAFTL7f1bSxCXa0fo5hzWCvFzQBz38PZIIfuWZ02IfFleDt0lyn-NAZgv8XutecyRgVndhxdARn9ht6_vOYjnpdb8RgxH4wcxZikZA22OiGamh7Eb
authorization : Bearer 
content-length : 5
content-range : bytes 0-4/5
content-type : application/octet-stream
14af14b7f5

--- RESPONSE ---
201 
content-type : application/json; charset=utf-8
date : Wed, 21 Feb 2024 22:54:00 GMT
ms-cv : hhMVBRuH00+EX0dNqNlZsA.0
p3p : CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
strict-transport-security : max-age=31536000; includeSubDomains
www-authenticate : Bearer realm="OneDriveAPI", error="invalid_token", error_description="Invalid auth token"
x-asmversion : UNKNOWN; 19.1338.129.2007
x-cache : CONFIG_NOCACHE
x-msedge-ref : Ref A: 48BD001FEEB6405EBCD806A8492DD553 Ref B: MIL30EDGE1310 Ref C: 2024-02-21T22:54:00Z
x-msnserver : AMS0PF48CFBC45A
{"createdBy":{"application":{"id":"4c3cb947"},"user":{"id":"83a259e7e5f58e69"}},"createdDateTime":"2024-02-21T22:54:00.85Z","cTag":"aYzo4M0EyNTlFN0U1RjU4RTY5ITE3NjQyOC4yNTc","eTag":"aODNBMjU5RTdFNUY1OEU2OSExNzY0MjguMA","id":"83A259E7E5F58E69!176428","lastModifiedBy":{"application":{"id":"4c3cb947"},"user":{"id":"83a259e7e5f58e69"}},"lastModifiedDateTime":"2024-02-21T22:54:00.85Z","name":"theName","parentReference":{"driveId":"83a259e7e5f58e69","driveType":"personal","id":"83A259E7E5F58E69!106","path":"/drive/root:"},"size":5,"webUrl":"https://1drv.ms/u/s!AGmO9eXnWaKDiuIs","items":[],"file":{"hashes":{"quickXorHash":"FHgFBW5RDwAAAAAABQAAAAAAAAA=","sha1Hash":"EB7056C190724D15F3AD39CCE2AE90D9ABBEB609","sha256Hash":"243550BE6C4662D9D105238A5AAF2E5FE24C024E38F0EFBE6F2885B30C057C25"},"mimeType":"application/octet-stream"},"fileSystemInfo":{"createdDateTime":"2024-02-21T22:54:00.85Z","lastModifiedDateTime":"2024-02-21T22:54:00.85Z"},"reactions":{"commentCount":0},"tags":[],"lenses":[]}

[baywet]

Thanks for reporting this.
Related microsoft/kiota-java#1088
The content length and range are both being set here

I believe that when creating the ok http request from the request information in the request adapter (see linked PR), the request body content length mechanism takes precedence. And because it's not down-casting the InputStream (unknown length) to the BinaryArrayInputStream (known length), it's not emitting the header.

The difference in behavior with the interceptor can be explained by the following lines:

Buffer sink = new Buffer();
            body.writeTo(sink);
            byte[] bytes = sink.readByteArray();

1. when the interceptor is a network interceptor, the request headers and body have already been written. And the content length is missing due to the explanation above.
2. when the interceptor is an application interceptor, reading the stream to the end makes its length known, and the header is emitted. (that'd lead to memory pressure if the source was an actual large file, because it'd load the content of the file entirely in memory)

At this point I don't think there's much you can do besides waiting for a resolution on the PR I linked.

[kekolab]

With SDK version 6.10 this issue disappeared. I hereby close it.