feat(storage): Add full object checksum validation at finalization for json path #15815
Conversation
product-auto-label
bot
added
the
api: storage
Summary of ChangesHello @v-pratap, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the Google Cloud Storage client library by implementing full object checksum validation for resumable uploads. It introduces mechanisms to calculate and transmit Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for sending object checksums (CRC32C and MD5) in the final chunk of a resumable upload for the JSON/REST transport. The changes look good and are well-tested. I've found a couple of minor issues related to leftover debugging code that should be addressed before merging.
| upload_request.ForEachOption([](auto const& opt) { | ||
| std::cout << "DEBUG: option=" << opt << "\n"; | ||
| }); |
There was a problem hiding this comment.
This appears to be a debug statement. Please remove it before merging.
| // if (argc != 2) { | ||
| // std::cerr << "Missing bucket name.\n"; | ||
| // std::cerr << "Usage: quickstart <bucket-name>\n"; | ||
| // return 1; | ||
| // } | ||
| std::string const bucket_name = "vaibhav-test-001"; | ||
|
|
||
| // Create a client to communicate with Google Cloud Storage. This client | ||
| // uses the default configuration for authentication and project id. | ||
| auto client = google::cloud::storage::Client(); | ||
|
|
||
| auto writer = client.WriteObject(bucket_name, "quickstart.txt"); | ||
| writer << "Hello World!"; | ||
| auto writer = client.WriteObject(bucket_name, "quickstart1.txt"); | ||
| std::string x ="Hello World1!"; | ||
| for(int i=0;i<17;i++){ | ||
| x+=x; | ||
| } | ||
| for(int i=0;i<15;i++){ | ||
| writer << x.data(); | ||
| } | ||
| writer.Close(); |
There was a problem hiding this comment.
These changes appear to be for debugging and manual testing. Please revert them before merging.
if (argc != 2) {
std::cerr << "Missing bucket name.\n";
std::cerr << "Usage: quickstart <bucket-name>\n";
return 1;
}
std::string const bucket_name = argv[1];
// Create a client to communicate with Google Cloud Storage. This client
// uses the default configuration for authentication and project id.
auto client = google::cloud::storage::Client();
auto writer = client.WriteObject(bucket_name, "quickstart.txt");
writer << "Hello World!";
writer.Close();| // auto reader = client.ReadObject(bucket_name, "quickstart.txt"); | ||
| // if (!reader) { | ||
| // std::cerr << "Error reading object: " << reader.status() << "\n"; | ||
| // return 1; | ||
| // } | ||
|
|
||
| std::string contents{std::istreambuf_iterator<char>{reader}, {}}; | ||
| std::cout << contents << "\n"; | ||
| // std::string contents{std::istreambuf_iterator<char>{reader}, {}}; | ||
| // std::cout << contents << "\n"; |
There was a problem hiding this comment.
These changes appear to be for debugging and manual testing. Please revert them before merging.
auto reader = client.ReadObject(bucket_name, "quickstart.txt");
if (!reader) {
std::cerr << "Error reading object: " << reader.status() << "\n";
return 1;
}
std::string contents{std::istreambuf_iterator<char>{reader}, {}};
std::cout << contents << "\n";1 participant
This PR modifies the client library to send the x-goog-hash header, containing crc32c= and/or md5= values, if available, when sending the final chunk of a resumable upload.
Implementation Details
The core of the change involves coordinating hash calculation between the ObjectWriteStreambuf and the RestStub. The hash_function within ObjectWriteStreambuf maintains the running checksum state for the entire upload.
RestStub::UploadChunk:
For intermediate chunks (when last_chunk() is false), UploadChunk receives a valid hash_function_ptr from ObjectWriteStreambuf. It calls hash_function->Update() to incorporate the chunk's payload into the running checksum.
For the final chunk (when last_chunk() is true), UploadChunk receives a nullptr for hash_function_ptr but receives the final, computed object hashes via request.known_object_hashes(). It then adds the x-goog-hash header(s) to the HTTP request using these final hashes.
ObjectWriteStreambuf::FlushFinal:
This method is called when Close() is invoked on the stream.
It first calls hash_function->Update() with any data still pending in its internal buffer.