Skip to content

Conversation

@chandra-siri
Copy link
Collaborator

@chandra-siri chandra-siri commented Oct 27, 2025

Multiple download_ranges should be able to run via multiple async tasks in a single event loop.

ie when a user runs a code like below , it works

... # some code 
await mrd.download_ranges(...)
await mrd.download_ranges()

but when a user runs code like below it doesn't work. This PR is fixing this issue. More details in internal b/449916743

... # some code 
task1 = asyncio.create_task(mrd.download_ranges(..))
task2 = asyncio.create_task(mrd.download_ranges(..))
_ = await asyncio.gather(task1, task2)

This PR achieves the above goal by

  • converts read_id_to_writable_buffer_dict into a class attribute instead of local variable of download_ranges
  • makes read_id random int, instead of ordered i + j
  • Provide an id (random int) for each method.
  • Add two more maps for keeping track of read_id to function_id and buffer
  • Because reads on client-to-server stream can be made one at a time, hence asyncio.lock is used.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: storage Issues related to the googleapis/python-storage API. labels Oct 27, 2025
@chandra-siri chandra-siri force-pushed the concurrency_for_download_ranges branch from 58fdf9d to 76b2327 Compare October 28, 2025 10:39
@chandra-siri chandra-siri changed the title Concurrency support for download_ranges fix: "download_ranges" should be able to run from multiple coroutines Oct 28, 2025
@chandra-siri chandra-siri marked this pull request as ready for review October 28, 2025 11:34
@chandra-siri chandra-siri requested review from a team as code owners October 28, 2025 11:34
@chandra-siri chandra-siri changed the title fix: "download_ranges" should be able to run from multiple coroutines fix: Make download_ranges compatible with asyncio.create_task Oct 28, 2025
@chandra-siri chandra-siri changed the title fix: Make download_ranges compatible with asyncio.create_task fix: Make download_ranges compatible with asyncio.create_task(..) Oct 28, 2025
Pulkit0110
Pulkit0110 previously approved these changes Oct 28, 2025
@chandra-siri chandra-siri merged commit faf8b83 into main Oct 29, 2025
15 checks passed
@chandra-siri chandra-siri deleted the concurrency_for_download_ranges branch October 29, 2025 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the googleapis/python-storage API. size: m Pull request size is medium.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants