Skip to content

.Net: Add Async Support for ConverseStreamResponse to Avoid Thread Blocking#10159

Merged
rogerbarreto merged 4 commits intomicrosoft:mainfrom
shethaadit:shethaadit/FixBug9519-New
Jan 16, 2025
Merged

.Net: Add Async Support for ConverseStreamResponse to Avoid Thread Blocking#10159
rogerbarreto merged 4 commits intomicrosoft:mainfrom
shethaadit:shethaadit/FixBug9519-New

Conversation

@shethaadit
Copy link
Copy Markdown
Contributor

Motivation and Context

This change replaces the use of response.Stream.AsEnumerable() with await foreach for async streaming in the Amazon Bedrock SDK. The previous method blocked the calling thread, leading to potential thread exhaustion in high-concurrency scenarios. This update aligns with best practices for asynchronous operations in .NET.

Description

  • Replaced blocking foreach with await foreach for streaming ConverseStreamResponse chunks asynchronously.
  • Upgraded Amazon Bedrock runtime and core SDKs to support this functionality.

Contribution Checklist

Fixes #9519

@shethaadit shethaadit requested a review from a team as a code owner January 11, 2025 08:50
@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel labels Jan 11, 2025
@github-actions github-actions bot changed the title Add Async Support for ConverseStreamResponse to Avoid Thread Blocking .Net: Add Async Support for ConverseStreamResponse to Avoid Thread Blocking Jan 11, 2025
@RogerBarret0
Copy link
Copy Markdown
Contributor

@shethaadit Do we really need to rely on 4.0.0-preview package for this? Can't we keep the current GA version?

@shethaadit
Copy link
Copy Markdown
Contributor Author

shethaadit commented Jan 12, 2025

Hi @RogerBarret0, this fix just got available yesterday in latest Nuget. We would need to keep the new version to get this fix.

aws/aws-sdk-net#3542 (comment)

@shethaadit
Copy link
Copy Markdown
Contributor Author

Hi @RogerBarret0 / @markwallace-microsoft, can we merge this as it's already approved?

@rogerbarreto
Copy link
Copy Markdown
Member

Hi @shethaadit, seems that the solution currently does not compile, please before posting the PR ensure the solution compiles and the all UT are passing.

Thanks.

@rogerbarreto rogerbarreto added this pull request to the merge queue Jan 16, 2025
Merged via the queue into microsoft:main with commit b34b16e Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel Issues or pull requests impacting the core kernel .NET Issue or Pull requests regarding .NET code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.Net: Bug: .NET Bedrock streaming completion is keeping the request thread busy

5 participants