Enable specifying Grpc options like " MaxReceiveMessageSize" in WorkflowServiceCollectionExtensions

[hsinghdeol-wpay]

Currently dapr workflows fail with following error
2023-10-11T06:26:20.075785954Z Grpc.Core.RpcException: Status(StatusCode="ResourceExhausted", Detail="Received message exceeds the maximum configured message size.") 2023-10-11T06:26:20.075793354Z at Grpc.Net.Client.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
2023-10-11T06:26:20.075801654Z at Grpc.Net.Client.Internal.HttpContentClientStreamReader2.MoveNextCore(CancellationToken cancellationToken) 2023-10-11T06:26:20.075833955Z at Microsoft.DurableTask.GrpcExtensions.ReadAllAsync[T](IAsyncStreamReader1 reader, CancellationToken cancellationToken)+MoveNext()
2023-10-11T06:26:20.075841955Z at Microsoft.DurableTask.GrpcExtensions.ReadAllAsync[T](IAsyncStreamReader1 reader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() 2023-10-11T06:26:20.075852855Z at Microsoft.DurableTask.Worker.Grpc.GrpcDurableTaskWorker.Processor.ProcessWorkItemsAsync(AsyncServerStreamingCall1 stream, CancellationToken cancellation)
2023-10-11T06:26:20.075860455Z at Microsoft.DurableTask.Worker.Grpc.GrpcDurableTaskWorker.Processor.ProcessWorkItemsAsync(AsyncServerStreamingCall1 stream, CancellationToken cancellation) 2023-10-11T06:26:20.075866755Z at Microsoft.DurableTask.Worker.Grpc.GrpcDurableTaskWorker.Processor.ExecuteAsync(String target, CancellationToken cancellation)
I found out that there is no way to override default limit of 4mb in the following file.

dotnet-sdk/src/Dapr.Workflow/WorkflowServiceCollectionExtensions.cs

Line 159 in 3b979e6

GrpcChannelOptions options = new() { HttpClient = client};

There should be way to override default GRPC options

[zminot]

I've run into this as well. I know this was referenced in #978 (comment), but also bringing this up in this issue.

It seems like DurableTasks has a hard limit of 4MB as noted in microsoft/durabletask-dotnet#150 (comment), and the Dotnet SDK for Dapr Workflow uses that underneath. Do we think it'd be a good idea to update the Dapr documentation for this if this is now a known (and presumedly unsolvable by Dapr) issue?

Not sure if microsoft/durabletask-dotnet#154 is something that will alleviate this, however.

[cgillum]

I don't think this should require any work on the Durable Task SDK side since the Dapr Workflow client already controls its gRPC channel configuration as referenced in the OP. I can imagine that we may need to expose this in the sidecar as well, but it's possible this may already be supported if it's a server setting (durabletask-go is part of the same server as the rest of the Dapr APIs).

Agreed that this is important to fix given that it can cause workflows to fail mid-execution.

[WhitWaldo]

Should be resolved by #1244