feat: Long timeout on actions to detect runaway Futures/CompletionStages

[johanandren]

In case an async reply or effect is returned that never completes, for example in a consumer, this will make sure it is logged as an error and retried, so that it hopefully is unstuck or at least shows up in logs.

Sample log output:

15:44:52.699 ERROR k.javasdk.impl.action.ActionsImpl - Failure during handling of command customer.action.CustomerByName.ProcessCustomerCreated
java.util.concurrent.TimeoutException: Command to action [CustomerByName] method [ProcessCustomerCreated], subject: [vip], sequence: [1] did not complete within 20 seconds
        at kalix.javasdk.impl.action.ActionsImpl.timeoutErrorFor(ActionsImpl.scala:193)
        at kalix.javasdk.impl.action.ActionsImpl.$anonfun$effectToResponse$2(ActionsImpl.scala:150)
        at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:50)
        at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:50)
        at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:473)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:60)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:511)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1450)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2019)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
15:44:52.716 WARN  kalix.javasdk.impl.DiscoveryImpl - Warning reported from Kalix system: KLX-00433 Eventing in service [customer.action.CustomerByName] is failing, will be retried.

[patriknw]

LGTM, with a small suggestion

[patriknw]

btw, can there be a risk that we add many such timer tasks to the scheduler and they will not be done (removed) until after 1 hour (1000 per second would be 3.6 million)

[johanandren]

Yeah, good point. Maybe we need something smarter here, would be enough with one such timeout future per minute or maybe even fewer.

[johanandren]

Added something that should create fewer timers and share them in 0b510bd

[patriknw]

so we create one instance of ActionsImpl per projection instance and keep that instance? not creating new instances of ActionsImpl for each request?

[johanandren]

ActionsImpl is the actual gRPC service implementation, so it's even one for all actions in the same Kalix SDK service.

[patriknw]

just thinking if there is an easier way? it's pretty cheap to add and cancel short lived scheduler tasks, so wonder if we could keep the previous Cancellable instead and when scheduling new we always cancel previous, since we know that it was handled when processing next message.

[johanandren]

Even simpler: we can close over it as well, and cancel on completion of the other future. I'll do that instead.

[johanandren]

Rewritten like that in 009faf1

[johanandren]

Note: safe to cancel an already cancelled

[patriknw]

LGTM