Some subscription questions regarding scaling and test cases

[dedels]

I'm working on something requiring subscriptions to hit an external service which might result in more events to append. Here are some questions I have regarding subscriptions. Would these kind of Q's be useful for subscription FAQ?

1. If I run projections async or subscriptions, can I run them in an external worker application?
2. Can I get those workers to horizontally scale? or is it single threaded per projection/subscription?
3. If a subscription processing fails and I use the controller to send to deadletter - how do I re-process dead letters?
4. Do deadletters have a retry count?
5. If the retry count is exceeded, should the deadletters be manually deleted?
6. How can I run the subscription processor in a test case? I have successfully run projections in unit test using daemon.RebuildProjectionAsync

Thank you for any feedback

[jeremydmiller]

@dedels So, that's a lot.

1. Yes, but you'll need to do that by just having a little different Marten configuration for your workers vs your web apps. I.e., register the async projections and the daemon on the workers, but have the exact same Marten config otherwise (that's important)
2. The docs do discuss the limited ability Marten has to distribute projections, or if that's not enough, go full "Critter Stack" and use Wolverine's ability to distribute Marten projections: https://www.linkedin.com/pulse/scaling-event-projections-subscriptions-critter-stack-jeremy-miller-9ilrc/. That was originally going to be a paid add on model, but we gave up and moved it here
3. Marten isn't really a message bus, if you want that functionality, use Wolverine's Marten integration
4. No, again, out of Marten's scope. And again, you can get that with the integration with Wolverine. Just think of Marten as pretty well nothing but persistence
5. Again, see the above
6. Same API. There's a "Replay******" API as well in Marten, but it's the same thing under the covers.

We're working on the commercial addon management console to add UI monitoring and controls for a lot of what you're asking about, but we're aiming for end of the year.

[dedels]

Thank you for the detailed response. I understand the design considerations here better now.

6. RE: Subscriptions in test cases. I actually was literally just trying to run the subscription job. I did get this to work by using:

        await daemon.StartAllAsync();
        await daemon.WaitForNonStaleData(5.Minutes());

One more q:
7. In the subscription ProcessEventsAsync method, I am trying to enqueue more events to the stream:

operations.Events.Append(groupedEvents.Key, newEvents.Events);

but receive this

System.NotSupportedException: Specified method is not supported.
   at Marten.Events.Daemon.Internals.ProjectionUpdateBatch.Marten.Services.ISessionWorkTracker.TryFindStream(String streamKey, StreamAction& stream)
   at Marten.Events.EventGraph.Append(DocumentSessionBase session, String stream, Object[] events)
   at Marten.Events.EventStore.Append(String stream, Object[] events)
   at Marten.Events.EventStore.Append(String stream, IEnumerable`1 events)

Is this not supported on purpose b/c of design issue?

Thank you.

[jeremydmiller]

Yep, that's not supported (obviously). Not something I ever foresaw this being used for, even though it's not unreasonable. You can do that now through a "side effect" model with projections now. I'd be okay with a new GitHub issue to ask for this.

You can do this quite easily though if you were using Wolverine too and just letting events from Marten flow right to Wolverine message handlers. That would work OOTB.

[dedels]

So you write a subscriber from marten and then just send the event into wolverine? or is this a configuration?