Skip to content

EventHub messages ignoring both checkpoint and StreamSequenceToken #7380

Open

Description

@galvesribeiro

We've recently noticed that what is happening on #2935 is happening again. Regardless of the checkpoint being write, all the streams seems to always start from the beginning of the EH causing duplicated messages to arrive on the grain.

Both @jiilaa and myself had the same issues as on that issue.

In my case, I tried to create the following measures to avoid that:

  1. Save every single StreamSequenceToken to the grain state as soon as I have the message processed.
  2. At OnActivateAsync(), pass the token to SubscribeAsync() (implicit subscription stream).

@benjaminpetit can you have a look on those? I've discussed with @ReubenBond earlier Today and it seems something that is affect again all users of EH.

Thanks!

Activity

  1. oising commented on Nov 9, 2021

    @oising
    Contributor

    @ReubenBond I wonder if this is why my cluster is thrashing a lot during an upgrade - maybe it's reprocessing an entire day's worth of messages again each time? Duplicate messages don't affect me negatively, as long as they're in order the second time around.

  2. benjaminpetit commented on Nov 9, 2021

    @benjaminpetit
    Contributor

    Do you have any logs you can share? What version of Orleans are you using?

    We have some logic to ignore the checkpoint value, if we think it might be bad.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-streamingCategory for Orleans streaming issues

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions