Skip to content

EventHub Stream Provider and EventHub Checkpointer behaviour. #2935

Closed

Description

@centur

This question may be a bit weird, but we observing some unexplained behaviour with the eventhub stream provider.
My understanding that checkpointer should work as the following:
This is a stream of messages in Event hub from 1 to 10 (assume we have only 1 partition)

      | - checkpointer points here, offset == 4
1,2,3,4,5,6,7,8,9,10
              | - last delivered messageId = 8

As processing moves on, checkpointer saves it's offset position in Azure table for each partition.
If cluster restarts - checkpointer reads the position and starts pulling messages from that point, avoiding the duplication\re-reading messages that were already delivered. The checkpoint frequency defines how many messages would be re-delivered in case of silo cluster crash e.g. last ~1 minute of messages if checkpointer on default freq, so if pullers delivered 4 to 8 messages in last minute and then server dies before new checkpoint offset written down to the table - on fresh start it will re-deliver these messages because they are after the last written offset.

For some unknown reason - EH checkpointer in our case saves offset only once, and never updates that value ( we tried default checkpointer interval and custom - 3 seconds interval too).
When there is no checkpoint record in the table - it will be created (not always though), but after that - it's not being updated at all - we have offsets on different environments stuck back in time like 2-3 weeks.

As a consequence of this - every time we are deploying new version of code - all events in EH stream are being replayed from the beginning of the stream. For us it might be a full day of annotations. It's currently on the test environment only, so no customers harmed yet, but

Interesting though, that if the offset record is deleted - no events are replayed (I'd expect the opposite behaviour as there is no offset record - everything should be replayed). As soon as I annotate on the page and new message is sent to EHStream - offset record appears and the replay saga starts all over - message that triggered offset record is being replayed again and again on every cluster restart.
It feels that checkpointer is doing the opposite:

  1. It writes the offset that is pointing to the position of the stream that is before the message
  2. It replays everything from that offset, including the message that triggered the offset change,
  3. It fails to write new offset and doomed to replay all messages from that offset.

I can consistently reproduce this behaviour on my local machine but I can't identify what is configured wrong. Could it be a bug in the EH Stream Provider Checkpointer code in version 1.3.1 (can't test it on latest 1.4.1 as we can't upgrade yet)

For better illustration here is slimmed down log of what happens and what I've done (I'll use last 2 digits of each guid)
image

Initial setup: clean storage, no checkpoint record.

  1. Created 2 annotations - 62 and 52, they are delivered to the listener ( 2 last records on the log picture). Offset saved.
  2. Restarted local cluster - 62 and 52 re-delivered. Offset is unchanged.
  3. Deleted offset record. Started cluster - nothing is re-delivered. Added 9a annotation
  4. Restarted again - 9a was re-delivered. Offset is unchanged
  5. Restarted again - 9a was re-delivered. Added another annotation - 40. Offset is unchanged
  6. Restarted again - 9a and 40 are redelivered together. Offset is unchanged

Activity

  1. centur commented on Apr 12, 2017

    @centur
    ContributorAuthor

    Current solution to the problem is ridiculously stupid - I've configured checkpointNamespace to be unique on every start (it's $"{DateTimeOffset.UtcNow:O}-{Guid.NewGuid()}) which means - there will be no checkpoint record found and we will accumulate trash in the checkpointer table over time.

    Unfortunately I can't 'unconfigure' Checkpointer settings - they must be configured for EHSP to start....

    cc @jason-bragg

  2. jason-bragg commented on Apr 12, 2017

    @jason-bragg
    Contributor

    EH checkpointer in our case saves offset only once, and never updates that value.

    Unfortunately, what you're seeing is the results of a bug in that release. Please see #2613

    As for the expected checkpoint behavior, checkpoints are currently triggered when the queue cache purges messages, with a max frequency of the configured PersistInterval. We are looking at changing this a bit, as this means the current code will not write checkpoints until all of the blocks in the block pool have been allocated, and start being reclaimed. As the block size in the pool is 1mb, in general it should be expected that a queue will write a checkpoint each 1mb of data processed. This is fine for hot (or warm) loads, but for low flow queues with large recovery caches this can introduce some unexpected behaviors.

    As for the behavior to expect when the checkpoints are not there (either not written or deleted), that will depend on the StartFromNow setting in the EventHubSettings, which defaults to 'true'. If there is no checkpoints the eventhub can start from the beginning of the partition, or from the most recent messages ('now'). By default, without checkpoints, the eventhub stream provider will start processing new messages, and not reprocess older messages. If StartFromNow is set to false and there is no checkpoint, the stream provider will start reading from the beginning of the partition. If a checkpoint is found, it will always be used, regardless of the StartFromNow setting.

  3. centur commented on Apr 12, 2017

    @centur
    ContributorAuthor

    Good news - it's not my misconfiguration or misunderstanding, bad news - we can't upgrade to 1.4.x yet.

    Is there any chance this fix can be back-ported to current-1 release ?
    Current workaround behaviour seems working on a low flow stream but I can think about the case when stream would have non-empty queue and restarting the app would cause this piece to be unprocessed. With that case - Azure Queues would be more reliable in terms of delivery guarantees, but we reluctant to go back to them for annotations.

  4. centur commented on Apr 12, 2017

    @centur
    ContributorAuthor

    Also keen to know how EH pullers would work in the case when there is 2 copies of the orleans cluster is running - ClusterA pullers would pull messages from partition according to their inner state and cluster B pullers would start pulling from 'now' so there will be a period when both pullers deliver the same message to ClusterA and ClusterB grain activation - is this sounds correct ?

  5. added this to the Triage milestone on Apr 13, 2017
  6. jason-bragg commented on Apr 13, 2017

    @jason-bragg
    Contributor

    In the case of two clusters running the same service (a production and staging deployment for example), the second cluster will start reading from the last checkpoints written from the first cluster, and this will duplicate processing.

    One way of handling this has been to spin up the second cluster with the stream providers disabled, then enable them after the vip swap.

    Another approach is to have strong etag validation in any grains writing data from processed streams, and upon conflicts, reload the grain state and rewind the stream (assuming recoverable streams like EH) to the last message successfully processed by the grain. This will create storage conflicts while both clusters are processing the same data, but not introduced data corruption or duplicate results.

    There has also been a request for a queue balancer that balances queues across deployments so that a second deployment will be allocated queues that are moved from the first deployment, thereby eliminating the conflicts.

  7. ghost locked as resolved and limited conversation to collaborators on Sep 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions