[coordinator] Fix status reporting of out-of-order component updates

[VihasMakwana]

What does this PR do?

When transitioning from otel to process runtime, if an otel component takes too long to stop, it will emit Stopped state only after timeout expiration. By this time, the process runtime would have already reported a Starting state.
Upon receiving a Stopped state from old runtime, we will erroneously remove the new Starting state.

This PR fixes the flow by introducing a new LastCreatedAt variable for a component. We will only process a state update when the state update is either from same instance of the component, or from a newer instance.

Why is it important?

Buggy scenario:

1. Component c is created at time=0s
2. It transitions to Starting state. We will report this state as it's the first state for this component.
3. The user updates the config and changes the runtime.
4. The following steps take place concurrently
   • Coordinator stops the component and waits for it to exit.
   • Coordinator creates a new component model and runs it in process mode and time=1s. It also reports a Starting state for a given component.
5. The older components takes too long to stop and we forcefully kill it and report a Stopped state.
6. The coordinator receives this stale Stopped event and erroneously removes the new component from the status map.

After the PR:

1. Component c is created at startTime=0s
2. It transitions to Starting state. We will report this state as it's the first state for this component.
3. The user updates the config and changes the runtime.
4. The following steps take place concurrently
   • Coordinator stops the component and waits for it to exit.
   • Coordinator creates a new component model and runs it in process mode and startTime=1s. It also reports a Starting state for a given component.
5. The older components takes too long to stop and we forcefully kill it and report a Stopped state.
6. The coordinator receives this stale Stopped event but ignores it, since the stored startTime of the current component is later than the received event.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

• See [beats receivers] Switching from otel to process runtime can remove components from status output #12931

Related issues

• Closes [beats receivers] Switching from otel to process runtime can remove components from status output #12931

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[swiatekm]

This is a pretty nice solution to the problem, well done! Have you tested whether it solves it in practice? I recall @cmacknz being able to reproduce it consistently on his dev machine.

[VihasMakwana]

This is a pretty nice solution to the problem, well done! Have you tested whether it solves it in practice? I recall @cmacknz being able to reproduce it consistently on his dev machine.

Unfortunately, I haven't been able to reproduce this. I'm thinking of adding time.Sleep in filebeat receiver's Shutdown method to simulate a delay for now and test this out.

[VihasMakwana]

This one is needed because the state coordinator needs LastConfiguredAt to correctly handle state transitions.

[VihasMakwana]

@swiatekm @cmacknz I was able to test my fix after adding a time.Sleep(5s) to the beatreciever's shutdown method. The stale Stopped state doesn't overwrite process runtime's new Starting state

[cmacknz]

Thanks, agree this is a nice solution. I'll let Mikolaj do the approving.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 1b40be6

Failed CI Steps

• Win2022:sudo:upgrade
• x86_64:sudo:rpm - platform-ingest-elastic-agent-rhel-10-1772525581

History

• 💔 Build #35716 failed 33d0e37
• 💔 Build #35700 failed faa0fdf

cc @VihasMakwana