feat(logs): add Buffering and Batching

[Flash0ver]

Add minimal support for Buffering / Batching for Sentry Logs.
Planned follow-up to avoid having high-frequency logging drop other, potentially more important, envelopes: #4306

Changes (Updated):

• SentryStructuredLogger is now IDisposable
  • and is Disposed when Hub is Disposed
  • also Flushes all logs when Hub is Disposed
• the user-facing SentryLog is no longer ISentryJsonSerializable
• but the new internal StructuredLog is ISentryJsonSerializable
  • this new internal type groups one or many SentryLog instances for serialization
• the DefaultSentryStructuredLogger no longer directly captures and enqueues Log-Envelopes
  • but indirectly via the new StructuredLogBatchProcessor instead
• new type StructuredLogBatchProcessor
  • buffers Logs
    • double-buffer strategy
    • swap buffer when full or timeout exceeded
  • sends Logs (Flush) when
    • Count-Threshold is reached
    • Timeout is reached
  • utilizes a lock-free solution, based on a CountdownEvent that only signals/Waits when Flushing
• using a new helper type, StructuredLogBatchBuffer
  • a wrapper over an Array
  • that provides thread-safe Add and Flush operations
• using a new synchronization primitive, ScopedCountdownLock
  • based on CountdownEvent
  • has an initial count / offset of 1
    • when the offset has not been signaled yet, cannot Set the event (used for tracking active Enqueue/Add operations
    • when the Lock is engaged (when a Flush is requested), can no longer Enqueue/Add and can Wait until existing Enqueue/Add operations have completed
    • after the Flush has completed, disengage the Lock to reset the count to the offset of 1 to allow Enqueue/Add operations again
• add Benchmark for StructuredLogBatchProcessor

---

#skip-changelog

[github-actions]

	Messages
📖	Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against 49fefc1

[bruno-garcia]

@sentry review

[seer-by-sentry]

On it! We are reviewing the PR and will provide feedback shortly.

[seer-by-sentry]

PR Description

This pull request introduces a batch processing mechanism for Sentry logs. The goal is to improve performance and reduce overhead by collecting log messages and sending them to Sentry in batches, rather than individually.

Click to see more

Key Technical Changes

1. BatchBuffer Class: A new BatchBuffer<T> class is introduced to act as a fixed-size buffer for collecting log messages. It provides methods for adding items, checking if the buffer is full or empty, and converting the buffer to an array.
2. BatchProcessor Class: A BatchProcessor class is implemented to manage the batching process. It uses a timer to periodically flush the buffer and send the logs to Sentry. It also flushes the buffer when it reaches its maximum capacity.
3. BatchProcessorTimer Abstraction: An abstract BatchProcessorTimer class and its TimersBatchProcessorTimer implementation are added to abstract the timer functionality, allowing for different timer implementations.
4. StructuredLog Class: A StructuredLog class is introduced to encapsulate an array of SentryLog items, representing a batch of logs to be sent to Sentry.
5. Modified SentryLog Class: The SentryLog class is modified to remove the ISentryJsonSerializable interface implementation and the WriteTo method is made internal.
6. Envelope Modifications: The Envelope and EnvelopeItem classes are modified to handle the new StructuredLog class, allowing for the creation of envelopes containing batches of logs.
7. SentryOptions Configuration: The SentryOptions class is extended with InternalBatchSize and InternalBatchTimeout properties to configure the batch processing behavior.
8. DefaultSentryStructuredLogger Modifications: The DefaultSentryStructuredLogger is modified to use the BatchProcessor for enqueueing logs, and the Dispose method is updated to dispose of the BatchProcessor.

Architecture Decisions

1. Batching Mechanism: A timer-based batching approach was chosen to balance latency and throughput. Logs are sent either when the buffer is full or when a specified time interval has elapsed.
2. Locking Strategy: A lock is used to synchronize access to the buffer and prevent race conditions. This ensures that log messages are added and flushed safely.
3. Timer Abstraction: The BatchProcessorTimer abstraction allows for flexibility in choosing the timer implementation. The default implementation uses System.Timers.Timer, but other implementations could be used in different environments.
4. StructuredLog Encapsulation: The StructuredLog class encapsulates the array of SentryLog items, providing a clear separation of concerns and simplifying the envelope creation process.

Dependencies and Interactions

1. IHub: The BatchProcessor depends on the IHub interface to capture envelopes and send them to Sentry.
2. SentryOptions: The DefaultSentryStructuredLogger relies on SentryOptions to configure the batch processing behavior.
3. System.Timers.Timer: The TimersBatchProcessorTimer implementation uses the System.Timers.Timer class for scheduling the batch flushing.
4. SentryLog: The StructuredLog class depends on the SentryLog class to represent individual log messages.

Risk Considerations

1. Lock Contention: The lock used to synchronize access to the buffer could become a bottleneck under high load. Performance testing should be conducted to ensure that the locking strategy is efficient.
2. Timer Accuracy: The accuracy of the System.Timers.Timer class can be affected by system load and other factors. This could lead to variations in the batch flushing interval.
3. Unhandled Exceptions: Exceptions thrown during the flush operation or in event handlers could crash the timer thread. Exception handling should be implemented to prevent this.
4. ObjectDisposedExceptions: The SentryStructuredLogger needs to be properly disposed to prevent ObjectDisposedExceptions when logging after disposal.

Notable Implementation Details

1. The Debug.Assert statement in BatchProcessor.EnqueueCore is used to check for unexpected conditions. This assertion should be replaced with a more robust error handling mechanism in production code.
2. The ClampBatchCount and ClampBatchInterval methods in DefaultSentryStructuredLogger are used to ensure that the batch size and interval are within reasonable bounds. These methods should be carefully reviewed to ensure that the limits are appropriate.
3. The FakeBatchProcessorTimer class in the test suite uses reflection to create ElapsedEventArgs instances for .NET Framework. This approach is brittle and could break if the internal constructor changes. A more robust approach should be used for testing.

[bruno-garcia]

please add a benchmark for this too, we definitely want to get an idea of how this behaves under load. In the tests, even if you're not committing that consider writing something that writes in a tight loop using a few threads (like 1 per CPU core you have) and check how that behaves too.