Skip to content

[FEATURE] Encina.AzureEventHubs - Azure Event Hubs Transport #611

@dlrivada

Description

@dlrivada

Summary

Implement Encina.AzureEventHubs transport for high-throughput event streaming.

Motivation

Azure Event Hubs is essential for:

  1. Massive Scale: Millions of events per second
  2. Azure Native: First-class Azure integration
  3. Kafka Compatible: Kafka protocol support
  4. Event Streaming: Real-time analytics pipelines

Use Cases

  • Telemetry ingestion
  • Real-time analytics
  • Event streaming pipelines
  • IoT data collection

Proposed Solution

public class AzureEventHubsTransport : IMessageTransport
{
    private readonly EventHubProducerClient _producer;
    private readonly EventProcessorClient _processor;
    
    public async Task PublishAsync<T>(T message, string? partitionKey = null, CancellationToken ct = default)
    {
        var eventData = new EventData(JsonSerializer.SerializeToUtf8Bytes(message));
        
        var options = partitionKey != null 
            ? new SendEventOptions { PartitionKey = partitionKey }
            : null;
        
        await _producer.SendAsync(new[] { eventData }, options, ct);
    }
}

Features

  • Partitioned publishing (partition key)
  • Consumer groups
  • Checkpointing with blob storage
  • Batch publishing
  • Schema Registry integration

Alternatives Considered

  1. Use Azure Service Bus instead: Service Bus is message-queue focused while Event Hubs is stream-focused. Event Hubs is better for high-throughput analytics scenarios.

  2. Use Kafka on Azure (Event Hubs with Kafka protocol): Event Hubs supports Kafka protocol, so this transport could optionally expose that interface for Kafka client compatibility.

  3. Use Azure Event Grid: Event Grid is for event routing while Event Hubs is for event streaming. They complement each other for different patterns.

Affected Packages

  • Encina (core)
  • Encina.EntityFrameworkCore
  • Encina.Dapper.*
  • Encina.ADO.*
  • Encina.AspNetCore
  • Encina.OpenTelemetry
  • Encina.Caching.*
  • Other: Encina.AzureEventHubs (new package)

Additional Context

Acceptance Criteria

  • Encina.AzureEventHubs package
  • Producer implementation
  • Consumer implementation with checkpointing
  • Partition key support
  • Consumer group support
  • Batch operations
  • Azure Schema Registry integration (optional)
  • ServiceCollection extensions
  • Unit tests
  • Integration tests
  • Documentation

Package Dependencies

  • Azure.Messaging.EventHubs
  • Azure.Messaging.EventHubs.Processor

Related Issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions