Kafka topics are not automatically created when using franz-go

[keatsfonam]

Component(s)

exporter/kafka

What happened?

Description

When using topic auto-creation with franz-go enabled topics are not created on-demand. When using the current default sarama lib the topics are created without any issue. I am running this against warpstream, i'm not sure if the same behavior is exhibited in kafka oss.

Steps to Reproduce

• Enable topic auto creation
• Run collector with --feature-gates exporter.kafkaexporter.UseFranzGo

Expected Result

Topics are automatically created

Actual Result

Example error:

issue exporting from topic \"otlp_logs\": UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition

stacktrace:

github.com/open-telemetry/opentelemetry-collector-contrib/exporter/kafkaexporter.(*kafkaExporter[...]).exportData
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/kafkaexporter@v0.134.0/kafka_exporter.go:184
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v1.40.0/logs.go:27

Collector version

0.134.0

Environment information

Environment

OS: OSX
Compiler: go 1.24.2 darwin/arm64
Kafka: warpstream

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  kafka:
    brokers: ["localhost:9092"]
    client_id: "otel-collector"


service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [kafka]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [kafka]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [kafka]

Log output

issue exporting from topic \"otlp_logs\": UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy @axw

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[paulojmdias]

I'll look into this.

@pavolloffay @MovieStoreGuy @axw, you can assign it to me.

[axw]

@keatsfonam thanks for the opening the issue.

@paulojmdias thank you, done. If it's not too much to ask, could you please add an allow_auto_topic_creation option to the producer config? Although I'd rather this were opt-in, I suppose we'll need to make it opt-out since otherwise it's a breaking change.

[paulojmdias]

@keatsfonam I have confirmed, and it also happens with Kafka OSS.

@axw Sure, I’ll add the allow_auto_topic_creation option with a default value of true, so the behavior remains consistent with Sarama and avoids breaking existing setups.