Burrow reports Kafka lag from a topic that a consumer stopped subscribing.

[won0817]

We lately had to update one of our consumers to stop subscribing events from a specific topic. Since then, Burrow has started reporting that the consumer has Kafka lag due to the topic stopped subscribing from. Is there a way to resolve this issue? Thanks.

[killerfish16]

We too have this issue. After the consumer stopped subscribing, Kafka cluster shows the right data.

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
Consumer group Name Topic Name 4 115340 115343 3
Consumer group Name Topic Name 5 118037 118039 2
Consumer group Name Topic Name 2 117487 117490 3
Consumer group Name Topic Name 3 115092 115093 1
Consumer group Name Topic Name 0 114706 114706 0
Consumer group Name Topic Name> 1 116951 116951 0
Consumer group Name Topic Name> 6 119787 119788 1
Consumer group Name Topic Name> 7 120258 120258 0
Consumer group Name Topic Name> 10 118424 118427 3
Consumer group Name Topic Name> 11 121063 121063 0
Consumer group Name Topic Name> 8 119425 119425 0
Consumer group Name Topic Name> 9 121588 121589 1

But burrow still considers the old topic.

burrow_kafka_consumer_lag_total{cluster="prd",consumer_group="",exported_cluster="prd",instance="hostname:9112",job="native_burrow"}

lag Value:
1205164

We are using burrow-1.3.4.

[orobardet]

Same problem for us. The only workaround is to restart kafka burrow, so that it will rediscover the actual and real state of consumer_groups and topics within the Kafka cluster.

#657 seems to fix the removal of entire CG, but not the case when the CG still exists but some of its topics were unsubscribed.

Would that be an acceptable solution to enrich the reaper introduced in #657 so that for each remaining CG, it also checks and cleans removed topics?

[arohter]

We run into this same problem, where topics are unsubscribed, but the CG remains, so existing reaping does not clean up. Would love if the new CG reaping was enhanced to pick up this use case ❤️

[won0817]

Please check this.