Slow updates with 3K+ objects

[IliaGe]

What happened: After creating a Service (Headless) object - it takes ~15 minutes to update Route53 DNS record

What you expected to happen: Record will be created upon creation

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• External-DNS version (use external-dns --version): v0.13.5
• DNS provider: AWS Route53
• Others: Sources: service, ingress, istio-gateway, istio-virtualservice

We are running on EKS with BatchSizeConfig of 1,000 (default), we have ~3K Isito virtualService objects & 1K Service objects.
The actual update to R53 happens only after 10-15 min:

time="2025-01-19T08:34:20Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T08:42:11Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T08:50:02Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T08:57:53Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T09:05:44Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T09:13:36Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T09:21:28Z" level=info msg="Applying provider record filter for domains: [xxx]"

In between, K8S objects scanning, which is slow (3-4 VS's every seconds)
time="2025-01-19T09:21:50Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:50Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:50Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:50Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:53Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:53Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:53Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"

Anything I can Do ?
I have tried enabling the "--events" , but nothing happen upon object creation :(

[IliaGe]

Just to update, that I have created additional external-dns workers, with annotation filtering
to be able to process items faster :/

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[ivankatliarchuk]

I think I found one of the bottlenecks

/lifecycle frozen
/assign