Advice on how to achieve reliable events from the APIServerSource

[darkgnotic]

Hello, I'm trying to achieve a reliable stream of kubernetes api server events.

My setup:

• an APIServerSource
• outputting to a Broker
• via a NATSChannel
• backed by a jetstream cluster with 3 replicas

I'm investigating gaps in kubernetes events (e.g. pod and replicaset events that never appear), and they seem to coincide with knative pods getting shut down (logs below).

I understand that pods periodically need to be shut down and/or moved, but I had assumed that by using jetstream, the event stream would pick up where it left off and resume sending the events that were missed while the workers were shutdown.

I don't believe that's happening and am wondering if there is a way to achieve this.

More generally, what's the recommended approach for getting a reliable stream of kube apiserver events (i.e. resilient to pod restarts, etc.)?

Thank you in advanced for your help!!

line	time	pod	level	msg	error
35	2026-01-16T00:18:42.059Z	eventing-webhook-6d99b99cff-b6wdt	info	Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0x40022e7c20), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, /"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"29175"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*request.BodyWrapper)(0x4002560e80), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:29175, TransferEncoding:[]string(nil), Close:false, Host:"eventing-webhook.knative-eventing.svc:443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"192.168.79.91:34132", RequestURI:"/sinkbindings?timeout=10s", TLS:(*tls.ConnectionState)(0x40023d2000), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:"/sinkbindings", ctx:(*context.valueCtx)(0x40025561e0), pat:(*http.pattern)(0x400041fec0), matches:[]string(nil), otherValues:map[string]string(nil)}
36	2026-01-16T00:18:42.060Z	eventing-webhook-6d99b99cff-b6wdt	info	remote admission controller audit annotations=map[string]string(nil)
37	2026-01-16T00:18:42.065Z	eventing-webhook-6d99b99cff-b6wdt	info	Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0x4002424360), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, /"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"16814"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*request.BodyWrapper)(0x4002561a00), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:16814, TransferEncoding:[]string(nil), Close:false, Host:"eventing-webhook.knative-eventing.svc:443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"192.168.79.91:55882", RequestURI:"/sinkbindings?timeout=10s", TLS:(*tls.ConnectionState)(0x40023d32c0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:"/sinkbindings", ctx:(*context.valueCtx)(0x4002557350), pat:(*http.pattern)(0x400041fec0), matches:[]string(nil), otherValues:map[string]string(nil)}
38	2026-01-16T00:18:42.066Z	eventing-webhook-6d99b99cff-b6wdt	info	remote admission controller audit annotations=map[string]string(nil)
39	2026-01-16T00:18:42.165Z	imc-controller-6c5766465-88hjk	info	Shutting down workers
40	2026-01-16T00:18:42.167Z	eventing-webhook-6d99b99cff-b6wdt	info	Reconcile succeeded
41	2026-01-16T00:18:42.167Z	imc-controller-6c5766465-88hjk	info	Shutting down workers
42	2026-01-16T00:18:42.167Z	imc-controller-6c5766465-88hjk	info	Shutting down workers
43	2026-01-16T00:18:42.167Z	imc-controller-6c5766465-88hjk	info	Shutting down workers
44	2026-01-16T00:18:42.171Z	job-sink-5685879598-jgtb5	info	Exiting...
45	2026-01-16T00:18:42.185Z	imc-controller-6c5766465-88hjk	info	Starting to fail readiness probes...
46	2026-01-16T00:18:42.200Z	jetstream-ch-controller-7b48477cf4-zt8m5	info	Shutting down workers
47	2026-01-16T00:18:42.232Z	jetstream-ch-dispatcher-56c75879b4-xtgcl	info	Shutting down workers
48	2026-01-16T00:18:42.232Z	jetstream-ch-controller-7b48477cf4-zt8m5	info	"jetstream-ch-controller-7b48477cf4-zt8m5_bbf3bb78-d703-4f35-b2d0-41f7e2a4ce69" has stopped leading "jetstream-channel-controller.knative.dev.eventing-natss.pkg.channel.jetstream.controller.reconciler.00-of-01"
49	2026-01-16T00:18:42.262Z	request-reply-0	info	Exiting...
50	2026-01-16T00:18:42.269Z	jetstream-ch-dispatcher-56c75879b4-xtgcl	info	"jetstream-ch-dispatcher-56c75879b4-xtgcl_c2a49cf3-6e7a-4ad0-bbce-c3766f24345e" has stopped leading "jetstream-channel-dispatcher.knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.reconciler.00-of-01"
51	2026-01-16T00:18:42.287Z	cluster-events-sns-sink-transform-jsonata-74f4b7c768-j4ttm
52	2026-01-16T00:18:42.291Z	cluster-events-sns-sink-transform-jsonata-74f4b7c768-j4ttm
53	2026-01-16T00:18:42.301Z	jetstream-ch-dispatcher-56c75879b4-xtgcl	error	Error while running server	listen tcp :8080: bind: address already in use
54	2026-01-16T00:18:42.301Z	cluster-events-sns-sink-transform-jsonata-74f4b7c768-j4ttm
55	2026-01-16T00:18:42.329Z	eventing-webhook-6d99b99cff-b6wdt	info	Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0x4002425cb0), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, /"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"7816"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*request.BodyWrapper)(0x4000bdf880), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:7816, TransferEncoding:[]string(nil), Close:false, Host:"eventing-webhook.knative-eventing.svc:443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"192.168.100.108:46018", RequestURI:"/defaulting?timeout=10s", TLS:(*tls.ConnectionState)(0x400276b200), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:"/defaulting", ctx:(*context.valueCtx)(0x400252b6b0), pat:(*http.pattern)(0x400041fce0), matches:[]string(nil), otherValues:map[string]string(nil)}
56	2026-01-16T00:18:42.330Z	eventing-webhook-6d99b99cff-b6wdt	info	Kind: "eventing.knative.dev/v1alpha1, Kind=EventTransform" PatchBytes: null
57	2026-01-16T00:18:42.330Z	eventing-webhook-6d99b99cff-b6wdt	info	remote admission controller audit annotations=map[string]string(nil)
58	2026-01-16T00:18:42.337Z	eventing-webhook-6d99b99cff-b6wdt	info	Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0x4002577d40), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, /"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"7816"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*request.BodyWrapper)(0x4002668140), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:7816, TransferEncoding:[]string(nil), Close:false, Host:"eventing-webhook.knative-eventing.svc:443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"192.168.100.108:46022", RequestURI:"/resource-validation?timeout=10s", TLS:(*tls.ConnectionState)(0x400276b740), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:"/resource-validation", ctx:(*context.valueCtx)(0x4001ecda40), pat:(*http.pattern)(0x400041e7e0), matches:[]string(nil), otherValues:map[string]string(nil)}
59	2026-01-16T00:18:42.338Z	eventing-webhook-6d99b99cff-b6wdt	info	remote admission controller audit annotations=map[string]string(nil)
60	2026-01-16T00:18:42.346Z	imc-controller-6c5766465-88hjk	info	"imc-controller-6c5766465-88hjk_21a05560-e94e-492f-a0d8-594499c56c8f" has stopped leading "inmemorychannel-webhook.knative.dev.eventing.pkg.reconciler.inmemorychannel.controller.reconciler.00-of-01"
61	2026-01-16T00:18:42.346Z	imc-controller-6c5766465-88hjk	info	"imc-controller-6c5766465-88hjk_1207dc49-abf5-411a-8ea5-ced43b732631" has stopped leading "inmemorychannel-webhook.validationwebhook.00-of-01"
62	2026-01-16T00:18:42.346Z	imc-controller-6c5766465-88hjk	info	"imc-controller-6c5766465-88hjk_8a10352d-dbd0-4f50-988f-acc48fea4e96" has stopped leading "inmemorychannel-webhook.defaultingwebhook.00-of-01"
63	2026-01-16T00:18:42.346Z	imc-controller-6c5766465-88hjk	info	"imc-controller-6c5766465-88hjk_3438bf00-6e15-411c-b53a-61acbb9bb686" has stopped leading "inmemorychannel-webhook.webhookcertificates.00-of-01"
64	2026-01-16T00:18:42.381Z	mt-broker-filter-664b8fdcfd-cjp77	info	Exiting...
65	2026-01-16T00:18:42.383Z	nats-webhook-5cd4d796-szj7g	info	Starting to fail readiness probes...
66	2026-01-16T00:18:42.383Z	nats-webhook-5cd4d796-szj7g	info	Shutting down workers
67	2026-01-16T00:18:42.383Z	nats-webhook-5cd4d796-szj7g	info	Shutting down workers
68	2026-01-16T00:18:42.383Z	nats-webhook-5cd4d796-szj7g	info	Shutting down workers
69	2026-01-16T00:18:42.383Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
70	2026-01-16T00:18:42.383Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
71	2026-01-16T00:18:42.393Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
72	2026-01-16T00:18:42.413Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
73	2026-01-16T00:18:42.415Z	nats-webhook-5cd4d796-szj7g	info	"nats-webhook-5cd4d796-szj7g_15feb1ca-9f6d-46f3-82e0-734f95bc9b9e" has stopped leading "nats-webhook.validationwebhook.00-of-01"
74	2026-01-16T00:18:42.433Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
75	2026-01-16T00:18:42.469Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
76	2026-01-16T00:18:42.469Z	imc-dispatcher-645cd5c75b-dvqmr	info	Shutting down workers
77	2026-01-16T00:18:42.472Z	nats-webhook-5cd4d796-szj7g	info	"nats-webhook-5cd4d796-szj7g_f6881f7e-b9e5-458a-b998-2147fe60fe67" has stopped leading "nats-webhook.defaultingwebhook.00-of-01"
78	2026-01-16T00:18:42.489Z	nats-webhook-5cd4d796-szj7g	info	"nats-webhook-5cd4d796-szj7g_a8199827-c90f-4bf1-a277-2f1975373f4b" has stopped leading "nats-webhook.webhookcertificates.00-of-01"
79	2026-01-16T00:18:42.489Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
80	2026-01-16T00:18:42.511Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
81	2026-01-16T00:18:42.552Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
82	2026-01-16T00:18:42.579Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
83	2026-01-16T00:18:42.601Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
84	2026-01-16T00:18:42.621Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
85	2026-01-16T00:18:42.639Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
86	2026-01-16T00:18:42.669Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
87	2026-01-16T00:18:42.704Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
88	2026-01-16T00:18:42.729Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
89	2026-01-16T00:18:42.735Z	eventing-webhook-6d99b99cff-b6wdt	info	Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0x4002475cb0), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, /"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"16792"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*request.BodyWrapper)(0x4002668b40), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:16792, TransferEncoding:[]string(nil), Close:false, Host:"eventing-webhook.knative-eventing.svc:443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"192.168.79.91:55882", RequestURI:"/sinkbindings?timeout=10s", TLS:(*tls.ConnectionState)(0x40023d32c0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:"/sinkbindings", ctx:(*context.valueCtx)(0x4002390240), pat:(*http.pattern)(0x400041fec0), matches:[]string(nil), otherValues:map[string]string(nil)}
90	2026-01-16T00:18:42.736Z	eventing-webhook-6d99b99cff-b6wdt	info	remote admission controller audit annotations=map[string]string(nil)
91	2026-01-16T00:18:42.759Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
92	2026-01-16T00:18:42.773Z	eventing-webhook-6d99b99cff-b6wdt	info	Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0x400228a2d0), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, /"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"29153"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*request.BodyWrapper)(0x4002669300), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:29153, TransferEncoding:[]string(nil), Close:false, Host:"eventing-webhook.knative-eventing.svc:443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"192.168.79.91:55882", RequestURI:"/sinkbindings?timeout=10s", TLS:(*tls.ConnectionState)(0x40023d32c0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:"/sinkbindings", ctx:(*context.valueCtx)(0x4002390d20), pat:(*http.pattern)(0x400041fec0), matches:[]string(nil), otherValues:map[string]string(nil)}
93	2026-01-16T00:18:42.774Z	eventing-webhook-6d99b99cff-b6wdt	info	remote admission controller audit annotations=map[string]string(nil)
94	2026-01-16T00:18:42.786Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
95	2026-01-16T00:18:42.787Z	eventing-webhook-6d99b99cff-b6wdt	info	Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0x400228b170), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, /"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"18271"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*request.BodyWrapper)(0x4001d9c0c0), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:18271, TransferEncoding:[]string(nil), Close:false, Host:"eventing-webhook.knative-eventing.svc:443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"192.168.79.91:34132", RequestURI:"/sinkbindings?timeout=10s", TLS:(*tls.ConnectionState)(0x40023d2000), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:"/sinkbindings", ctx:(*context.valueCtx)(0x4001c08e70), pat:(*http.pattern)(0x400041fec0), matches:[]string(nil), otherValues:map[string]string(nil)}
96	2026-01-16T00:18:42.788Z	eventing-webhook-6d99b99cff-b6wdt	info	remote admission controller audit annotations=map[string]string(nil)
97	2026-01-16T00:18:42.810Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
98	2026-01-16T00:18:42.824Z	eventing-webhook-6d99b99cff-b6wdt	info	Reconcile succeeded
99	2026-01-16T00:18:42.836Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
100	2026-01-16T00:18:42.861Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers
101	2026-01-16T00:18:42.891Z	eventing-controller-5649bc7769-4tpjm	info	Shutting down workers

[darkgnotic]

Update: The missing events did eventually arrive!

But about an hour later ...

So I guess my next questions are:

• whether this type of delay is expected
• what can I investigate or tune to avoid such large delays?

[darkgnotic]

I may have found the problem. Even though I'm running a jetstream cluster with 3 replicas, the default NatsJetstreamChannel is only configured with 1 replica. I've configured the NatsJetstreamChannel to use all 3 replicas. Hopefully this fixes the issue. 🤞