Skip to content

Commit acea43f

Browse files
danegstaCopilot
andauthored
[release/13.6] Keep DCP watches draining and close abandoned connections (#21002)
## Description Backport of #20972 to release/13.6. Aspire can leave a DCP watch open without an active reader when watch setup is canceled or resource handlers pause event consumption. An unread HTTP/2 watch can block DCP watcher publication and resource mutations, leaving resource operations stalled. This backport explicitly owns and closes watch responses and independently drains events without changing resource event ordering or lifecycle callback dispatch. The existing five-minute reconnection safeguard remains in place. Related to [#20609](#20609) and [microsoft/dcp#286](microsoft/dcp#286). These are client-side fixes, not a replacement for API-server slow-consumer handling. No dependency or public API changes. ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [x] Yes - [ ] No - Did you add public API? - [ ] Yes - If yes, did you have an API Review for it? - [ ] Yes - [ ] No - Did you add `<remarks />` and `<code />` elements on your triple slash comments? - [ ] Yes - [ ] No - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - If yes, have you done a threat model and had a security review? - [ ] Yes - [ ] No - [x] No ## Customer Impact Customers whose DCP watch setup is canceled or whose resource handlers stall event consumption can experience blocked resource updates and mutations. This fix keeps the watch reader draining and closes abandoned connections while preserving event ordering and lifecycle dispatch. ## Testing On the resolved release/13.6 implementation, `.\restore.cmd` passed. The following Windows run passed: **364 tests, 0 failures, 1 Linux-only platform skip**. Coverage includes watch cancellation/disposal, paused consumers, ordering across reconnections, periodic restart, framing/malformed records, Status handling, retryable HTTP statuses, existing release timeout behavior, and DCP executor lifecycle behavior. ```powershell $env:MSBUILDTERMINALLOGGER='false' dotnet test --project .\tests\Aspire.Hosting.Tests\Aspire.Hosting.Tests.csproj --no-launch-profile -- --filter-class '*.KubernetesServiceTests' --filter-class '*.DcpExecutorTests' --filter-class '*.PeriodicRestartAsyncEnumerableTests' --filter-not-trait 'quarantined=true' --filter-not-trait 'outerloop=true' ``` Source-PR-only evidence: the original head passed 367 focused tests with one Windows platform skip, all PR checks, and Linux dogfood with concurrent restart rounds, exit-7 recovery, and mutations after more than five minutes idle. That dogfood was not repeated on this release branch; no local E2E tests were run. ## Risk Medium. No public API changes, but the fix changes core watch concurrency and HTTP response lifetimes. Lossless buffering preserves events; a persistently blocked handler can accumulate queued events instead of backpressuring DCP. Release-specific API timeout and create behavior are unchanged. ## Regression? Unknown — no specific regression introduction or affected release was established. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4e69fd0-651a-47cb-aeed-cff843f6aecc
1 parent de0b8eb commit acea43f

5 files changed

Lines changed: 1250 additions & 47 deletions

File tree

‎src/Aspire.Hosting/Dcp/DcpKubernetesClient.cs‎

Lines changed: 246 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,16 @@
11
// Licensed to the .NET Foundation under one or more agreements.
22
// The .NET Foundation licenses this file to you under the MIT license.
33

4+
using System.Buffers;
5+
using System.IO.Pipelines;
46
using System.Net;
57
using System.Net.Http.Headers;
8+
using System.Text;
9+
using System.Text.Json;
610
using Aspire.Hosting.Dcp.Model;
711
using k8s;
812
using k8s.Autorest;
13+
using k8s.Models;
914

1015
namespace Aspire.Hosting.Dcp;
1116

@@ -20,6 +25,67 @@ public DcpKubernetesClient(KubernetesClientConfiguration config, params Delegati
2025
{
2126
}
2227

28+
/// <summary>
29+
/// Opens a DCP resource watch without waiting for its first event.
30+
/// This avoids KubernetesClient's setup-cancellation path that can leave an unread response open.
31+
/// </summary>
32+
/// <typeparam name="T">The resource type returned by watch events.</typeparam>
33+
/// <param name="group">The Kubernetes API group.</param>
34+
/// <param name="version">The resource API version.</param>
35+
/// <param name="plural">The plural resource name, e.g. "executables".</param>
36+
/// <param name="namespaceParameter">The namespace, or null/empty for a cluster-scoped watch.</param>
37+
/// <param name="connectionCancellationToken">Closes the watch response on cancellation, even before enumeration starts.</param>
38+
/// <param name="cancellationToken">Cancels connection establishment only; does not affect the returned stream.</param>
39+
/// <returns>
40+
/// A single-use event stream, returned once response headers arrive.
41+
/// Ending enumeration closes its HTTP response.
42+
/// </returns>
43+
public async Task<IAsyncEnumerable<(WatchEventType, T)>> CreateWatchAsync<T>(
44+
string group,
45+
string version,
46+
string plural,
47+
string? namespaceParameter,
48+
CancellationToken connectionCancellationToken,
49+
CancellationToken cancellationToken)
50+
{
51+
var collectionPath = string.IsNullOrEmpty(namespaceParameter)
52+
? $"apis/{group}/{version}/{plural}"
53+
: $"apis/{group}/{version}/namespaces/{Uri.EscapeDataString(namespaceParameter)}/{plural}";
54+
using var request = new HttpRequestMessage(HttpMethod.Get, new Uri(BaseUri, $"{collectionPath}?watch=true"));
55+
request.Version = HttpVersion.Version20;
56+
if (Credentials is not null)
57+
{
58+
await Credentials.ProcessHttpRequestAsync(request, cancellationToken).ConfigureAwait(false);
59+
}
60+
61+
// KubernetesClient's list-and-watch adapter peeks the first event during setup and
62+
// can abandon its reader when that wait is canceled. Own the raw response instead.
63+
// https://github.com/microsoft/dcp/issues/286
64+
var response = await HttpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
65+
try
66+
{
67+
cancellationToken.ThrowIfCancellationRequested();
68+
if (!response.IsSuccessStatusCode)
69+
{
70+
var content = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
71+
throw new HttpOperationException($"Operation returned an invalid status code '{response.StatusCode}', response body {content}")
72+
{
73+
Request = new HttpRequestMessageWrapper(request, string.Empty),
74+
Response = new HttpResponseMessageWrapper(response, content)
75+
};
76+
}
77+
78+
var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
79+
cancellationToken.ThrowIfCancellationRequested();
80+
return new WatchConnection<T>(response, stream, connectionCancellationToken);
81+
}
82+
catch
83+
{
84+
response.Dispose();
85+
throw;
86+
}
87+
}
88+
2389
/// <summary>
2490
/// Asynchronously reads a sub-resource from a Kubernetes resource as a stream.
2591
/// </summary>
@@ -148,6 +214,186 @@ public async Task<ApiServerExecution> PatchExecutionDocumentAsync(
148214
}
149215
}
150216

217+
private sealed class WatchConnection<T> : IAsyncEnumerable<(WatchEventType, T)>
218+
{
219+
private readonly HttpResponseMessage _response;
220+
private readonly PipeReader _reader;
221+
private readonly CancellationToken _connectionToken;
222+
private readonly CancellationTokenRegistration _cancellationRegistration;
223+
private bool _firstLine = true;
224+
225+
public WatchConnection(HttpResponseMessage response, Stream stream, CancellationToken cancellationToken)
226+
{
227+
_response = response;
228+
_reader = PipeReader.Create(stream);
229+
_connectionToken = cancellationToken;
230+
// A canceled body read does not necessarily reset an HTTP/2 stream. Close the
231+
// response even when enumeration is paused in the consumer or has not started.
232+
_cancellationRegistration = cancellationToken.Register(static state => ((HttpResponseMessage)state!).Dispose(), response);
233+
}
234+
235+
public async IAsyncEnumerator<(WatchEventType, T)> GetAsyncEnumerator(CancellationToken cancellationToken = default)
236+
{
237+
using var response = _response;
238+
using var cancellationRegistration = _cancellationRegistration;
239+
using var consumerCancellation = cancellationToken.Register(static state => ((HttpResponseMessage)state!).Dispose(), response);
240+
using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(_connectionToken, cancellationToken);
241+
var receivedEvent = false;
242+
243+
try
244+
{
245+
while (true)
246+
{
247+
string? line;
248+
bool terminated;
249+
try
250+
{
251+
(line, terminated) = await ReadEventLineAsync(linkedCts.Token).ConfigureAwait(false);
252+
}
253+
catch (Exception) when (linkedCts.IsCancellationRequested)
254+
{
255+
// Disposing the response can surface as IOException or ObjectDisposedException.
256+
// Preserve the original token so the periodic-restart helper recognizes its deadline.
257+
_connectionToken.ThrowIfCancellationRequested();
258+
cancellationToken.ThrowIfCancellationRequested();
259+
throw;
260+
}
261+
catch (IOException ex)
262+
{
263+
// An HTTP/2 reset or connection loss interrupts the watch body without
264+
// necessarily canceling either token. Use the existing watch retry path.
265+
_connectionToken.ThrowIfCancellationRequested();
266+
cancellationToken.ThrowIfCancellationRequested();
267+
268+
throw new EndOfStreamException("DCP watch connection was interrupted.", ex);
269+
}
270+
271+
_connectionToken.ThrowIfCancellationRequested();
272+
cancellationToken.ThrowIfCancellationRequested();
273+
if (line is null)
274+
{
275+
if (!receivedEvent)
276+
{
277+
// An empty response is a failed watch, not a successful short poll.
278+
// Surface it to the retry pipeline rather than reconnecting in a tight loop.
279+
throw new EndOfStreamException("DCP watch ended before returning an event.");
280+
}
281+
282+
yield break;
283+
}
284+
285+
if (DeserializeEvent(line, terminated) is { } watchEvent)
286+
{
287+
receivedEvent = true;
288+
yield return watchEvent;
289+
}
290+
}
291+
}
292+
finally
293+
{
294+
await _reader.CompleteAsync().ConfigureAwait(false);
295+
}
296+
}
297+
298+
private async ValueTask<(string? Line, bool Terminated)> ReadEventLineAsync(CancellationToken cancellationToken)
299+
{
300+
SequencePosition? searchedTo = null;
301+
while (true)
302+
{
303+
var result = await _reader.ReadAsync(cancellationToken).ConfigureAwait(false);
304+
var buffer = result.Buffer;
305+
var unsearched = searchedTo is { } position ? buffer.Slice(position) : buffer;
306+
if (unsearched.PositionOf((byte)'\n') is { } newline)
307+
{
308+
var line = DecodeLine(buffer.Slice(0, newline));
309+
_reader.AdvanceTo(buffer.GetPosition(1, newline));
310+
311+
return (line, true);
312+
}
313+
if (result.IsCompleted)
314+
{
315+
var line = buffer.IsEmpty ? null : DecodeLine(buffer);
316+
_reader.AdvanceTo(buffer.End);
317+
318+
return (line, false);
319+
}
320+
321+
// No bytes are consumed until a complete record is found, so this position
322+
// stays valid across reads and avoids rescanning the retained prefix.
323+
searchedTo = buffer.IsEmpty ? null : buffer.End;
324+
_reader.AdvanceTo(buffer.Start, buffer.End);
325+
}
326+
}
327+
328+
private string DecodeLine(in ReadOnlySequence<byte> buffer)
329+
{
330+
var line = Encoding.UTF8.GetString(buffer);
331+
if (_firstLine)
332+
{
333+
_firstLine = false;
334+
if (line.Length > 0 && line[0] == '\uFEFF')
335+
{
336+
line = line[1..];
337+
}
338+
}
339+
340+
return line;
341+
}
342+
343+
private static (WatchEventType, T)? DeserializeEvent(string line, bool terminated)
344+
{
345+
// Kubernetes watches contain newline-delimited events:
346+
// {"type":"MODIFIED","object":{"kind":"Executable","metadata":{...},...}}
347+
// An ERROR can contain a Status instead; BOOKMARK objects contain only metadata.
348+
// https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes
349+
if (!terminated)
350+
{
351+
// A missing newline does not make a complete payload such as null truncated.
352+
// Validate syntax separately so resource shape errors remain non-retryable.
353+
try
354+
{
355+
using var document = JsonDocument.Parse(line);
356+
}
357+
catch (JsonException ex)
358+
{
359+
var reader = new Utf8JsonReader(Encoding.UTF8.GetBytes(line), isFinalBlock: false, state: default);
360+
if (!JsonDocument.TryParseValue(ref reader, out var document))
361+
{
362+
// EOF can leave a partial record such as {"type":"MODIFIED","object":
363+
throw new EndOfStreamException("DCP watch ended in an incomplete event.", ex);
364+
}
365+
document.Dispose();
366+
throw;
367+
}
368+
}
369+
370+
var envelope = KubernetesJson.Deserialize<Watcher<KubernetesObject>.WatchEvent>(line);
371+
if (envelope?.Object is null)
372+
{
373+
throw new JsonException("DCP watch event does not contain an object.");
374+
}
375+
if (envelope.Object.Kind == "Status")
376+
{
377+
var status = KubernetesJson.Deserialize<Watcher<V1Status>.WatchEvent>(line).Object;
378+
if (status.Status == "Failure")
379+
{
380+
throw new KubernetesException(status);
381+
}
382+
if (status.Status != "Success")
383+
{
384+
throw new JsonException("DCP watch Status does not report Success or Failure.");
385+
}
386+
387+
// A successful Status is not a resource update and must not be deserialized as T.
388+
return null;
389+
}
390+
391+
var watchEvent = KubernetesJson.Deserialize<Watcher<T>.WatchEvent>(line);
392+
393+
return (watchEvent.Type, watchEvent.Object);
394+
}
395+
}
396+
151397
private sealed class QueryBuilder
152398
{
153399
private readonly List<string> _parameters = new List<string>();

‎src/Aspire.Hosting/Dcp/DcpPipelineBuilder.cs‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
// Licensed to the .NET Foundation under one or more agreements.
22
// The .NET Foundation licenses this file to you under the MIT license.
33

4+
using System.Net;
5+
using k8s;
6+
using k8s.Autorest;
47
using Microsoft.Extensions.Logging;
58
using Polly;
69
using Polly.Retry;
@@ -65,7 +68,10 @@ public static ResiliencePipeline BuildWatchResourcePipeline(ILogger logger)
6568
{
6669
var retryUntilCancelled = new RetryStrategyOptions()
6770
{
68-
ShouldHandle = new PredicateBuilder().HandleInner<EndOfStreamException>(),
71+
ShouldHandle = new PredicateBuilder()
72+
.HandleInner<EndOfStreamException>()
73+
.HandleInner<KubernetesException>(ex => IsRetryableWatchStatusCode(ex.Status.Code))
74+
.HandleInner<HttpOperationException>(ex => IsRetryableWatchStatusCode((int)ex.Response.StatusCode)),
6975
BackoffType = DelayBackoffType.Exponential,
7076
MaxRetryAttempts = int.MaxValue,
7177
UseJitter = true,
@@ -85,4 +91,13 @@ public static ResiliencePipeline BuildWatchResourcePipeline(ILogger logger)
8591
var pipeline = new ResiliencePipelineBuilder().AddRetry(retryUntilCancelled).Build();
8692
return pipeline;
8793
}
94+
95+
private static bool IsRetryableWatchStatusCode(int? statusCode) => statusCode is
96+
(int)HttpStatusCode.Gone or
97+
(int)HttpStatusCode.RequestTimeout or
98+
(int)HttpStatusCode.TooManyRequests or
99+
(int)HttpStatusCode.InternalServerError or
100+
(int)HttpStatusCode.BadGateway or
101+
(int)HttpStatusCode.ServiceUnavailable or
102+
(int)HttpStatusCode.GatewayTimeout;
88103
}

0 commit comments

Comments
 (0)