Change feed: Adds id and pk to ChangeFeedMetadata for delete operations #5191

dibahlfi · 2025-05-09T19:13:47Z

Adds id and partition key to ChangeFeedMetadata. This is populated for delete operations using all versions and deletes change feed mode so that when customers get the delete, they know what was deleted. Note it doesn't have the value of the document that was deleted only the id and partiton key through which a document can be uniquely identified.
Please note that user provided serializer is not being used.

New feature (non-breaking change which adds functionality)

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs

kirankumarkolli · 2025-05-13T16:21:47Z

Please update PR description with usage samples and also call out that user-serializer is not used.

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs

kirankumarkolli · 2025-05-13T19:45:11Z

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs

    /// The metadata of a change feed resource with <see cref="ChangeFeedMode"/> is initialized to <see cref="ChangeFeedMode.AllVersionsAndDeletes"/>.
    /// </summary>
    [System.Text.Json.Serialization.JsonConverter(typeof(ChangeFeedMetadataConverter))]
+    [JsonConverter(typeof(ChangeFeedMetadataNewtonSoftConverter))]


Why is newtonsoftconverter needed?

because we added list of tuple<string, object> for partition key in the metadata which the default JsonProperty is not able to deserialize

True all changeFeed is applied with user-serializer.

Thinking loud here, for non-primitive types respective de-serializers are needed.
For primitive types may be it not needed., thoughts?

One question I have is what is the purpose of moving all the annotations from individual properties to the class?

One drawback I see of this approach is that we loose the capability to do different null value handling behavior for properties

I ran into it and what I observed was when you have a class-level JsonConverter attribute, the converter takes full control. Any property-level annotations (like [JsonPropertyName], [JsonProperty], etc.) are bypassed.

kirankumarkolli

What's the use-case for NewtonSoftConverter needed?

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs

...t.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataNewtonSoftConverter.cs

JensSchadron · 2025-06-29T14:45:22Z

Out-of-curiosity, is there anything holding up this PR?

dibahlfi · 2025-09-10T16:08:55Z

Please update PR description with usage samples and also call out that user-serializer is not used.

Justine already has the samples ready and she will update them once the PR goes out.

dibahlfi · 2025-09-10T16:12:45Z

What's the use-case for NewtonSoftConverter needed?

because of the introduction of List<(string, object)>for which a custom serializator was needed to serialize/deserialize.

dibahlfi · 2025-09-10T16:14:03Z

ngeFeedMetadata. This is populated for delete operations using all versions and deletes change feed mode. Please note that user provided serializer is not being used.

New feature (non-breaking change which adds functionality)

Sorry somehow it slipped through the cracks. will get this going.

dibahlfi · 2025-09-10T17:23:49Z

What's the use-case for NewtonSoftConverter needed?
because of the introduction of List<(string, object)>for which a custom serializator was needed to serialize/deserialize.

FabianMeiswinkel

LGTM - Thanks

JensSchadron · 2025-10-08T09:41:43Z

Given that this received an approval last week, is there anything still blocking this PR? ( @kirankumarkolli )
Really looking forward to this becoming available in the (preview?) SDK

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataConverter.cs

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataConverter.cs

...t.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataNewtonSoftConverter.cs

ananth7592 · 2025-10-17T21:16:42Z

/azp run

azure-pipelines · 2025-10-17T21:16:49Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

yash2710 · 2025-10-17T21:23:44Z

/azp run

azure-pipelines · 2025-10-17T21:23:51Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

ananth7592 · 2025-10-17T21:24:59Z

/azp run

azure-pipelines · 2025-10-17T21:25:06Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

kirankumarkolli · 2025-10-29T19:18:25Z

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs

@@ -16,43 +16,49 @@ namespace Microsoft.Azure.Cosmos
    /// The metadata of a change feed resource with <see cref="ChangeFeedMode"/> is initialized to <see cref="ChangeFeedMode.AllVersionsAndDeletes"/>.
    /// </summary>
    [System.Text.Json.Serialization.JsonConverter(typeof(ChangeFeedMetadataConverter))]


On overall approach: new possible approach is to leverage https://learn.microsoft.com/en-us/dotnet/api/system.text.json.serialization.jsonincludeattribute?view=net-9.0&viewFallbackFrom=netstandard-2.0 (assuming that we can use it with-out rebasing to latest .NET versions.

/cc: @yash2710 , @dibahlfi

kirankumarkolli · 2025-11-06T19:49:49Z

Failed PreviewContractChanges [101 ms]
  Error Message:
   Assert.Fail failed. Public API has changed. If this is expected, then run (EnlistmentRoot)\UpdateContracts.ps1 . To see the differences run the update script and use git diff.
  Stack Trace:
     at Microsoft.Azure.Cosmos.Tests.Contracts.ContractEnforcement.ValidateJsonAreSame(String baselineJson, String currentJson) in D:\a\_work\1\s\Microsoft.Azure.Cosmos\tests\Microsoft.Azure.Cosmos.Tests\Contracts\ContractEnforcement.cs:line 542
   at Microsoft.Azure.Cosmos.Tests.Contracts.ContractEnforcement.ValidateContract(String dllName, ContractType contractType, String baselinePattern, String breakingChangesPattern, String officialBaselinePattern) in D:\a\_work\1\s\Microsoft.Azure.Cosmos\tests\Microsoft.Azure.Cosmos.Tests\Contracts\ContractEnforcement.cs:line 423
   at Microsoft.Azure.Cosmos.Tests.Contracts.ContractEnforcementTests.PreviewContractChanges() in D:\a\_work\1\s\Microsoft.Azure.Cosmos\tests\Microsoft.Azure.Cosmos.Tests\Contracts\ContractEnforcementTests.cs:line 22

  Standard Output Messages:

Contract changes needs refresh

…ns (#5470) ## Description Adds id and partition key to ChangeFeedMetadata. This is populated for delete operations using all versions all deletes change feed mode so that when customers get the delete, they know what was deleted. Note it doesn't have the value of the document that was deleted only the id and partition key through which a document can be uniquely identified. Please note that user provided serializer is not being used. This PR includes commits from #5191 and additional simplifications and unit test changes ## Type of change New feature (non-breaking change which adds functionality) ## Basic Usage ### Single Partition Key Example ```csharp using Microsoft.Azure.Cosmos; public class Product { public string id { get; set; } public string categoryId { get; set; } public string name { get; set; } public decimal price { get; set; } } // Create Change Feed Processor with All Versions and Deletes mode ChangeFeedProcessor processor = container .GetChangeFeedProcessorBuilderWithAllVersionsAndDeletes<Product>( processorName: "productProcessor", onChangesDelegate: async (context, changes, cancellationToken) => { foreach (ChangeFeedItem<Product> change in changes) { switch (change.Metadata.OperationType) { case ChangeFeedOperationType.Create: case ChangeFeedOperationType.Replace: Console.WriteLine($"Document: {change.Current.id}"); break; case ChangeFeedOperationType.Delete: // Access deleted document's id and partition key from metadata Console.WriteLine($"Deleted Document Id: {change.Metadata.Id}"); // PartitionKey is a dictionary: key = path name, value = partition key value string categoryId = change.Metadata.PartitionKey["categoryId"]?.ToString(); Console.WriteLine($"Partition Key: {categoryId}"); // Check if deleted due to TTL expiration if (change.Metadata.IsTimeToLiveExpired) { Console.WriteLine("Deleted via TTL expiration"); } // Cleanup related data await RemoveFromCacheAsync(change.Metadata.Id, categoryId); break; } } }) .WithInstanceName("instance1") .WithLeaseContainer(leaseContainer) .Build(); await processor.StartAsync(); ``` ### Hierarchical Partition Key Example ```csharp public class UserSession { public string id { get; set; } public string tenantId { get; set; } public string userId { get; set; } public string sessionId { get; set; } } // Container with HPK: ["/tenantId", "/userId", "/sessionId"] ChangeFeedProcessor processor = container .GetChangeFeedProcessorBuilderWithAllVersionsAndDeletes<UserSession>( processorName: "sessionProcessor", onChangesDelegate: async (context, changes, cancellationToken) => { foreach (ChangeFeedItem<UserSession> change in changes) { if (change.Metadata.OperationType == ChangeFeedOperationType.Delete) { Console.WriteLine($"Deleted Session Id: {change.Metadata.Id}"); // PartitionKey dictionary contains all hierarchy levels in order string tenantId = change.Metadata.PartitionKey["tenantId"]?.ToString(); string userId = change.Metadata.PartitionKey["userId"]?.ToString(); string sessionId = change.Metadata.PartitionKey["sessionId"]?.ToString(); Console.WriteLine($"HPK: tenant={tenantId}, user={userId}, session={sessionId}"); await CleanupSessionAsync(tenantId, userId, sessionId); } } }) .WithInstanceName("instance1") .WithLeaseContainer(leaseContainer) .Build(); await processor.StartAsync(); ``` ### Cache Synchronization Example ```csharp ChangeFeedProcessor processor = container .GetChangeFeedProcessorBuilderWithAllVersionsAndDeletes<dynamic>( processorName: "cacheSync", onChangesDelegate: async (context, changes, cancellationToken) => { foreach (ChangeFeedItem<dynamic> change in changes) { if (change.Metadata.OperationType == ChangeFeedOperationType.Delete) { // Get id and partition key from metadata string id = change.Metadata.Id; string pk = change.Metadata.PartitionKey.Values.FirstOrDefault()?.ToString(); // Remove from cache and search index await cache.RemoveAsync(id); await searchIndex.DeleteAsync(id); Console.WriteLine($"Removed {id} from cache and index"); } else { // Update cache and index for creates/updates await cache.SetAsync(change.Current.id.ToString(), change.Current); await searchIndex.IndexAsync(change.Current); } } }) .WithInstanceName("cacheSync1") .WithLeaseContainer(leaseContainer) .Build(); await processor.StartAsync(); ``` ## Property Details ### `ChangeFeedMetadata.Id` - **Type**: `string` - **Populated**: Delete operations only (null for create/replace) - **Description**: The document id of the deleted item - **Example**: `"order-12345"` ### `ChangeFeedMetadata.PartitionKey` - **Type**: `Dictionary<string, object>` - **Populated**: Delete operations only (null for create/replace) - **Description**: Dictionary with partition key path(s) as keys and values as objects - **Key Format**: Partition key property name without leading `/` - **Value Types**: Can be string, number, boolean, or null **Examples:** Single Partition Key (`/categoryId`): ```csharp { "categoryId": "electronics" } ``` Hierarchical Partition Key (`["/tenantId", "/userId", "/sessionId"]`): ```csharp { "tenantId": "tenant123", "userId": "user456", "sessionId": "session789" } ``` Mixed Types (`["/category", "/priority", "/isActive"]`): ```csharp { "category": "electronics", "priority": 1, "isActive": true } ``` ## Notes 1. **Only for Deletes**: `Id` and `PartitionKey` are only populated when `OperationType == Delete` 2. **No Document Body**: The deleted document body is not available (Current will be null) 3. **No Custom Serializer**: User-provided serializers are not applied to these metadata properties 4. **TTL Detection**: Use `IsTimeToLiveExpired` to distinguish TTL deletes from explicit deletes 5. **Container Setup Required**: Container must have `ChangeFeedPolicy.FullFidelityRetention` configured --------- Co-authored-by: Justine Cocchi <[email protected]> Co-authored-by: Dikshi Bahl <[email protected]> Co-authored-by: dibahlfi <[email protected]> Co-authored-by: Kiran Kumar Kolli <[email protected]> Co-authored-by: ananth7592 <[email protected]>

dibahlfi requested review from a team, FabianMeiswinkel, Pilchie, adityasa, khdang, kirankumarkolli, kirillg, neildsh and sboshra as code owners May 9, 2025 19:13

kirankumarkolli reviewed May 13, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs Outdated Show resolved Hide resolved

kirankumarkolli reviewed May 13, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs Outdated Show resolved Hide resolved

kirankumarkolli reviewed May 13, 2025

View reviewed changes

kirankumarkolli requested changes May 13, 2025

View reviewed changes

JensSchadron mentioned this pull request May 27, 2025

Id and PartitionKey properties are missing on ChangeFeedMetadata class #4933

Open

kirankumarkolli reviewed May 27, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs Outdated Show resolved Hide resolved

kirankumarkolli reviewed May 27, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs Outdated Show resolved Hide resolved

kirankumarkolli reviewed May 29, 2025

View reviewed changes

...t.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataNewtonSoftConverter.cs Show resolved Hide resolved

kirankumarkolli reviewed Jun 3, 2025

View reviewed changes

...t.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataNewtonSoftConverter.cs Outdated Show resolved Hide resolved

dibahlfi closed this Sep 10, 2025

dibahlfi reopened this Sep 10, 2025

dibahlfi self-assigned this Sep 10, 2025

dibahlfi closed this Sep 10, 2025

dibahlfi reopened this Sep 10, 2025

Update ChangeFeedItem to include id and pk metadata

f629e50

dibahlfi added 3 commits September 25, 2025 19:01

fix: addressing comments

079e9aa

fix: refactoring

aa60a72

updating contract file

2658b33

dibahlfi force-pushed the users/dibahl/updateChangeFeedMetadata branch from 3daf85e to 2658b33 Compare September 26, 2025 00:03

FabianMeiswinkel previously approved these changes Oct 1, 2025

View reviewed changes

kirankumarkolli reviewed Oct 15, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataConverter.cs Outdated Show resolved Hide resolved

kirankumarkolli reviewed Oct 15, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataConverter.cs Outdated Show resolved Hide resolved

kirankumarkolli reviewed Oct 15, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataConverter.cs Outdated Show resolved Hide resolved

kirankumarkolli reviewed Oct 15, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataConverter.cs Show resolved Hide resolved

kirankumarkolli reviewed Oct 15, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs Show resolved Hide resolved

yash2710 reviewed Oct 15, 2025

View reviewed changes

Microsoft.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataConverter.cs Show resolved Hide resolved

yash2710 reviewed Oct 15, 2025

View reviewed changes

...t.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataNewtonSoftConverter.cs Outdated Show resolved Hide resolved

Merge branch 'master' into users/dibahl/updateChangeFeedMetadata

8cd1360

ananth7592 dismissed FabianMeiswinkel’s stale review via 8cd1360 October 17, 2025 21:26

dibahlfi requested a review from FabianMeiswinkel October 28, 2025 19:56

kirankumarkolli reviewed Oct 29, 2025

View reviewed changes

dibahlfi and others added 2 commits October 29, 2025 14:34

enhanced logging

8694811

Merge branch 'master' into users/dibahl/updateChangeFeedMetadata

c175e7d

This was referenced Oct 29, 2025

Proposal: Change feed metadata to use attribute level json converter #5466

Closed

Change feed: Adds id and pk to ChangeFeedMetadata for delete operations #5470

Merged

JensSchadron mentioned this pull request Nov 20, 2025

[Change Feed] ConflictResolutionTimestamp became nullable since 3.55.0 #5508

Closed

Change feed: Adds id and pk to ChangeFeedMetadata for delete operations #5191

Are you sure you want to change the base?

Change feed: Adds id and pk to ChangeFeedMetadata for delete operations #5191

Uh oh!

Conversation

dibahlfi commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kirankumarkolli commented May 13, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kirankumarkolli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JensSchadron commented Jun 29, 2025

Uh oh!

dibahlfi commented Sep 10, 2025

Uh oh!

dibahlfi commented Sep 10, 2025

Uh oh!

dibahlfi commented Sep 10, 2025

Uh oh!

dibahlfi commented Sep 10, 2025

Uh oh!

FabianMeiswinkel left a comment

Choose a reason for hiding this comment

Uh oh!

JensSchadron commented Oct 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ananth7592 commented Oct 17, 2025

Uh oh!

azure-pipelines bot commented Oct 17, 2025

Uh oh!

yash2710 commented Oct 17, 2025

Uh oh!

azure-pipelines bot commented Oct 17, 2025

Uh oh!

ananth7592 commented Oct 17, 2025

Uh oh!

azure-pipelines bot commented Oct 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kirankumarkolli commented Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

dibahlfi commented May 9, 2025 •

edited

Loading