[Client encryption] Add streaming JSON processing support to feed iterators #5478
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces JsonProcessorPropertyBag to centralize handling of JsonProcessor overrides using RequestOptions.Properties. Updates encryption and decryption workflows to support processor selection, adds related tests and helpers, and refactors code to use the new override mechanism for both production and test scenarios.
Introduces CosmosDiagnosticsContext for lightweight diagnostics and scope tracking, with ActivitySource integration for telemetry. Updates EncryptionProcessor to record diagnostic scopes and enforce streaming mode restrictions. Adds comprehensive unit tests for diagnostics context and processor, and merges stream processor end-to-end tests into MdeCustomEncryptionTests.
Moved diagnostics scope creation for MDE decryption to cover the entire decryption logic in EncryptionProcessor. Updated tests to assert scope presence/absence more robustly and removed conditional compilation for ENCRYPTION_CUSTOM_PREVIEW. Project files now always define ENCRYPTION_CUSTOM_PREVIEW constant.
Introduces direct serialization to output streams via WriteToStream to reduce intermediate memory allocations. Refactors decryption logic to streamline processor selection and diagnostics scope naming, and adds EncryptionDiagnostics constants for improved diagnostics context management. Updates MDE encryption processor to support direct stream handling and processor selection for .NET 8+ preview builds.
Introduces TestEncryptorFactory to centralize and simplify the creation of mock Encryptor and DataEncryptionKey instances in tests, reducing repetitive Moq setup code. Updates all relevant test classes to use this factory. Refactors MdeEncryptionProcessor to improve fallback handling for legacy and unencrypted streams, ensuring correct stream positioning and graceful fallback to legacy decryption paths.
Replaces separate diagnostic prefixes for JSON processor selection with a unified prefix in EncryptionDiagnostics and updates all usages and tests accordingly. Adds documentation for the JSON processor property bag override and introduces new tests for large payloads, concurrency, and cancellation scenarios with the streaming JSON processor.
Adds validation to ensure that streaming encryption only processes JSON documents with an object as the root element. Root arrays and other root types are now explicitly rejected, improving contract enforcement and error handling.
Refactored root validation logic to ensure streaming encryption only accepts JSON documents with an object root. Added tests to verify rejection of root arrays and primitive values, improving contract enforcement and error messaging.
Expanded EncryptionProcessorDiagnosticsTests to cover diagnostics scope emission for various decryption paths, including MDE payloads, provided-output, stream override, malformed JSON, duplicate scope prevention, null input, cancellation, and encryption. Added a fake MDE DataEncryptionKey implementation and a SlowCancelableStream helper for cancellation tests. Conditional assertions ensure correct scope behavior for NET8 preview builds.
Introduces IMdeJsonProcessorAdapter and concrete adapters for Newtonsoft and Stream processors to unify JSON processing logic in MdeEncryptionProcessor. Diagnostic scope tracking for encryption and decryption is improved, and related tests are updated to verify scope tracking for both processors. Legacy logic for processor selection and property inspection is moved into the respective adapters for better maintainability.
Unifies and updates diagnostics scope naming for encryption and decryption selection, removing legacy/impl scopes and ensuring only one selection scope is emitted per operation. Refactors MDE processor logic to directly handle Newtonsoft and Stream processors, improving fallback and error handling. Adds comprehensive unit tests for NewtonsoftAdapter and StreamAdapter, covering encryption, decryption, legacy/unencrypted payloads, and diagnostics scope assertions.
…rypted stream used; sync request options with processor; add legacy override guard and fallback JObject decrypt
Added a test to verify that explicitly overriding the JSON processor with Newtonsoft maintains the expected encryption/decryption behavior. Also added assertions to change feed processor tests to ensure both documents are processed and the processor does not time out.
Changed expected diagnostics.Scopes.Count from 1 to 0 in multiple StreamAdapterTests to reflect updated behavior. Ensures tests align with current diagnostics tracking implementation.
Introduces EncryptionRequestOptionsExperimental for configuring the JSON processor pipeline via request options, including helper methods and tests. Refactors JsonProcessorPropertyBag for improved property handling and updates test helpers to use the new API.
Replaces static calls to RequestOptionsPropertiesExtensions with instance extension method calls on RequestOptions throughout the codebase and tests. This improves code readability and aligns with extension method usage patterns.
Simplifies and unifies decryption logic by removing preprocessor directives and consolidating JsonProcessor selection. Refactors MdeEncryptionProcessor to use switch expressions and helper methods for both stream and Newtonsoft processors, improving maintainability and clarity. Ensures consistent handling of legacy and MDE-encrypted documents, and introduces utility methods for stream position management and result writing.
Simplified the CreateStreamOptions method in StreamProcessorConcurrencyAndCancellationTests by removing compression-related parameters and settings. Also added a missing using directive in MdeEncryptionProcessorTests.
Renamed StreamAdapter to SystemTextJsonStreamAdapter and RequestOptionsPropertiesExtensions to JsonProcessorRequestOptionsExtensions for improved clarity and consistency. Updated all references in source and test files to use the new names, ensuring alignment with the JSON processor selection logic.
Replaced explicit ArgumentNullException checks with ArgumentValidation.ThrowIfNull across multiple classes for consistency and code simplification. Also fixed equality checks in MdeEncryptionTests to use '==' instead of '!=' for certain properties.
…ream-switch' into feature/encryption-feed-iterator-rework
MartinSarkany
changed the title
MartinSarkany
requested review from
FabianMeiswinkel,
Pilchie,
kirankumarkolli and
sboshra
as code owners
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Template
Description
This PR extends support for streaming JSON processing from CRUD operations introduced in #5403 to feed iterators.
API Changes
DecryptableItemimplementsIAsyncDisposableand exposesDisposeAsync()for resource cleanup in streaming scenarios.EncryptionContainerExtensionsgainsUseStreamingJsonProcessingByDefault()to change default JSON processing mode inEncryptionContainer.#4678
Type of change
Please delete options that are not relevant.