Allow checking the entire ProducerMessage in the mock producers#1956
Merged
dnwe merged 1 commit intoIBM:masterfrom Jun 9, 2021
Merged
Allow checking the entire ProducerMessage in the mock producers#1956dnwe merged 1 commit intoIBM:masterfrom
ProducerMessage in the mock producers#1956dnwe merged 1 commit intoIBM:masterfrom
Conversation
This is done by introducing `MessageChecker`, a function which validates an entire `ProducerMessage`, and reimplementing the existing expectation interface on top of that. Since `producerExpectation` was not a public API, the only visible effect of this to users of the mock producers should be slightly different error messages when encoding fails. To better mimic the real producers’ behavior, this also runs partitioning before passing the message to the checker. By default all topics have 32 partitions but this can be configured per-topic in either producer via the embedded `TopicConfig`. (This does not allow mocking unavailable partitions; I don’t see a use case for this which can’t be better handled by unit testing the partitioner.) This may affect existing tests if someone was passing a configuration with a `Partitioner` that would fail on their test messages, but it seems more likely they want to know if this is the case (we did). This is useful for (at least) two reasons, - I would often like to test more than just the message value; services may need to produce to different topics or with specific partitioning schemes. In some cases valuable data may only be in the message key, or I want to test that a header and message body agree. - Since the topic usually also corresponds to the data format, exposing the topic makes it easier to collect messages of different types across different topics which may appear in known quantities but non-deterministic orders. Without the topic this is only possible by trying to content-sniff the format from the value, which leads to weaker and more fragile tests.
Contributor
Author
|
After filling out the CLA it asked me to re-run the CI check but don't see how to do that. I don't think I can until I'm approved to run workflows. |
Contributor
Author
|
Sorry, I can't see anything actually broken in the CI, nor a way to trigger a retry myself. Given exit code 28 in the 1.16/2.7.1 case it seems like curl might have timed out trying to install |
dnwe
approved these changes
Jun 9, 2021
Collaborator
dnwe
left a comment
There was a problem hiding this comment.
Thanks for the first-time contribution @joewreschnig !
These changes look good to me — thank you for the detailed description
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is done by introducing
MessageChecker, a function which validatesan entire
ProducerMessage, and reimplementing the existing expectationinterface on top of that. Since
producerExpectationwas not a publicAPI, the only visible effect of this to users of the mock producers
should be slightly different error messages when encoding fails.
To better mimic the real producers’ behavior, this also runs
partitioning before passing the message to the checker. By default all
topics have 32 partitions but this can be configured per-topic in either
producer via the embedded
TopicConfig. (This does not allow mockingunavailable partitions; I don’t see a use case for this which can’t be
better handled by unit testing the partitioner.) This may affect
existing tests if someone was passing a configuration with a
Partitionerthat would fail on their test messages, but it seems morelikely they want to know if this is the case (we did).
This is useful for (at least) two reasons,
I would often like to test more than just the message value; services
may need to produce to different topics or with specific partitioning
schemes. In some cases valuable data may only be in the message key,
or I want to test that a header and message body agree.
Since the topic usually also corresponds to the data format, exposing
the topic makes it easier to collect messages of different types
across different topics which may appear in known quantities but
non-deterministic orders. Without the topic this is only possible by
trying to content-sniff the format from the value, which leads to
weaker and more fragile tests.