Fix Key History Dirty Read Bug#449
Merged
dillongeorge merged 2 commits intofacebook:mainfrom Aug 23, 2024
Merged
Conversation
be7b21c to
fe0dc4f
Compare
dillongeorge
commented
Aug 23, 2024
| } | ||
|
|
||
| if start_version == 0 { | ||
| if start_version == 0 || end_version == 0 { |
Contributor
Author
There was a problem hiding this comment.
Realistically, end_version should never be 0 assuming start_version isn't (i.e. this will short circuit). I'm opting to be a bit more explicit here to err on the side of caution given the bug we're fixing :)
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #449 +/- ##
==========================================
- Coverage 88.61% 88.03% -0.58%
==========================================
Files 39 38 -1
Lines 9109 8283 -826
==========================================
- Hits 8072 7292 -780
+ Misses 1037 991 -46 ☔ View full report in Codecov by Sentry. |
added 2 commits
August 23, 2024 11:42
**Bug** In the `key_history` API exposed in `directory.rs`, it is possible that states read from storage are not fully committed yet (i.e. the transaction has not been committed), but the associated epoch in the states is higher than what is seen in the `aZKS` entry read from storage. In fact, we explicitly account for that scenario when inspecting versions across the states: https://github.com/facebook/akd/blob/main/akd/src/directory.rs#L484-L490. Prior to this patch, we always initiated the `end_version` variable at 0. However, the `end_version` variable should **never** be 0 since it will panic in downstream API calls: https://github.com/facebook/akd/blob/main/akd_core/src/utils.rs#L182-L184. The bug in this case is simply that we default `end_version` to 0, but do not update it in the event that we have performed a dirty read for states. To resolve the bug, we default `end_version` to the same value as `start_version`. **Verification** Prior to updating `directory.rs`, the added tests to repro cases where a dirty read is performed and a newer epoch is returned as part of the `KeyData` fails with the expected panic: ``` failures: ---- tests::test_errors::test_key_history_dirty_reads_experimental_config stdout ---- thread 'tests::test_errors::test_key_history_dirty_reads_experimental_config' panicked at akd_core/src/utils.rs:183:9: find_max_index_in_skiplist called with input less than smallest element of MARKER_VERSION_SKIPLIST note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- tests::test_errors::test_key_history_dirty_reads_whatsapp_v1_config stdout ---- thread 'tests::test_errors::test_key_history_dirty_reads_whatsapp_v1_config' panicked at akd_core/src/utils.rs:183:9: find_max_index_in_skiplist called with input less than smallest element of MARKER_VERSION_SKIPLIST failures: tests::test_errors::test_key_history_dirty_reads_experimental_config tests::test_errors::test_key_history_dirty_reads_whatsapp_v1_config test result: FAILED. 100 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.36s ``` After updating `directory.rs`, the panic no longer occurs.
fe0dc4f to
5818ea1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
In the
key_historyAPI exposed indirectory.rs, it is possible that states read from storage are not fully committed yet (i.e. the transaction has not been committed), but the associated epoch in the states is higher than what is seen in theaZKSentry read from storage. In fact, we explicitly account for that scenario when inspecting versions across the states: https://github.com/facebook/akd/blob/main/akd/src/directory.rs#L484-L490.Prior to this patch, we always initiated the
end_versionvariable at 0. However, theend_versionvariable should never be 0 since it will panic in downstream API calls: https://github.com/facebook/akd/blob/main/akd_core/src/utils.rs#L182-L184. The bug in this case is simply that we defaultend_versionto 0, but do not update it in the event that we have performed a dirty read for states.To resolve the bug, we default
end_versionto the same value asstart_version.Verification
Prior to updating
directory.rs, the added tests to repro cases where a dirty read is performed and a newer epoch is returned as part of theKeyDatafails with the expected panic:After updating
directory.rs, the panic no longer occurs.