fix(tests): stabilize flaky Hub LFS integration test #4
Conversation
|
Warning Rate limit exceeded@The-Obstacle-Is-The-Way has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 0 minutes and 3 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (1)
WalkthroughAdds a Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
tests/test_upstream_hub.py (3)
270-281: Polling helper for Hub consistency looks solid; only minor nits possibleThe retry loop around
list_repo_fileswith a bounded wait and explicitTimeoutErroris a good fit to make the test deterministic and fail loudly if the Hub is slow or inconsistent. CatchingExceptionis acceptable in this test-only helper, and the extrasleep(1)after a successful probe gives LFS a bit more buffer.If you ever need finer control, a small optional refinement would be:
- Track elapsed time via
time.monotonic()somax_waitis strictly “seconds” rather than “iterations”.- Optionally narrow the
exceptto hub-related errors to avoid masking unexpected issues, or at least log them.Not blockers; current implementation is fine for the intended use.
294-313: Well-placed waits between first push, manual upload, and second pushAdding
_wait_for_repo_ready(ds_name)after the initialpush_to_huband again after theupload_filecall is a reasonable way to avoid the LFS propagation race that was causing the flakyBadRequestError. The helper is scoped to this test class and only used here, so the additional latency is limited to this one integration test.If further flakes appear, a non-blocking follow-up could be to:
- Also use
_wait_for_repo_readyafter the secondpush_to_hubbefore asserting onlist_repo_files, to guard against any remaining backend lag.
But given the current problem description, the added waits should already address the primary race.
345-364: Symmetric guarding of the second overwrite scenario looks appropriateMirroring the same wait pattern in the second half of
test_push_dataset_dict_to_hub_overwrite_files(push → wait → upload → wait → push) keeps both branches consistent and should prevent the same LFS race when the second run changes the shard count in the opposite direction.No issues from a correctness standpoint. As with the first block, you might consider adding a final readiness wait before the concluding
load_datasetif you ever see residual flakiness, but it's not strictly necessary right now.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/test_upstream_hub.py(5 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (13)
- GitHub Check: test (unit, windows-latest, deps-minimum)
- GitHub Check: test (integration, windows-latest, deps-latest)
- GitHub Check: test (unit, ubuntu-latest, deps-minimum)
- GitHub Check: test (integration, windows-latest, deps-minimum)
- GitHub Check: test (unit, windows-latest, deps-latest)
- GitHub Check: test (integration, ubuntu-latest, deps-minimum)
- GitHub Check: test_py314_future (unit, windows-latest, deps-latest)
- GitHub Check: test (unit, ubuntu-latest, deps-latest)
- GitHub Check: test (integration, ubuntu-latest, deps-latest)
- GitHub Check: test_py314 (unit, ubuntu-latest, deps-latest)
- GitHub Check: test_py314_future (unit, ubuntu-latest, deps-latest)
- GitHub Check: test_py314 (unit, windows-latest, deps-latest)
- GitHub Check: build / build_pr_documentation
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/test_upstream_hub.py (1)
270-296: Reasonable approach to stabilize flaky Hub tests.The helper addresses the race condition with a polling + sleep strategy. A few observations:
Exception specificity: Catching all
HfHubHTTPErroris broad. Consider logging transient errors for debugging intermittent CI failures.Import placement: Moving the import to the top of the file (alongside other
huggingface_hubimports on line 15) would be more idiomatic.Delay tuning: The 1-second post-success sleep is a heuristic. If flakiness persists, this may need adjustment or a more robust verification (e.g., retrying the subsequent operation with backoff rather than pre-waiting).
Consider moving the import to the file header:
-from huggingface_hub import DatasetCard, HfApi +from huggingface_hub import DatasetCard, HfApi +from huggingface_hub.errors import HfHubHTTPErrorThen remove line 284 from within the method.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/test_upstream_hub.py(5 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (13)
- GitHub Check: test (integration, ubuntu-latest, deps-minimum)
- GitHub Check: test (unit, windows-latest, deps-latest)
- GitHub Check: test (integration, windows-latest, deps-minimum)
- GitHub Check: test (integration, windows-latest, deps-latest)
- GitHub Check: test (unit, windows-latest, deps-minimum)
- GitHub Check: test (integration, ubuntu-latest, deps-latest)
- GitHub Check: test (unit, ubuntu-latest, deps-minimum)
- GitHub Check: test (unit, ubuntu-latest, deps-latest)
- GitHub Check: test_py314_future (unit, ubuntu-latest, deps-latest)
- GitHub Check: test_py314_future (unit, windows-latest, deps-latest)
- GitHub Check: test_py314 (unit, windows-latest, deps-latest)
- GitHub Check: test_py314 (unit, ubuntu-latest, deps-latest)
- GitHub Check: build / build_pr_documentation
🔇 Additional comments (4)
tests/test_upstream_hub.py (4)
309-311: LGTM!Appropriate placement after the initial push to ensure Hub consistency before the subsequent upload operation.
326-328: LGTM!Correctly placed between the upload and the subsequent push operation.
360-362: LGTM!Mirrors the synchronization pattern from the first test block.
377-379: LGTM!Completes the synchronization pattern for the second test scenario.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/test_upstream_hub.py (1)
270-296: Consider broadening exception handling to cover more transient network errors.The helper only catches
HfHubHTTPError, but transient connection issues (e.g.,ConnectionError,requests.exceptions.Timeout) would propagate and fail the test. Consider catching a broader set of transient errors:- except HfHubHTTPError: + except (HfHubHTTPError, ConnectionError) as e: + # Retry on transient errors time.sleep(1)Also, moving the import to the top of the file with other
huggingface_hubimports would be more idiomatic:-from huggingface_hub import DatasetCard, HfApi +from huggingface_hub import DatasetCard, HfApi +from huggingface_hub.errors import HfHubHTTPError
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/test_upstream_hub.py(6 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
tests/test_upstream_hub.py (1)
tests/fixtures/hub.py (1)
temporary_repo(87-99)
🪛 Gitleaks (8.29.1)
tests/test_upstream_hub.py
[high] 358-358: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (13)
- GitHub Check: test (unit, windows-latest, deps-latest)
- GitHub Check: test (integration, ubuntu-latest, deps-minimum)
- GitHub Check: test (integration, windows-latest, deps-latest)
- GitHub Check: test (integration, ubuntu-latest, deps-latest)
- GitHub Check: test (unit, windows-latest, deps-minimum)
- GitHub Check: test (integration, windows-latest, deps-minimum)
- GitHub Check: test (unit, ubuntu-latest, deps-minimum)
- GitHub Check: test (unit, ubuntu-latest, deps-latest)
- GitHub Check: test_py314_future (unit, windows-latest, deps-latest)
- GitHub Check: test_py314_future (unit, ubuntu-latest, deps-latest)
- GitHub Check: test_py314 (unit, windows-latest, deps-latest)
- GitHub Check: test_py314 (unit, ubuntu-latest, deps-latest)
- GitHub Check: build / build_pr_documentation
🔇 Additional comments (4)
tests/test_upstream_hub.py (4)
306-311: LGTM!Correctly placed synchronization point after the first push ensures Hub has processed LFS objects before the subsequent file upload.
326-329: LGTM!Synchronization before the second push ensures the manually uploaded file is fully propagated.
357-361: LGTM - variable rename and synchronization points are correct.The rename to
ds_name_2for the second test block improves clarity. The static analysis warning about "Generic API Key" on line 358 is a false positive—self._tokenreferences the test fixture constantCI_HUB_USER_TOKEN, not a hardcoded secret.
369-395: LGTM!All API calls in the second test block consistently reference
ds_name_2, and the wait synchronization is correctly placed before the second push operation.
`test_push_dataset_dict_to_hub_overwrite_files` intermittently fails with: ``` BadRequestError: LFS pointer pointed to a file that does not exist ``` Root cause: Two race conditions in the test design: 1. Rapid successive `push_to_hub` calls don't wait for Hub's LFS object propagation between pushes 2. Second test scenario reused the same repo name, creating a race between repo deletion and recreation Fix: - Add `_wait_for_repo_ready()` helper that ensures Hub repository is in a consistent state before subsequent operations - Use unique repo name (`ds_name_2`) for second scenario to eliminate the delete/create race entirely Tested: All 4 integration test variants now pass consistently (ubuntu/windows, deps-latest/deps-minimum).
The-Obstacle-Is-The-Way
force-pushed
the
fix/flaky-lfs-test
branch
from
e98efc8 to
109720e
Compare
Summary
_wait_for_repo_readyhelper to ensure Hub repository consistency between operationstest_push_dataset_dict_to_hub_overwrite_filesthat caused intermittent LFS pointer errorsProblem
test_push_dataset_dict_to_hub_overwrite_filesintermittently fails with:This occurs because rapid successive
push_to_hubcalls don't wait for Hub's LFS object propagation between pushes.Solution
Add synchronization delay between push operations to ensure Hub has fully processed LFS objects before the next push.
Test plan
Related Issues
push_to_hubis not concurrency safe (dataset schema corruption) huggingface/datasets#7600 (push_to_hub concurrency safety)push_to_hubis not robust to hub closing connection huggingface/datasets#6392 (push_to_hub connection robustness)This fix is intended for upstream submission to huggingface/datasets after validation.
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.