Skip to content

Conversation

@chrisguidry
Copy link
Owner

Summary

Adds an Agenda class that collects tasks and scatters them evenly over time periods to avoid overwhelming systems during find-and-flood workloads.

The implementation provides a clean API for batch scheduling:

  • agenda.add(task)(args) to collect tasks
  • agenda.scatter(docket, over=timedelta(minutes=50)) to distribute them
  • Optional jitter to prevent thundering herd effects
  • Honest partial scheduling behavior when failures occur

Key design decisions made during implementation:

  • Single tasks are positioned at window midpoint rather than endpoints
  • The over parameter is required for explicit timing control
  • Simple partial scheduling without complex rollback guarantees
  • Auto-registration of unregistered task functions

Closes #131

🤖 Generated with Claude Code

chrisguidry and others added 4 commits August 25, 2025 08:11
Introduces the Agenda class that collects tasks and distributes them
evenly over a time period. Key features:
- Scatter tasks over a timeframe with even distribution
- Optional jitter to prevent thundering herd
- Single task positioned at midpoint of window
- Required 'over' parameter for explicit timing

Still needs true atomicity implementation using Redis pipelines.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
The scatter method now provides true atomic scheduling - either all
tasks are scheduled successfully or none are. This is achieved by:

1. Validating all tasks upfront before scheduling any
2. Building Execution objects with validation
3. Rolling back (cancelling) any scheduled tasks if a failure occurs

The atomicity test now properly demonstrates:
- Failure when unregistered tasks are included (no tasks scheduled)
- Successful scheduling when all tasks are valid
- Rollback behavior when scheduling fails partway through

This ensures predictable behavior for batch task scheduling operations.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Added tests to cover the remaining uncovered lines:
- Auto-registration of unregistered task functions (line 172)
- Exception handling during rollback when cancel fails (lines 205-206)

The agenda module now has full test coverage demonstrating all
code paths work correctly, including edge cases.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Removes the complex compensation-based rollback in favor of simple,
predictable behavior:

- No rollback attempts when scheduling fails
- Clear documentation about partial scheduling
- Honest test expectations about failure behavior
- Removed misleading atomicity claims

Also:
- Removed task scattering from main README (too detailed)
- Added comprehensive Agenda documentation to advanced-patterns.md
- Maintained 100% test coverage with simplified implementation

The behavior is now: if scheduling fails partway through, earlier
tasks remain scheduled and later tasks are not scheduled. This is
simple, predictable, and honest about the limitations.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@claude
Copy link
Contributor

claude bot commented Aug 25, 2025

Claude finished @chrisguidry's task —— View job


Code Review for PR #159: Add Agenda class for task scattering

Todo List

  • Read and analyze the Agenda implementation
  • Review test coverage and test quality
  • Check the example code
  • Review documentation updates
  • Verify public API exports
  • Provide comprehensive review feedback

Summary

This PR successfully implements a well-designed Agenda class that provides an elegant solution for batch task scheduling with temporal distribution. The implementation is solid, with excellent test coverage and clear documentation.

✅ Strengths

  1. Clean API Design: The fluent interface (agenda.add(task)(args)) is intuitive and follows established Python patterns.

  2. Comprehensive Test Coverage: The test suite is thorough, covering edge cases like large jitter values, single tasks, empty agendas, and partial failure scenarios. The 100% coverage requirement is maintained.

  3. Safety Implementations: Good defensive programming with:

    • Validation for non-positive over parameter
    • Jitter safety check preventing negative timestamps (src/docket/agenda.py:161)
    • Auto-registration of unregistered task functions
  4. Documentation Quality: Clear docstrings and a practical example demonstrating real-world rate limiting use cases.

  5. Proper Error Handling: The partial scheduling behavior is well-documented and tested, with clear distinction between validation failures (fail fast) and scheduling failures (partial success).

🎯 Code Quality Observations

  1. Type Annotations: Excellent use of Python typing with proper generic types (ParamSpec, TypeVar) for type safety (src/docket/agenda.py:17-18).

  2. Pythonic Patterns: Good use of Python idioms like __len__, __iter__ making the Agenda behave like a proper collection.

  3. UUID7 Usage: Smart choice using UUID7 for task keys (src/docket/agenda.py:182), providing time-ordered unique identifiers.

💡 Minor Suggestions for Future Improvements

  1. Consider Batch Operations: The current implementation schedules tasks one-by-one (src/docket/agenda.py:194-199). For very large batches, a batch scheduling API might improve performance.

  2. Scatter Strategy Extensibility: While the current even distribution with optional jitter works well, consider making the scatter strategy pluggable for future needs (e.g., exponential backoff, weighted distribution).

  3. Progress Callback: For very large agendas, consider adding an optional progress callback for monitoring during the scatter operation.

🔒 Security Considerations

No security concerns identified. The implementation properly:

  • Validates all inputs
  • Uses safe random number generation for jitter
  • Doesn't expose sensitive information in error messages

⚡ Performance Considerations

The implementation is efficient for typical use cases:

  • O(n) time complexity for scattering n tasks
  • Minimal memory overhead with generator patterns where applicable
  • Smart positioning of single tasks at midpoint rather than start

🐛 No Bugs Found

The implementation appears bug-free with robust error handling and edge case coverage.

✅ Recommendation

This PR is ready to merge. The Agenda class is a valuable addition to Docket that elegantly solves the task distribution problem. The implementation is production-ready with excellent test coverage, clear documentation, and thoughtful design decisions.

@github-actions
Copy link

github-actions bot commented Aug 25, 2025

📚 Documentation has been built for this PR!

You can download the documentation directly here:
https://github.com/chrisguidry/docket/actions/runs/17267446361/artifacts/3863400241

@codecov-commenter
Copy link

codecov-commenter commented Aug 25, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (fd97363) to head (8312d56).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##              main      #159    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           31        33     +2     
  Lines         4536      4789   +253     
  Branches       247       267    +20     
==========================================
+ Hits          4536      4789   +253     
Flag Coverage Δ
python-3.12 100.00% <100.00%> (ø)
python-3.13 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/docket/__init__.py 100.00% <100.00%> (ø)
src/docket/agenda.py 100.00% <100.00%> (ø)
tests/test_agenda.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

chrisguidry and others added 2 commits August 25, 2025 08:40
The example was using Worker without the async context manager, causing
AttributeError on _execution_counts. Also reduced timing windows to make
the example complete within 2 minutes instead of 3+ minutes.

Fixed:
- Worker now properly used with async context manager
- Reduced scatter windows from 30s/20s to 10s/10s
- Reduced future window from 2min to 30s
- Reduced worker timeout from 3min to 1.5min
- Reduced task sleep times from 0.5s/0.2s to 0.1s/0.1s

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@chrisguidry
Copy link
Owner Author

@claude let's implement handling for your code review suggestion:

Edge Case: The jitter implementation (src/docket/agenda.py:148-158) could theoretically produce negative timestamps if jitter exceeds the distance to start. While unlikely with reasonable values, consider adding a safety check

@claude
Copy link
Contributor

claude bot commented Aug 25, 2025

Claude finished @chrisguidry's task —— View job


Task: Implement safety check for jitter in Agenda.scatter()

Todo List

  • Read current implementation in src/docket/agenda.py
  • Understand the jitter logic and identify the issue
  • Implement safety check to prevent negative timestamps
  • Check if tests need updating
  • Run tests and linting
  • Commit and push changes

Summary

Successfully implemented the safety check to prevent jittered times from going before the start time. The changes include:

  1. Safety Check Added (src/docket/agenda.py:157-160): When applying jitter, the code now checks if the resulting time would be before start and clamps it to start if necessary.

  2. Test Coverage Added (tests/test_agenda.py:127-154): Added test_agenda_scatter_with_large_jitter to verify that even with extremely large jitter values (5 minutes jitter on a 1 minute window), all scheduled times remain at or after the start time.

The fix ensures that no matter how large the jitter value is, tasks will never be scheduled before the specified start time, preventing potential negative timestamp issues.

claude bot and others added 4 commits August 25, 2025 12:55
When using large jitter values, the random offset could potentially
push scheduled times before the start time. Added a safety check to
clamp jittered times to never be earlier than the start time.

Added test case to verify this edge case is handled correctly.

Co-authored-by: Chris Guidry <[email protected]>
@chrisguidry
Copy link
Owner Author

@claude let's add that validation for over, please, it needs to be > 0 and we should have a test for this specifically

@claude
Copy link
Contributor

claude bot commented Aug 27, 2025

Claude finished @chrisguidry's task —— View job


Task: Add validation for over parameter in Agenda.scatter()

Todo List

  • Read current implementation in src/docket/agenda.py
  • Add validation to ensure over parameter is positive
  • Write test case for the validation
  • Run tests to ensure everything passes (verified code is syntactically correct)
  • Run linting and formatting checks (fixed trailing whitespace)
  • Commit and push changes

Summary

Successfully added validation for the over parameter in Agenda.scatter():

  1. Added validation check (src/docket/agenda.py:129-130): The method now raises a ValueError if over.total_seconds() <= 0
  2. Added comprehensive test (tests/test_agenda.py:283-298): New test test_agenda_scatter_with_non_positive_over_parameter verifies that both zero and negative durations are properly rejected
  3. Updated docstring to document the new ValueError exception

The validation ensures users cannot accidentally scatter tasks over zero or negative time periods, preventing potential issues with task scheduling logic.


claude bot and others added 3 commits August 27, 2025 12:43
- Ensure 'over' parameter is positive (> 0 seconds)
- Add test cases for zero and negative durations
- Update docstring to document the ValueError

Co-authored-by: Chris Guidry <[email protected]>
@chrisguidry chrisguidry merged commit d1eb84c into main Aug 27, 2025
17 checks passed
@chrisguidry chrisguidry deleted the agenda-scatter-131 branch August 27, 2025 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shortcut for "scattering" a set of tasks over a timeframe

3 participants