You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: guides/custom-enumerator.md
+20-8Lines changed: 20 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,19 +1,25 @@
1
-
Iteration leverages the [Enumerator](https://ruby-doc.org/3.2.1/Enumerator.html) pattern from the Ruby standard library,
1
+
# Custom Enumerator
2
+
3
+
`Iteration` leverages the [Enumerator](https://ruby-doc.org/3.2.1/Enumerator.html) pattern from the Ruby standard library,
2
4
which allows us to use almost any resource as a collection to iterate.
3
5
4
6
Before writing an enumerator, it is important to understand [how Iteration works](iteration-how-it-works.md) and how
5
7
your enumerator will be used by it. An enumerator must `yield` two things in the following order as positional
6
8
arguments:
7
9
- An object to be processed in a job `each_iteration` method
8
-
- A cursor position, which Iteration will persist if `each_iteration` returns succesfully and the job is forced to shut
10
+
- A cursor position, which `Iteration` will persist if `each_iteration` returns successfully and the job is forced to shut
9
11
down. It can be any data type your job backend can serialize and deserialize correctly.
10
12
11
-
A job that includes Iteration is first started with `nil` as the cursor. When resuming an interrupted job, Iteration
13
+
A job that includes `Iteration` is first started with `nil` as the cursor. When resuming an interrupted job, `Iteration`
12
14
will deserialize the persisted cursor and pass it to the job's `build_enumerator` method, which your enumerator uses to
13
15
find objects that come _after_ the last successfully processed object. The [array enumerator](https://github.com/Shopify/job-iteration/blob/v1.3.6/lib/job-iteration/enumerator_builder.rb#L50-L67)
14
16
is a simple example which uses the array index as the cursor position.
15
17
16
-
For a more complex example, consider this Enumerator that wraps a third party API (Stripe) for paginated iteration and
18
+
In addition to the remainder of this guide, we recommend you read the implementation of the other enumerators that come with the library (`CsvEnumerator`, `ActiveRecordEnumerator`) to gain a better understanding of building enumerators.
19
+
20
+
## Enumerator with cursor
21
+
22
+
For a more complex example, consider this `Enumerator` that wraps a third party API (Stripe) for paginated iteration and
17
23
stores a string as the cursor position:
18
24
19
25
```ruby
@@ -58,6 +64,8 @@ class StripeListEnumerator
58
64
end
59
65
```
60
66
67
+
### Usage
68
+
61
69
Here we leverage the Stripe cursor pagination where the cursor is an ID of a specific item in the collection. The job
62
70
which uses such an `Enumerator` would then look like so:
Sometimes you can ignore the cursor. Consider the following custom Enumerator that takes items from a Redis list, which
104
+
## Cursorless enumerator
105
+
106
+
Sometimes you can ignore the cursor. Consider the following custom `Enumerator` that takes items from a Redis list, which
97
107
is essentially a queue. Even if this job doesn't need to persist a cursor in order to resume, it can still use
98
-
Iteration's signal handling to finish `each_iteration` and gracefully terminate.
108
+
`Iteration`'s signal handling to finish `each_iteration` and gracefully terminate.
99
109
100
110
```ruby
101
111
classRedisPopListJob < ActiveJob::Base
@@ -115,7 +125,9 @@ class RedisPopListJob < ActiveJob::Base
115
125
end
116
126
```
117
127
118
-
We recommend that you read the implementation of the other enumerators that come with the library (`CsvEnumerator`, `ActiveRecordEnumerator`) to gain a better understanding of building Enumerator objects.
128
+
## Caveats
129
+
130
+
### Post-`yield` code
119
131
120
132
Code that is written after the `yield` in a custom enumerator is not guaranteed to execute. In the case that a job is
121
133
forced to exit ie `job_should_exit?` is true, then the job is re-enqueued during the yield and the rest of the code in
0 commit comments