Skip to content

Commit f6bd25c

Browse files
authored
Merge pull request #439 from Shopify/tweak-custom-enumerator-guide
2 parents 5464da0 + e556bb6 commit f6bd25c

File tree

1 file changed

+20
-8
lines changed

1 file changed

+20
-8
lines changed

guides/custom-enumerator.md

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,25 @@
1-
Iteration leverages the [Enumerator](https://ruby-doc.org/3.2.1/Enumerator.html) pattern from the Ruby standard library,
1+
# Custom Enumerator
2+
3+
`Iteration` leverages the [Enumerator](https://ruby-doc.org/3.2.1/Enumerator.html) pattern from the Ruby standard library,
24
which allows us to use almost any resource as a collection to iterate.
35

46
Before writing an enumerator, it is important to understand [how Iteration works](iteration-how-it-works.md) and how
57
your enumerator will be used by it. An enumerator must `yield` two things in the following order as positional
68
arguments:
79
- An object to be processed in a job `each_iteration` method
8-
- A cursor position, which Iteration will persist if `each_iteration` returns succesfully and the job is forced to shut
10+
- A cursor position, which `Iteration` will persist if `each_iteration` returns successfully and the job is forced to shut
911
down. It can be any data type your job backend can serialize and deserialize correctly.
1012

11-
A job that includes Iteration is first started with `nil` as the cursor. When resuming an interrupted job, Iteration
13+
A job that includes `Iteration` is first started with `nil` as the cursor. When resuming an interrupted job, `Iteration`
1214
will deserialize the persisted cursor and pass it to the job's `build_enumerator` method, which your enumerator uses to
1315
find objects that come _after_ the last successfully processed object. The [array enumerator](https://github.com/Shopify/job-iteration/blob/v1.3.6/lib/job-iteration/enumerator_builder.rb#L50-L67)
1416
is a simple example which uses the array index as the cursor position.
1517

16-
For a more complex example, consider this Enumerator that wraps a third party API (Stripe) for paginated iteration and
18+
In addition to the remainder of this guide, we recommend you read the implementation of the other enumerators that come with the library (`CsvEnumerator`, `ActiveRecordEnumerator`) to gain a better understanding of building enumerators.
19+
20+
## Enumerator with cursor
21+
22+
For a more complex example, consider this `Enumerator` that wraps a third party API (Stripe) for paginated iteration and
1723
stores a string as the cursor position:
1824

1925
```ruby
@@ -58,6 +64,8 @@ class StripeListEnumerator
5864
end
5965
```
6066

67+
### Usage
68+
6169
Here we leverage the Stripe cursor pagination where the cursor is an ID of a specific item in the collection. The job
6270
which uses such an `Enumerator` would then look like so:
6371

@@ -90,12 +98,14 @@ end
9098
and you initiate the job with
9199

92100
```ruby
93-
LoadRefundsForChargeJob.perform_later(_charge_id = "chrg_345")
101+
LoadRefundsForChargeJob.perform_later(charge_id = "chrg_345")
94102
```
95103

96-
Sometimes you can ignore the cursor. Consider the following custom Enumerator that takes items from a Redis list, which
104+
## Cursorless enumerator
105+
106+
Sometimes you can ignore the cursor. Consider the following custom `Enumerator` that takes items from a Redis list, which
97107
is essentially a queue. Even if this job doesn't need to persist a cursor in order to resume, it can still use
98-
Iteration's signal handling to finish `each_iteration` and gracefully terminate.
108+
`Iteration`'s signal handling to finish `each_iteration` and gracefully terminate.
99109

100110
```ruby
101111
class RedisPopListJob < ActiveJob::Base
@@ -115,7 +125,9 @@ class RedisPopListJob < ActiveJob::Base
115125
end
116126
```
117127

118-
We recommend that you read the implementation of the other enumerators that come with the library (`CsvEnumerator`, `ActiveRecordEnumerator`) to gain a better understanding of building Enumerator objects.
128+
## Caveats
129+
130+
### Post-`yield` code
119131

120132
Code that is written after the `yield` in a custom enumerator is not guaranteed to execute. In the case that a job is
121133
forced to exit ie `job_should_exit?` is true, then the job is re-enqueued during the yield and the rest of the code in

0 commit comments

Comments
 (0)