Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
ecbfd65
Fully implement RESP (#40)
cjbottaro Jul 2, 2019
dc8d4c4
Merge branch 'master' into v0.8.0
cjbottaro Jul 7, 2019
9c3f9a2
Handle quiet and terminate (#42)
cjbottaro Jul 7, 2019
65ee820
Changelog
cjbottaro Jul 7, 2019
d646ac8
Shutdown OTP style with grace period (#43)
cjbottaro Jul 7, 2019
5df786d
Changelog
cjbottaro Jul 7, 2019
1c291b9
Clean up and reorg
cjbottaro Jul 7, 2019
331356d
A for Ack?
cjbottaro Jul 7, 2019
afdb9c4
Documentation
cjbottaro Jul 7, 2019
15cc7f7
Comments
cjbottaro Jul 7, 2019
ca03805
Version bump
cjbottaro Jul 7, 2019
a8d2513
Non-priority queues aka individual queues (#44)
cjbottaro Jul 7, 2019
3eeab82
Changelog
cjbottaro Jul 7, 2019
afa0162
Specify job options on (#45)
cjbottaro Jul 8, 2019
7d63291
Changelog
cjbottaro Jul 8, 2019
8b011fe
Make changelog prettier maybe?
cjbottaro Jul 8, 2019
444bb1d
Words
cjbottaro Jul 8, 2019
1a0298d
Characters bother me
cjbottaro Jul 8, 2019
13a9503
Update architecture doc
cjbottaro Jul 8, 2019
91c6254
Typos
cjbottaro Jul 8, 2019
23a0120
Words
cjbottaro Jul 8, 2019
4399b62
Compatibility with releases (#46)
cjbottaro Jul 8, 2019
26cf8d7
Changelog
cjbottaro Jul 8, 2019
af9dde4
Secret hidden option for me
cjbottaro Jul 9, 2019
1e67b24
I am not a smart man.
cjbottaro Jul 9, 2019
f826ba0
I am really not a smart man
cjbottaro Jul 9, 2019
e8bc2f0
I like words
cjbottaro Sep 7, 2019
749091e
Remove multiple fetchers (#52)
cjbottaro Nov 9, 2019
9c79f9f
Graceful shutdown (#56)
cjbottaro May 19, 2020
c4a7816
Changes to Faktory.push (#57)
cjbottaro Jun 6, 2020
7b99436
Erlang modules don't expose __info__/1 (#58)
cjbottaro Jun 6, 2020
b12f68e
Going to 1.0
cjbottaro Jul 9, 2020
25de5ae
Move protocol functions to client (#61)
cjbottaro Jul 10, 2020
10bd69c
--no-start option (#62)
cjbottaro Oct 1, 2020
cd223c5
Second stab at graceful shutdown (#63)
cjbottaro Nov 11, 2020
62f242c
Granular control over starting workers (#64)
cjbottaro Jan 31, 2021
efb7cca
Update CHANGELOG.md
cjbottaro Feb 5, 2021
b815b1d
Getting started
cjbottaro Apr 17, 2021
e90802d
Docs, Faktory.Job stuff, logging, other stuff
cjbottaro Apr 17, 2021
7bf29e1
Hey, look... connection pool
cjbottaro Apr 18, 2021
ed6cfa1
ConnectionPool tweaks
cjbottaro Apr 19, 2021
6d9ea32
connection pool stuff
cjbottaro Apr 21, 2021
7269ed4
Refactor Connection and Client
cjbottaro Apr 24, 2021
fb01fc5
Heartbeat, quiet, stop
cjbottaro Apr 24, 2021
585aff8
Hey look, workers work
cjbottaro Apr 26, 2021
a7cc9cf
Worker working with graceful shutdown
cjbottaro Apr 26, 2021
24261f9
Lots of logging/telemetry stuff, and of course, docs
cjbottaro Apr 29, 2021
32e6405
Get some tests going
cjbottaro May 20, 2021
ad41f36
Clean up tests
cjbottaro May 20, 2021
292e394
Config merging is expensive, don't do it on push
cjbottaro May 21, 2021
8e0fff3
Password tests
cjbottaro May 21, 2021
b8b02db
Make work with tls
cjbottaro May 21, 2021
f993099
Default client
cjbottaro May 21, 2021
df2131a
Make a Socket protocol
cjbottaro May 21, 2021
290a228
Cleanup
cjbottaro May 21, 2021
2474161
Naming issues, update deps, take care of jobtype_map issues
cjbottaro May 21, 2021
796f835
Names
cjbottaro May 21, 2021
5c8bf4c
Copy
cjbottaro May 21, 2021
0cd3a47
Get the mix task working
cjbottaro May 22, 2021
429670a
Documentation
cjbottaro May 22, 2021
f4e54ec
Shutdown on heartbeat error
cjbottaro May 23, 2021
94c4ff6
Restart transient?
cjbottaro May 23, 2021
1bd5657
It doesn't make sense if the process doesn't end
cjbottaro May 23, 2021
cee2be2
Make Faktory.Worker a behaviour
cjbottaro May 25, 2021
32cd999
Heartbeat immediately
cjbottaro May 25, 2021
894c5ce
Handle errors properly
cjbottaro May 26, 2021
38da91a
More robust error conversion
cjbottaro Jun 9, 2021
2314dbc
Better erlang errors
cjbottaro Jun 10, 2021
7d46f60
Fix json type
cjbottaro Jun 16, 2021
a3ab1f5
Maybe fix middleware
cjbottaro Oct 8, 2021
e2b2b40
Normalize job to a map
cjbottaro Oct 8, 2021
0bc872b
Allow modules for jobtype
cjbottaro Oct 9, 2021
ef6c5c3
Fix telemetry deps
cjbottaro Oct 28, 2021
71ecd63
Support for username
cjbottaro Mar 2, 2022
0731a98
Fix warnings and typespec
cjbottaro Apr 8, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 0 additions & 67 deletions Architecture.md

This file was deleted.

27 changes: 17 additions & 10 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,37 @@
# faktory_worker_ex changes

## 0.8.0
-----------
## 1.0.0
Detailed info in the linked pull requests.

* Basically an entire rewrite. (PR [#68](https://github.com/cjbottaro/faktory_worker_ex/pull/68))
* Granular control over starting workers. (PR [#64](https://github.com/cjbottaro/faktory_worker_ex/pull/64))
* Graceful shutdown. (PR [#43](https://github.com/cjbottaro/faktory_worker_ex/pull/43), [#56](https://github.com/cjbottaro/faktory_worker_ex/pull/56), [#63](https://github.com/cjbottaro/faktory_worker_ex/pull/63))
* Internally respect "reserve_for" (PR [#63](https://github.com/cjbottaro/faktory_worker_ex/pull/63))
* --no-start command line option (PR [#62](https://github.com/cjbottaro/faktory_worker_ex/pull/62))
* Move protocol functions to `Faktory.Client` (PR [#61](https://github.com/cjbottaro/faktory_worker_ex/pull/61))
* Erlang modules don't expose `__info__/1`. (PR [#58](https://github.com/cjbottaro/faktory_worker_ex/pull/58))
* Make Faktory.push/2 a low level API. (PR [#57](https://github.com/cjbottaro/faktory_worker_ex/pull/57))
* Make Faktory.push/2 return errors. (PR [#57](https://github.com/cjbottaro/faktory_worker_ex/pull/57))
* Use GenStage instead of BlockingQueue. (PR [#36](https://github.com/cjbottaro/faktory_worker_ex/pull/36))
* Find the right job module when jobtype doesn't match module name. (PR [#35](https://github.com/cjbottaro/faktory_worker_ex/pull/35))
* Graceful shutdown on SIGTERM. (PR [#41](https://github.com/cjbottaro/faktory_worker_ex/pull/41))
* Implement Redis Serialization Protocol (RESP). (PR [#40](https://github.com/cjbottaro/faktory_worker_ex/pull/40))
* Handle `quiet` and `terminate` from Faktory server. (PR [#42](https://github.com/cjbottaro/faktory_worker_ex/pull/42))
* `priority_queues: false`. See module documentation on `Faktory.Worker`. (PR [#44](https://github.com/cjbottaro/faktory_worker_ex/pull/44))
* Specify job options on `use Faktory.Job`. (PR [#45](https://github.com/cjbottaro/faktory_worker_ex/pull/45))
* Support for Elixir 1.9 releases. (PR [#46](https://github.com/cjbottaro/faktory_worker_ex/pull/46))
* Use Jason instead of Poison.


## 0.7.1
-----------
* Fix `mix faktory` not respecting CLI args.

## 0.7.0
-----------
* The great simplication refactor; everything done in this version is to reduce conceptual complexity and make the code easier to read and understand.
* Changed to a queue based architecture (reduces number of connections to Faktory server).
* No more Genservers, the components of a worker (producer, consumers, reporter) are all supervised tasks. Jobs are also run in tasks (spawned by the consumers).
* Removed `retryable_ex`; workers will retry when fetching, acking, and failing jobs, but all else is up to the user now.

## 0.6.0
-----------
* (breaking) Configuration simplified (again); see readme.
* Can override `jobtype` for enqueuing to workers in other languages.
* Simplified supervision; removed all custom supervisor modules.
Expand All @@ -30,15 +41,13 @@
* Faktory 0.9.3

## 0.5.0
-----------
* (breaking) Simplified middleware; no longer pass the `chain` arg.
* (breaking) Completely rewrote configuration system. See `Faktory.Configuration`.
* (breaking) Removed `set/1` and `set/2` in favor of passing `options` to `perform_async/2`.
* (breaking) `perform_async/2` argument order reversed (`options` is 2nd arg now).
* Can enqueue to different Faktory servers via `perform_async/2`.

## 0.4.0
-----------
* Faktory 0.6.x support @acj @valo
* Authentication support @acj
* TLS support @acj
Expand All @@ -48,11 +57,9 @@
* Update protocol to handle `$-1\r\n` @valo

## 0.3.0
-----------
* Configure with Mix Config and callback.
* Independently configurable logger.
* Connection pooling for workers.

## 0.2.1
-----------
* Start having a changelog.
205 changes: 155 additions & 50 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,102 +2,207 @@

Elixir worker for Faktory ([blog](http://www.mikeperham.com/2017/10/24/introducing-faktory/)) ([github](https://github.com/contribsys/faktory)).

## Installation
This is an Elixir library, so you should be reading the documentation
on Hexdocs [here](https://hexdocs.pm/faktory_worker_ex) instead of Github.

## Quickstart

The package can be installed by adding `faktory_worker_ex` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[
{:faktory_worker_ex, "~> 0.0"}
{:faktory_worker_ex, "~> 1.0"}
]
end
```

## Quickstart

```elixir
# For enqueuing jobs
defmodule MyFaktoryClient do
use Faktory.Client, otp_app: :my_cool_app
end

# For processing jobs
defmodule MyFaktoryWorker do
use Faktory.Worker, otp_app: :my_cool_app
end

# You must add them to your app's supervision tree
defmodule MyCoolApp.Application do
use Application
defmodule GreetJob do
use Faktory.Job
require Logger

def start(_type, _args) do
children = [MyFaktoryClient, MyFaktoryWorker]
Supervisor.start_link(children, strategy: :one_for_one)
def perform(name) do
Logger.info("Hello, #{name}!")
end
end

defmodule MyGreeterJob do
use Faktory.Job

def perform(greeting, name) do
IO.puts("#{greeting}, #{name}!!")
Logger.info("#{greeting}, #{name}!")
end
end

# List argument must match the arity of MyGreeterJob.perform
MyGreeterJob.perform_async(["Hello", "Genevieve"])
# Notice the argument is a list that must have the same arity as one of the perform functions.
GreetJob.perform_async(["Genevieve"])
GreetJob.perform_async(["Sup", "my dude"])
```

## Starting the worker
```plain
$ mix faktory

`mix faktory`
19:02:23.128 [info] Hello, Genevieve!

You should see logging output and the above job being processed.
19:02:23.128 [info] Sup, my dude!
```

`mix faktory -h`
## Connection options

To see command line options that can override in-app configuration.
By default, both client and worker connections will connect to `localhost:7419`.
You can change that via `Config`.

`iex -S mix faktory`
```elixir
import Config

If you want to debug your jobs using `IEx.pry`.
config :faktory_worker_ex, Faktory.Connection,
host: "faktory.foo.com",
port: 8000
```

## Configuration

Compile-time config is done with `Mix.Config`.
This library is massively configurable. It was meant to be used in large
umbrella applications and with multiple Faktory servers.

Most modules let you `use` them to make specific and individually configurable
modules.

Run-time config is done with environment variables and/or an `init/1` callback.
There is a hierarchy to configuration. Arguments to `start_link` override all,
`Config` overrides `use` arguments (for runtime configuration via `runtime.exs`),
and specific modules override the base modules.

See documentation on:
* [Client](https://hexdocs.pm/faktory_worker_ex/Faktory.Client.html)
* [Worker](https://hexdocs.pm/faktory_worker_ex/Faktory.Worker.html)
```elixir
import Config

config :faktory_worker_ex, Faktory.Client, host: "foo",

defmodule MyClient do
use Faktory.Client, host: "bar"
end

# Connect to "foo"
{:ok, client} = Faktory.Client.start_link()

# Connect to "bar"
{:ok, client} = Faktory.Client.start_link(host: "bar")

# Connect to "bar"
MyClient.start_link()

# Connect to "baz"
MyClient.start_link(host: "baz")
```

All the configuration options are deep merged according to the hierarchy rules.

## Running a Faktory server

To run this readme's example, you need to run a Faktory server.
To run the quickstart example, you need to run a Faktory server.

Easiest way is with Docker:
Easiest way is with Docker.
```
docker run --rm -p 7419:7419 -p 7420:7420 contribsys/faktory:latest -b :7419 -w :7420
```

You should be able to go to [http://localhost:7420](http://localhost:7420) and see the web ui.

## Using with multiple Faktory servers

```elixir
import Config

config :my_app, FooClient,
host: "foo-faktory.myapp.com"

config :my_app, BarClient,
host: "bar-faktory.myapp.com"

config :my_app, FooWorker,
host: "foo-faktory.myapp.com"

config :my_app, BarWorker,
host: "bar-faktory.myapp.com"

defmodule FooClient do
use Faktory.Client
end

defmodule BarClient do
use Faktory.Client
end

defmodule FooWorker do
use Faktory.Worker
end

defmodule BarWorker do
use Faktory.Worker
end

defmodule FooJob do
use Faktory.Job, client: FooClient
def perform(), do: nil
end

defmodule BarJob do
use Faktory.Job, client: BarClient
def perform(), do: nil
end

FooJob.perform_async([]) # Enqueues job to Faktory server at foo-faktory.myapp.com
BarJob.perform_async([]) # Enqueues job to Faktory server at bar-faktory.myapp.com
```

Be sure to add your workers to your application's supervision tree.
```elixir
defmodule MyApp.Application do
use Application

def start(_type, _args) do
children = [
FooWorker,
BarWorker,
]

Supervisor.start_link(children, strategy: :one_for_one)
end
end
```

And disable the default worker.
```
use Config

config :faktory_worker_ex, Faktory.DefaultWorker, start: false
```

## Starting individual workers

You can do this on the command line.
```
mix faktory --only FooWorker,BarWorker
mix faktory --only FooWorker --only BarWorker

mix faktory --except FooWorker,BarWorker
mix faktory --except FooWorker --except BarWorker
```

Or you can tell a worker to start (or not start) via `Config`.
```elixir
import Config

# Will start regardless of `mix faktory`
config :my_app, FooWorker, start: true

# Will not start regardless of `mix faktory`
config :my_app, BarWorker, start: false
```

## Features

* Middleware
* Connection pooling (for clients)
* Connection pooling
* Support for multiple Faktory servers
* Faktory server authentication and TLS support
* Supports 100% of Faktory features
* Comprehensive documentation
* Comprehensive supervision tree
* Decent integration tests

## Missing features

* Responding to `quiet` and `terminate`
* Running without `mix` (e.g. a Distillery release)

## Issues / Questions

[https://github.com/cjbottaro/faktory_worker_ex/issues](https://github.com/cjbottaro/faktory_worker_ex/issues)
Loading