Get rid of broadway by nickolaich · Pull Request #149 · opt-elixir/faktory_worker

nickolaich · 2022-01-14T17:59:18Z

removed broadway from dependencies.
It looks like PushPipeline isn't required at all.
It adds complexity with a Broadway (GenStage underneath) only for pushing jobs to Faktory.
I am not sure if I understood everything correctly but it seems we can just push job to Faktory. If assumption is correct I can clean-up code at job.ex ->perform_async function.

swelham · 2022-01-17T10:01:44Z

Hey, I can provide some context around this.

One of our original goals for building this library was to provide a way to reliably push a job or batch of jobs to Faktory without the caller having to be concerned about the outcome. We took a fire and forget approach and allowed another part of the library to deal with any failures. Broadway allowed us to do this very quickly because it already had a mechanism for dealing with and retrying failed push attempts.

We also deal with large spikes in our app that have in the past required us to schedule tens/hundreds of thousands of jobs in batch from a single run of a cron job. Without the above feature, the cron job would have to deal with parallelising the push to Faktory, then tracking and retrying any potential failures.

However, Faktory now has support for batching jobs that might solve the same issue without the need for a custom implementation (I haven't looked at the feature in-depth). In which case simplifying the library is always a win 😃

nickolaich · 2022-01-17T12:49:56Z

@swelham thank you for explanation. I thought the same. Since Faktory has batch operations and there is module FaktoryWorker.Batch to support creating batches and handle callbacks - broadway looks like an overhead.
Disadvantage of getting rid of broadway and concurrency scheduling jobs using elixir is that Batch operations are supported only by Enterprise version of Faktory

swelham · 2022-01-18T08:52:53Z

Yeah, it would be a disadvantage to some and a breaking change. However, I think for maintainability and parity with Faktory this tradeoff makes sense in the context of this library. After all, this only existed to plug a gap in Faktory where a desired feature previously didn't exist.

In addition, any users of this library that are using the OSS version of Faktory could implement their own solution (with or without Broadway). Maybe a blog post detailing the original implementation would be useful, this could even be linked from the changelog to help any users migrate.

Ch4s3 · 2022-01-18T15:24:20Z

Yeah, it would be a disadvantage to some and a breaking change. However, I think for maintainability and parity with Faktory this tradeoff makes sense in the context of this library. After all, this only existed to plug a gap in Faktory where a desired feature previously didn't exist.

Our plan is to more closely mirror the official client libraries. We'd be doing a major version bump and we'll write an upgrade guide.The right solution might be for people to implement their own batching if they're using OSS Faktory.

Ch4s3 · 2022-01-18T16:26:23Z

@swelham are you folks still using this or have you fully transitioned off?

jeremyowensboggs

It looks like the pipeline name can be removed entirely?

jeremyowensboggs · 2022-01-18T17:21:37Z

lib/faktory_worker/job.ex

+    |> format_pipeline_name()
+  end
+
+  defp format_pipeline_name(name) when is_atom(name) do


Is this still needed? Looks like the pipeline name may be able to be completely eliminated?

jeremyowensboggs · 2022-01-18T17:25:52Z

lib/faktory_worker/job.ex

+      # But Faktory is a job server and do we want to care and build one more not needed layer???
+      opts
+      |> faktory_name()
+      |> push(payload)


Switching from Broadway.push_messages to push is going to change the return of this function. From the PR discussion, I gather we are planning on doing a version change.

Are there additional changes needed to this code base to signal that change?

Does module/package documentation need to be updated?

Not for this PR, just documenting a thought - but if we can return a Job with the faktory job id here, that would be a great gateway into being able to check on the status of a job.

Broadway.push_messages and Sandbox.enqueue_job both always return :ok, so I think it should be OK to have this return {:ok, job_id} instead.

+1 for updating documentation as well

it looks like we don't need perform_async/3 at all and we can return same result for any call. private function handle_push_result prepares result and it returns {:ok, job}.
I am not sure is it better to change to {:ok, job_id} or leave it as is. job contains a bit more info, see sample below:

{:ok, %{ args: [%{hey: "there!"}], jid: "cc7cb6ec723b1e1f35815ba1", jobtype: "FaktoryWorker.DefaultWorker" }}

It should be OK to return it as-is. Can you also update Sandbox.enqueue_job to return a similar value? You can just generate a random jid.

@superhawk610 we have changed return to {:ok, job}, but I checked our app and I see it always expect :ok.
Question is -> revert changes and return :ok as before, change app and it's tests to have support of {:ok, job). or 3rd option: add configuration option and return optionally :ok to keep back compatibilities and return by default {:ok, job}.

Let's change our app to accept {:ok, job}

Make sure that this is mentioned in the upgrade guide as well, as it's a breaking change.

superhawk610 · 2022-01-18T18:18:25Z

This change means that MyApp.Job.perform_async makes a synchronous HTTP call, instead of just sending an Erlang message; that should be fine, so long as it's taken into account when writing/calling job modules. It should definitely be the top callout of the release notes.

For this PR, I think a couple more things are required:

update documentation to remove references to :skip_pipeline
update Job.perform_async/2 (since :skip_pipeline is no longer supported)

swelham · 2022-01-19T09:01:01Z

are you folks still using this or have you fully transitioned off?

@Ch4s3 We are no longer using Faktory and have completely transitioned off.

jeremyowensboggs · 2022-01-19T14:12:46Z

lib/faktory_worker/job.ex

-  def perform_async(_, {:error, _} = error, _), do: error
-
-  def perform_async(pipeline_name, payload, opts) do
-    if Sandbox.active?() do


I think we want to keep the sandbox, it is what we use to simulate the server during testing.

@jeremyowensboggs makes sense, I overdid it with simplifications

superhawk610

Looks good, I'd just like to see Sandbox.enqueue_job updated to return the same shape as perform_async. We'll also want to draft a migration guide, but that can be done as part of the release notes.

…aktory

…terprise tag for specific tests

nickolaich · 2022-01-20T10:18:13Z

I've added couple of fixes:

put :console to backends of logger. there were an error with heartbeat: "END" message wasn't expected at one test. I don't know exactly a reason to suppress all errors in the tests or not. Though it could be configured at test.secrets.exs
added tag :enterprise to batch testing. Anybody who doesn't have license can run mix test --exclude enterprise to exclude that tests from running.

jeremyowensboggs · 2022-01-20T14:34:37Z

It would be a good idea to add the bit about the tags and the enterprise tag to the README.md.

superhawk610

+1 to Jeremy's suggestion to add a note to the README about the enterprise tag

nickolaich added 3 commits January 14, 2022 14:34

ignore ide folders and replace deprecated functions

fba6032

removed push_pipeline

f383313

removed broadway from deps list

a20d6d0

jeremyowensboggs reviewed Jan 18, 2022

View reviewed changes

nickolaich added 3 commits January 19, 2022 13:11

fixed checking log message (warn vs warning)

02cac14

get rid of perform_async/3

72c3066

cleaned up code, changed docs

46837d2

jeremyowensboggs reviewed Jan 19, 2022

View reviewed changes

restored Sandbox

33fed68

jeremyowensboggs approved these changes Jan 19, 2022

View reviewed changes

superhawk610 suggested changes Jan 19, 2022

View reviewed changes

return same result for sanbox's enqueuing job as for pushing job to f…

54ee711

…aktory

nickolaich changed the title ~~WIP Get rid of broadway~~ Get rid of broadway Jan 20, 2022

fixed error, put console to backends at logger for tests and added en…

3a651a3

…terprise tag for specific tests

nickolaich mentioned this pull request Jan 20, 2022

Multiple failing tests #145

Closed

superhawk610 approved these changes Jan 20, 2022

View reviewed changes

nickolaich added 3 commits January 21, 2022 11:35

updated readme

c7fae27

exclude :enterprise tests by default

96b8b08

updated doc with new response from perform_async

66f7ae1

Ch4s3 merged commit e1061ab into master Feb 9, 2022

Ch4s3 deleted the get-rid-of-broadway branch February 9, 2022 22:06

Conversation

nickolaich commented Jan 14, 2022

Uh oh!

swelham commented Jan 17, 2022

Uh oh!

nickolaich commented Jan 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swelham commented Jan 18, 2022

Uh oh!

Ch4s3 commented Jan 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ch4s3 commented Jan 18, 2022

Uh oh!

jeremyowensboggs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

superhawk610 Jan 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

superhawk610 commented Jan 18, 2022

Uh oh!

swelham commented Jan 19, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

superhawk610 left a comment

Choose a reason for hiding this comment

Uh oh!

nickolaich commented Jan 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeremyowensboggs commented Jan 20, 2022

Uh oh!

superhawk610 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nickolaich commented Jan 17, 2022 •

edited

Loading

Ch4s3 commented Jan 18, 2022 •

edited

Loading

superhawk610 Jan 19, 2022 •

edited

Loading

nickolaich commented Jan 20, 2022 •

edited

Loading