Skip to content

[EPIC] Integrate backend with billing vendor #8463

Description

@lmossman

Tell us about the problem you're trying to solve

Our current billing / credit management system was quickly hacked together in order to release Airbyte Cloud. It lacks features desired by our product team (e.g. adding/deducting credits for a user in a UI, automatic credit expiry, generating usage reports), and uses up valuable engineering time to maintain and improve. The bottom line is that Airbyte is not a billing company, and we shouldn’t try to be.

See tech spec for more details: https://docs.google.com/document/d/1WLr7Leaj5Iyy5bBEjr1TGhNL8HhSB_evbM_psEwKyAI/edit#

Describe the solution you’d like

In order to reduce the amount of changes we are making in the first iteration of this solution, we will keep the existing CreditProcessingCronJob and continue to write CreditConsumptionRecords to the database for each attempt, but it will be changed to stop tracking and managing additions and deductions to credit balance, and instead just report usage to the billing vendor. This way we will still maintain our own record of credit consumption which will be helpful in validating that our billing vendor integration is working correctly, and it also maintains the existing setup of keeping credit consumption tracking completely isolated from the job lifecycle.

Describe the alternative you’ve considered or used

See tech spec

Execution

  1. Simplify CreditProcessingCronJob queries and time windows #8505
  2. Creating billing resources in billing vendor #8504
  3. Create customer in billing vendor upon CloudWorkspace creation for test billing user #8496
  4. Track credit usage in billing vendor for test billing user #8498
  5. Backfill and start tracking credits for all CloudWorkspaces in billing vendor #8500
  6. Compare billing vendor credit consumption to ours #8506
  7. Update the CloudWorkspaceHandler to retrieve credit balance from billing vendor #8507
  8. Update the CloudWorkspaceHandler to retrieve credit usage from billing vendor  #8509
  9. https://github.com/airbytehq/airbyte-internal-issues/issues/344
  10. Set up listener for credit threshold alerts from billing vendor - not creating an issue yet, want to understand the requirements here better first

Additional context

Are you willing to submit a PR?

Metadata

Metadata

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions