Stream "DAG", i.e. treat streams and their contexts as nodes in directed graph

[edgarrmondragon]

Use cases

1. One parent record spawns one child sync

stateDiagram-v2
    [*] --> Parent1Record1
    Parent1Record1 --> Context1
    Context1 --> Child1Sync1
    Child1Sync1 --> [*]

Loading

2. One parent record spawns multiple child syncs

stateDiagram-v2
    [*] --> Parent1Record1
    Parent1Record1 --> Context1
    Parent1Record1 --> Context2
    Context1 --> Child1Sync1
    Context2 --> Child1Sync2
    Child1Sync1 --> [*]
    Child1Sync2 --> [*]

Loading

3. Multiple records from different parent streams spawn a single child sync

Contexts are merged. The number of children syncs is equal to the Cartesian product of all records from all parent streams.

stateDiagram-v2
    [*] --> Parent1Record1
    [*] --> Parent2Record1
    Parent1Record1 --> Context1
    Parent2Record1 --> Context1
    Context1 --> ChildSync1
    ChildSync1 --> [*]

Loading

4. Multiple records from different parent streams each spawn a sync for the same child stream

stateDiagram-v2
    [*] --> Parent1Record1
    [*] --> Parent2Record1
    Parent1Record1 --> Context1
    Parent2Record1 --> Context2
    Context1 --> Child1Sync1
    Context2 --> Child1Sync2
    Child1Sync1 --> [*]
    Child1Sync2 --> [*]

Loading

5. All the records from the parent stream aggregate into a single context and a single child sync

stateDiagram-v2
    [*] --> Parent1Record1
    [*] --> Parent1Record2
    [*] --> Parent1RecordN
    Parent1Record1 --> Context1
    Parent1Record2 --> Context1
    Parent1RecordN --> Context1
    Context1 --> ChildSync1
    ChildSync1 --> [*]

Loading

Related

• feat: multiple parents for a child stream #2290
• feat: Child stream that runs after all records synced by parent stream #2809
• feat(taps): Abstract streams #2888

[ReubenFrankel]

Is this planned as part of the eventual v1 release?

[edgarrmondragon]

I think so, yes. Though I'd like to gather a few use cases for different types of DAGs and then come up with a design.

[edgarrmondragon]

I've updated the description. That should give us some inspiration for potential implementations.