Kraken ledger pps by goodboy · Pull Request #346 · pikers/piker

goodboy · 2022-06-29T17:50:58Z

piker.brokers.kraken pkg reorg into multiple submods (l like #334 for ib) as well as initial ledger file pull and update support and use of our new pps.toml system from #336.

~~Putting this up as a draft to start until the pps stuff is written.~~ now complete!

Probably will not be solving #290 related bugs until a follow up PR but this should give a good base to work on those.

This does solve #310's pp updates part (which was entirely missing, we were only getting pp updates on restarts of the backend..) and #311.

…nc batching

Just ignore them for now using new `match:` syntax B) but we'll do incremental update sooon! Resolves #311

No real-time update support (yet) but this is the first draft at writing trades ledgers and `pps.toml` entries for the kraken backend. Deatz: - drop `pack_positions()`, no longer used. - use `piker.pp` apis to both write a trades ledger file and update the `pps.toml` inside the `trades_dialogue()` endpoint startup. - drop the weird paper engine swap over if auth can't be done, we should be doing something with messaging in the ems over this.. - more web API error response raising. - pass the `pp.Transaction` set loaded from ledger into `process_trade_msgs()` do avoid duplicate sends of already collected trades msgs. - add `norm_trade_records()` public endpoing (used by `piker.pp` api) and `update_ledger()` helper. - rejig `process_trade_msgs()` to drop the weird `try:` assertion block and skip already-recorded-in-ledger trade msgs as well as yield *each* trade instead of sub-sequences.

Moves to using the new `piker.pp` apis to both store real-time trade events in a ledger file as well emit position update msgs (which were not in this backend at all prior) when new orders clear (aka fill). In terms of outstanding issues, - solves the pp update part of the bugs reported in #310 - starts a msg case block in prep for #293 Details of rework: - move the `subscribe()` ws fixture to module level and `partial()` in the client token instead of passing it to the instance; in prep for removal of the `.token` attr from the `NoBsWs` wrapper. - drop `make_auth_sub()` since it was too thin and we can just do it all succinctly in `subscribe()` - filter trade update msgs to those not yet stored int the toml ledger - much better kraken api msg unpacking using new `match:` synax B) Resolves #311

goodboy · 2022-07-03T18:57:47Z

piker/brokers/kraken/api.py

+    yield client
+
+
+def normalize_symbol(


Only code that's changed in this file was this routine, ow was taken verbatim from the old kraken,py.

assume not including the logic in get trades

heh, I also like this much better :)

goodboy · 2022-07-03T18:59:28Z

piker/brokers/kraken/broker.py

+
+            # process and relay trades events to ems
+            # https://docs.kraken.com/websockets/#message-ownTrades
+            async for msg in stream_messages(ws):


I'm thinking we might want to move this into a new async func to avoid so much indentation and also just be more narrowed in the scope of the code's purpose much like how we did so in ib (though that was also just a bigger mess to organize 😂)

yea, reading through it again now is a struggle haha >_<

goodboy · 2022-07-03T19:00:06Z

piker/brokers/kraken/feed.py

@@ -0,0 +1,464 @@
+# piker: trading gear for hackers


Everything here was again moved over verbatim from the old god module..

lol what a module name

goodboy · 2022-07-03T19:00:32Z

piker/pp.py

    if not pps:
        log.warning(
-            f'No trade history could be loaded for {brokername}:{acctid}'
+            f'No `pps.toml` positions cold be loaded for {brokername}:{acctid}'


goodboy · 2022-07-04T23:10:35Z

Just waiting on a stamp from @iamzoltan here and we can probably land dis stuff.

goodboy · 2022-07-05T20:37:32Z

Note, found something broke that is the same issue that is being addressed in #350 but the fix is coming in #349 so don't worry y'all 😂

Addressing same issue as in #350 where we need to compute position updates using the *first read* from the ledger **before** we update it to make sure `Position.lifo_update()` gets called and **not skipped** because new trades were read as clears entries but haven't actually been included in update calcs yet.. aka we call `Position.lifo_update()`. Main change here is to convert `update_ledger()` into a context mngr so that the ledger write is committed after pps updates using `pp.update_pps_conf()`.. This is basically a hotfix to #346 as well.

iamzoltan

Looks great @goodboy.

I really dig the new sub-module structure and it should make it easier for new eyes (and old ones) to grok this over :)

Merge Merge Merge!

iamzoltan · 2022-07-05T23:32:01Z

piker/brokers/kraken/__init__.py

+- ``broker.py`` part for orders / trading endpoints
+- ``feed.py`` for real-time data feed endpoints
+- ``api.py`` for the core API machinery which is ``trio``-ized
+  wrapping around ``ib_insync``.


I am a big fan of the sub modules!

why does the comment for api.py reference ib_insync?

lol yeah probably a mistake, i copy/pasta from the ib version of this..

iamzoltan · 2022-07-06T00:04:37Z

piker/brokers/kraken/api.py

+            by_id = resp['result']['trades']
+            trades_by_id.update(by_id)
+
+            # we can get up to 50 results per query


big fan of how clean this is. The logic is easier to follow

iamzoltan · 2022-07-06T00:14:48Z

piker/brokers/kraken/api.py

+    yield client
+
+
+def normalize_symbol(


assume not including the logic in get trades

iamzoltan · 2022-07-06T00:15:39Z

piker/brokers/kraken/api.py

+    yield client
+
+
+def normalize_symbol(


heh, I also like this much better :)

iamzoltan · 2022-07-06T00:27:08Z

piker/brokers/kraken/broker.py

+
+            # process and relay trades events to ems
+            # https://docs.kraken.com/websockets/#message-ownTrades
+            async for msg in stream_messages(ws):


yea, reading through it again now is a struggle haha >_<

iamzoltan · 2022-07-06T00:31:06Z

piker/brokers/kraken/broker.py

+
+    return records
+
+


fan of the ledger abstraction :)

iamzoltan · 2022-07-06T00:32:16Z

piker/brokers/kraken/feed.py

@@ -0,0 +1,464 @@
+# piker: trading gear for hackers


lol what a module name

iamzoltan · 2022-07-06T00:33:13Z

piker/brokers/kraken/feed.py

+
+) -> tuple[Callable, int]:
+
+    # TODO implement history getter for the new storage layer.


this would be slick!!!

Addressing same issue as in #350 where we need to compute position updates using the *first read* from the ledger **before** we update it to make sure `Position.lifo_update()` gets called and **not skipped** because new trades were read as clears entries but haven't actually been included in update calcs yet.. aka we call `Position.lifo_update()`. Main change here is to convert `update_ledger()` into a context mngr so that the ledger write is committed after pps updates using `pp.update_pps_conf()`.. This is basically a hotfix to #346 as well.

goodboy requested review from guilledk and iamzoltan June 29, 2022 17:50

goodboy force-pushed the kraken_ledger_pps branch from b5f5f0e to f93feba Compare June 29, 2022 19:32

goodboy mentioned this pull request Jul 2, 2022

Pps postmortem #347

Merged

goodboy added 12 commits July 2, 2022 16:20

Start kraken sub-pkg

b118bec

Factor clearing related endpoints into new .kraken.broker submod

90cc6eb

Move core api code into sub-module

208e2e9

Make broker mod import from new api mod

f87a2a8

Factor data feeds endpoints into new sub-mod

d3caad6

Drop wacky if block logic, while loop, handle errors and prep for asy…

9106d13

…nc batching

Avoid crash on trades ledger msgs

fcd7e0f

Just ignore them for now using new `match:` syntax B) but we'll do incremental update sooon! Resolves #311

Raise any error from response

735fbc6

Invert normalizer branching logic, raise on edge case

5d39b04

Just do a naive lookup for symbol normalization

f688805

Use new str.removeprefix() from py3.10

47777e4

goodboy force-pushed the kraken_ledger_pps branch from 4c3032b to 47777e4 Compare July 2, 2022 20:20

goodboy added 4 commits July 3, 2022 11:18

Inline process_trade_msgs() into relay loop

aea7bec

Send fill msg first

4c0f209

Handle ws style symbol schema

214f864

goodboy marked this pull request as ready for review July 3, 2022 18:54

goodboy changed the base branch from master to pps_postmortem July 3, 2022 18:56

goodboy commented Jul 3, 2022

View reviewed changes

guilledk previously approved these changes Jul 3, 2022

View reviewed changes

goodboy mentioned this pull request Jul 3, 2022

Drop token attr from NoBsWs #348

Merged

Base automatically changed from pps_postmortem to master July 4, 2022 19:28

goodboy added broker-backend `brokerd`/`datad` related backend tech brokers-can-smbz features that suits should provide ledger trade, accounts and other user focal event history tracking, management and storage labels Jul 4, 2022

Fix typo / line length

609034c

goodboy dismissed guilledk’s stale review via 609034c July 5, 2022 20:46

Create sub-pkg logger once during import

af01e89

iamzoltan approved these changes Jul 6, 2022

View reviewed changes

goodboy merged commit 419ebeb into master Jul 6, 2022

goodboy deleted the kraken_ledger_pps branch July 6, 2022 00:56


		) -> tuple[Callable, int]:

		# TODO implement history getter for the new storage layer.

Conversation

goodboy commented Jun 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

goodboy Jul 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

goodboy Jul 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

goodboy commented Jul 4, 2022

Uh oh!

goodboy commented Jul 5, 2022

Uh oh!

iamzoltan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

goodboy commented Jun 29, 2022 •

edited

Loading

goodboy Jul 3, 2022 •

edited

Loading

goodboy Jul 3, 2022 •

edited

Loading

iamzoltan left a comment •

edited

Loading