Rounding issue when matching orders

[abitmore]

As @alexpmorris reported in this steemit post, there is an rounding issue when matching orders. For example:

The last matched order was selling 0.185 BTS, but apparently it's not enough to get 21 satoshi of HERO, so finally it got 20. However, to get 20 satoshi of HERO, only need to pay 0.1838 BTS.

Current implementation is in favor of the maker (it gets all the amount that below a satoshi of the other token). I propose to change it to in favor of the taker, or anything that is fairer. For example, if the rest is less than a satoshi, cancel the order and return the rest to the taker. I guess it's hard to implement due to the design of price structure. Another side effect is that people may be unable to sell all balance of one token.

Earlier discussion in #338. Probably related: #132, #184.

[abitmore]

Actually there is a deep discussion in #132 which IMHO worth reading.

[alexpmorris]

@abitmore, as I posted on bitsharestalk last night...

I just updated the post to reflect a new understanding of the problem. You'll still have to get through @abit's flagging. However, it turns out that @abit, @pc, and myself are both right and wrong on particular aspects. It turns out it's not a margin-call related issue, though there are some things that could probably be revisited given some of the things that were brought up. And it turns out it is also a bit of a "rounding issue" after all, but not quite in a way that anyone's brought up.

Part of the update on my post:

The "problem", at least for the community to decide if they consider it a "problem", is that you're not just "buying", you're "Buying at least". THAT's where the "rounding errors" are compounding.
...
What if I want to buy 1.5 INTELLIGUYPIZZA at 4.50 TEST, but by accident I entered 345 instead. Yes, I'm aware of the "GUI warning". But that's not the point here. The offer is 4.50, and there's plenty available. So, no problem... instead, I'll just get my 1.5 at 4.50 instead of 345, right?

WRONG! Here's what you'll really be getting... bought 113 INTELLIGUYPIZZA at 4.50 TEST/INTELLIGUYPIZZA.

Now, while all this seems to answer most cases that I came across, it DID NOT seem to properly address the WHALESHARE_BTS buy 40% higher than the inside offer. UNLESS, of course, this "size adjustment" is targeting the NON-DECIMAL LEG for adjustment...

For reference, here's the link for original post with the full update: http://bit.ly/2tKiDfb

[abitmore]

@alexpmorris Thanks for the contribution. We all aware of the issue, the challenge is we need a solution or maybe find a better trade-off. If you've read through the comments in issue #132 (which I guess you haven't), perhaps you'll come with more ideas.

[abitmore]

The picture attached above explains exactly why we described it as a rounding issue: the taker has never overpaid more than 1 satoshi of WHALESHARE every take.

[alexpmorris]

There's got to be a better way to handle this @abitmore, and I'd be happy to work with you to figure it out. Now that I know exactly what's going on, I'm pretty sure I know what needs to be solved. I'll try to come up with an ideal solution, and defer to you on if and how it can be safely implemented with minimal likelihood of "unintended consequences".

[alexpmorris]

#master...alexpmorris:patch-1

@abitmore Here's a potential fix for this problem. As I described in the pull request:

I'm not sure this is the exact ideal solution, but it does appear to work in conjunction with my testnet orders, which I've included in the associated ticket for this pull request. It makes changes both in match() and fill_order() methods, with both segments marked "APM". I also had to change pays to rawPays so I could adjust the for_sale amount in fill_order().

I also tried running the tests over the new code. HOWEVER, it seems the test code anticipates that if you overpay, you will receive MORE at a lower price, as opposed to simply receiving the amount you requested at a better price. I have included that example in the ticket as well. In operation_tests2.cpp, the "buyback" tests places an order for 100 at 1000, with a buy of 1 for 100. My new code fills 1 at 10, as I believe would be expected. The test case expects to receive 10 at 10 instead (ie. from the testcase perspective, 1 at 100 should equal 10 at 10).

I also left in some additional idump() code that I used in attempting to tackle this issue, in case it is of assistance to others attempting to evaluate this as a potential fix / solution.

Here is what the test case is expecting to receive:

Since I couldn't get through all the tests, and I've already spent quite a bit of time on this, I at least wanted to share what I came up with so far, and give the community the opportunity to "rip it" apart a bit to see what else I may have missed so far.

Here were the results of my updated code, as shown in openledger when linked to my local testnet (though manually entered via cli_wallet). Notice how every trade is perfectly rounded off, and the sizes received was always exactly what I, as a trader, would expect them to be. Note in this case I was buying/selling back to a single account "nathan". However, in my test case I was using a buyer and seller on both sides, and it did not appear to trigger an assertion in verify_asset_supplies(db):

[abitmore]

1. We need to define "sell" and "buy". In BitShares core there was no "buy" but only "sell", so every order has an amount_to_sell field and a min_to_receive field. I guess if we want to implement "buy", we need a new operation, with fields amount_to_buy and max_to_pay, so it can have different behavior, and we'll need to change the limit_order data structure and a lot of related code, and need work on UI.
2. Never use double or float in consensus code. Otherwise it will lead to unintended behavior. For example, if( (a/b).to_real() > x ) can be changed to if( a > b*x ).

[pmconrad]

Was about to suggest the same.

[alexpmorris]

I used to_real() to extract the rounding error. It also seemed to work very well to identify a buy order from a sell order of the non-core asset, because all buy orders entered resulted in (real_order_price > 1), and all the sell orders resulted in (real_order_price < 1).

This makes perfect sense to me, because how else would one determine the side of the book to place an order (for both the UI and for order matching), which by extension, also enabled me to extrapolate "amount_to_sell" versus "amount_to_buy". And while this also seemed consistent throughout my testing, are you saying this may not always be the case?

Also, if there's a better way to extract both of these values using the template primitives versus using to_real() as you described, that should be a relatively minor adjustment to make.

[pmconrad]

Trying to distinguish a buy from a sell by looking at the price doesn't make sense at all. What happens to your logic if BTS goes above 1 USD, for example?
@abitmore is right, we need an explicit buy-operation. We'll never be able to reliably second-guess the traders intentions, so we must provide him with a means to tell us what he wants.

[alexpmorris]

That was my first thought as well, except that internally within each order object, base and core are maintained as total quantities to exchange versus each other, along with which side is "for_sale", which implicitly should contain all the information we need. And of course, there must still exist a consistent way that the front-end and the matching engine always knows which side to target (bids and offers).

Furthermore, every test case I've tried seems to verify this. I also gave examples where BTS was trading below 1, above 1, and where the matching even took place through 1. Here are all the test cases, along with a few screenshots as well. Again, all these test cases behaved exactly as I would have expected:

sell_asset nathan 250 BTS 1000 TEST 100000 false true <-- BUY 1000 TEST @ 0.25
sell_asset nathan 1000 TEST 250 BTS 100000 false true <-- SELL 1000 TEST @ 0.25

sell_asset nathan 1000 BTS 250 TEST 100000 false true <-- BUY 250 TEST @ 4
sell_asset nathan 250 TEST 1000 BTS 100000 false true <-- SELL 250 TEST @ 4

TAKER SELLS (below 1 / above 1)

sell_asset nathan 25 BTS 100 TEST 100000 false true <-- BUY 100 TEST @ 0.25 (bts)
sell_asset nathan 50 BTS 100 TEST 100000 false true <-- BUY 100 TEST @ 0.50
sell_asset nathan 300 TEST 50 BTS 100000 false true <-- SELL 300 TEST @ 0.16667
result: 100 TEST filled @0.50, 100 TEST filled @0.25, remainder: 100 TEST offered @0.16667

sell_asset nathan 400 BTS 100 TEST 100000 false true <-- BUY 100 TEST @ 4 (bts)
sell_asset nathan 300 BTS 100 TEST 100000 false true <-- BUY 100 TEST @ 3
sell_asset nathan 300 TEST 600 BTS 100000 false true <-- SELL 300 TEST @ 2
result: 100 TEST filled @4, 100 TEST filled @3, remainder: 100 TEST offered @2.00000

TAKER BUYS (below 1 / above 1)

sell_asset nathan 100 TEST 25 BTS 100000 false true <-- SELL 100 TEST @ 0.25 (bts)
sell_asset nathan 100 TEST 50 BTS 100000 false true <-- SELL 100 TEST @ 0.50
sell_asset nathan 300 BTS 300 TEST 100000 false true <-- BUY 300 TEST @ 1
result: 100 TEST filled @0.25, 100 TEST filled @0.50, remainder: 100 TEST bid @1.00000

sell_asset nathan 100 TEST 400 BTS 100000 false true <-- SELL 100 TEST @ 4 (bts)
sell_asset nathan 100 TEST 300 BTS 100000 false true <-- SELL 100 TEST @ 3
sell_asset nathan 1500 BTS 300 TEST 100000 false true <-- BUY 300 TEST @ 5
result: 100 TEST filled @3, 100 TEST filled @4, remainder: 100 TEST bid @5.00000

TAKER SELLS (above 1 through below 1)

sell_asset nathan 125 BTS 100 TEST 100000 false true <-- BUY 100 TEST @ 1.25 (bts)
sell_asset nathan 80 BTS 100 TEST 100000 false true <-- BUY 100 TEST @ 0.80
sell_asset nathan 300 TEST 150 BTS 100000 false true <-- SELL 300 TEST @ 0.50
result: 100 TEST filled @1.25, 100 TEST filled @0.80, remainder: 100 TEST offered @0.50000

TAKER BUYS (below 1 through above 1)

sell_asset nathan 100 TEST 80 BTS 100000 false true <-- SELL 100 TEST @ 0.80 (bts)
sell_asset nathan 100 TEST 125 BTS 100000 false true <-- SELL 100 TEST @ 1.25
sell_asset nathan 450 BTS 300 TEST 100000 false true <-- BUY 300 TEST @ 1.50
result: 100 TEST filled @0.80, 100 TEST filled @1.25, remainder: 100 TEST bid @1.50000

And finally...

TAKER SELLS small lot too low
sell_asset nathan 150 BTS 100 TEST 100000 false true <-- BUY 100 TEST @ 1.50 (bts)
sell_asset nathan 11 TEST 5 BTS 100000 false true <-- SELL 11 TEST @ 0.454545
result: 11 TEST sold / filled @1.5

TAKER BUYS small lot too high
sell_asset nathan 100 TEST 80 BTS 100000 false true <-- SELL 100 TEST @ 0.80 (bts)
sell_asset nathan 10 BTS 11 TEST 100000 false true <-- BUY 11 TEST @ 0.90909
result: 11 TEST bought / filled @0.80

Note, if you would like to try reproducing these results, every example I gave should work directly in the cli_wallet (just leave off the <-- comment). I also used:

get_limit_orders TEST BTS 100
cancel_order 1.7.8 true

between test sessions to retrieve order ids and cancel orders for the next test.