Add SIMD-0344: Dynamic State Rent #344
Conversation
There was a problem hiding this comment.
This SIMD correctly identifies the need to address the cost of Rent without further incentivizing excessive state growth, along with the need to decrease the size of snapshots and working memory. The proposed implementation, however, is full of holes and not really compatible with Solana as it exists today due to it proposing we violate the Solana account ownership model and break all existing deployed programs. I think this one should go back to the drawing board.
|
|
||
| **Access Control:** Only callable by the bank during epoch boundary processing | ||
|
|
||
| ### Account Rent Paid Field |
There was a problem hiding this comment.
I think there is about a 0% chance we would make such a drastic change to accounts. If the idea was to expose these values in the VM so that they could be used for Rent collection during compression, this would increase the cost of Rent across the board and break all currently deployed programs. If the idea was not to expose them to the VM and they were only used in the bank, then there isn't really a good way for the validator to express to the network that they have claimed these compression rewards.
proposals/0344-dynamic-state-rent.md
Outdated
| account.last_write_epoch = current_epoch; | ||
| ``` | ||
|
|
||
| #### 3. First Write Per Epoch |
There was a problem hiding this comment.
This is reintroducing the concept of non-rent-exempt accounts which is an absolute nightmare. I do not think anyone who has been around long enough to have experienced that would recommend pursuing this direction.
There was a problem hiding this comment.
IIUC the issue with the old rent system was that payments happened on stale accounts that had not been accessed during the epoch and these all happened around the epoch boundary which lead to problems with handling the bank state when forks crossed the epoch boundary. If that actually was the issue then I don't think we have the same problem here because we don't have the automatic rent payments on stale accounts at all.
There was a problem hiding this comment.
any behavior by which an account pays for <6960 lamports per byte, such as only paying for 15 epochs (a concept we have now voted to deprecate), must either result in that account underpaying for rent exemption, or subsequently becoming non-rent-exempt at an epoch boundary. Neither sound like desired outcomes to me.
There was a problem hiding this comment.
The difference vs the old rent system is that accounts aren’t deleted. They are compressed and cannot be used until decompressed. So the program STF remains exactly the same. Dev UX is much simpler too, because the account rent is paid on access. It’s amortized between all the users, so devs don’t need to worry about it.
There was a problem hiding this comment.
rent calculation sucks. we should be trying to make it suck less.
proposals/0344-dynamic-state-rent.md
Outdated
| } | ||
| ``` | ||
|
|
||
| ### Compression Reward |
There was a problem hiding this comment.
Conceptually, not a bad idea to incentivize nodes to compress accounts. In practice, this raises a question of whether or not the validators should have the authority to unilaterally compress someone else's accounts. I do not believe they should. The user paid a price for a service, was offered Rent exemption for that price, and should not just suddenly have that service denied to them. It should be up to the individual to decide to compress their account, and we should allow the user themselves to reclaim unused rent in the same way this works for compressed accounts in light protocol.
There was a problem hiding this comment.
Compression should be outside of the core account model imho - If users want to utilize compression on accounts why not move this up the stack as a superset of the system program instead?
Users then have the option to pay for actual state size, or can opt for the cheaper compressed version. Given compression has consistently failed to get any traction on Solana despite the multiple attempts at it - Forcing users to compress seems like a loosing battle.
There was a problem hiding this comment.
Conceptually, not a bad idea to incentivize nodes to compress accounts. In practice, this raises a question of whether or not the validators should have the authority to unilaterally compress someone else's accounts. I do not believe they should.
This is just flat out wrong. Nothing about the program STF changes.
There was a problem hiding this comment.
Users then have the option to pay for actual state size,
it is permissionless to deposit more lamports into any account and keep it resident. The amount of space is finite, so either rent has to go up or accounts need to be compressed. The network needs to lower the cost to allocate to stay competitive, so the only option is compression.
| } | ||
| ``` | ||
|
|
||
| **Fee Payer Validation:** |
There was a problem hiding this comment.
We should not be making fees more complicated here. As mentioned in a previous comment, this could just be a simple instruction in a program that does not violate any existing rules of ownership or authority and would be executed like any other instruction without the need to add any new complexity to fee collection.
| the sol burned in rent so no net new sol is created through this rent | ||
| mechanism. | ||
|
|
||
| ## Detailed Design |
There was a problem hiding this comment.
I do not think these values need to be dynamic at all. It is sufficient to simply pick a constant that is expensive enough to avoid DoS and throw in a future SIMD to reduce the cost later if required once the API, protocol and other economic aspects of this idea stabilize. The tradeoff in complexity of understanding how much rent is due is not worth it, and 2-day-long epochs are way too slow for us to actually take any kind of meaningful action against unsustainable state growth due to DoS anyway.
There was a problem hiding this comment.
Part of the goal of the SIMD is to ensure that account data can fit in memory so that we avoid the problem of differentially pricing memory/disk loads. Doing that requires a target account size which can only be achieved with a dynamic controller. We want to find D^-1(target_account_size) i.e. the price at which target_account_size account size is the quantity demanded but we don't know D.
There was a problem hiding this comment.
Correct me if I'm wrong, but if we take events with high amounts of state growth like the $TRUMP launch, or the $JUP airdrop as examples, I feel like, unless luckily timed on an epoch boundary, what this algorithm would actually accomplish is it would do nothing to disincentivise the high state growth event when it occurs, then would subsequently punish users by making it cost a lot more to grow state after the usage spike plummets due to the very long length of epochs. Hence I believe a well-reasoned constant would, on average, probably be more effective.
There was a problem hiding this comment.
You're absolutely correct that this controller isn't sufficiently sensitive or responsive to spikey state growth to be a complete solution. It's like having a thermostat with a 2 day reaction lag.
This is by design though. We don't want a hyper responsive controller because an increase in rent affects all accounts; a sudden spike may cause a significant portion of previously well funded accounts to become delinquent before rent normalizes again. Instead, the controller is concerned more with macro trends, adjusting rent more gradually, which has the dual effect of increasing account compression (more accounts become delinquent) and decreasing new allocations.
More responsive rate limiting can be achieved with explicit block level constraints on new allocations. This already exists (ref), though it can probably be improved.
There was a problem hiding this comment.
IMHO, the controller is a v2 problem is way over complicating things.
- use a simple static function: RentFunc(total allocs, total frees, requested bytes) => lamports per rent resolution slot per byte
Use a simple account.creation_slot
anytime RentFunc * (current slot - account.creation_slot) > account.balance, any validator can submit a TX to compress this account and claim the account.balance.
There is no epoch boundary, the system responds at any time. Yes, for older accounts, the threshold is much more sensitive. But who cares, they are the older accounts who are accessed infrequently, which is why they are old. So it doesn’t really matter if they are not cleaned up in the best possible order, as long as they are cleaned up before high frequency accounts.
if rent solution is 1 slot, then access fee to bump a 10mb account by one slot is like $0.002 at current prices. Which is totally fine.
There was a problem hiding this comment.
my point is orthogonal to the question of using a PID controller vs a hardcoded rent curve. Both options should be less sensitive to slot-local state growth, because PID uses epoch level granularity and rent curve doesn't care about rate of growth at all. Either way, we'd want to limit bytes allocated per block with an explicit constraint.
There was a problem hiding this comment.
Bytes allocated per block isn’t really important. The runtime needs a read and write limit that should be the VM scheduler constraint. Chilly Peppers covers this.
- Removed the rent payment syscall that allowed any program to pay rent for any account - This was a UX feature but violated Solana's account ownership principles - Removed decompressAccount from RPC methods as it should be client-side - Fixed typo: 'elligable' -> 'eligible' - Updated breaking changes section to remove rent payment handling requirement
|
I don't get how this proposal will actually prevent state bloat. A compressed account has at least still the account storage overhead right? If the goal of this proposal is to make account creation cheaper then people will airdrop more again and more compressed accounts (stale token accounts) will occupy the state. Actually in the airdrop example people might now even have the incentive to close the account and claim the rent anymore because they first would need to decompress with a transaction and there would be no rent to claim? |
In v0 compression, we are just compressing account data by hashing it but with a vector commitment, the accounts can be further compressed to something sub linear in the number of accounts. Then we must also handle index compression which requires a bloom filter. For now this proposal will compress a lot of stale account state which is likely to never be touched again. |
MaxResnick
changed the title
Would a v0 version actually have immediate positive impact given that every account will incur the overhead of two additional fields and there's still a hash to store? I don't have the numbers. I'm asking because going beyond v0 feels difficult. I don't understand how a bloom filter can be used in a multihreaded runtime in a safe and deterministic way. I think there's a high chance it would get stuck in v0. |
This is straight up wrong. A user can recreate the exact same state by decompressing the account and depositing lamports into it, so the program STF remains the same. |
|
Directionally correct, but some implementation problems. We can't mint inside txs. A ton of the code in the runtime correctly does assertion checks to make sure that pre/post lamports balances are the same. If txs can now mint lamports it would be impossible to guard against future bugs this way. There is just no way this approach will ship. The audit overhead and risk is way too high. Minor issue is that the PID controller depends on look back of state. This is a pita to maintain through forks but can be solved with a sysvar. If you want to actually ship this, I suggest simplifying the design to what original LSR, and addressing more of the DX pain points.
Current rent cost per "slot"
Current cost to decide deletion
I know that older accounts get hit harder without a pid controller. But these are the least important accounts because they haven't been accessed the longest. So it doesn't matter what happens to them. Because they are not frequently accessed, devs can infrequently deposit sol into them to maintain them. So it works out. The benefit is that its just hella simpler to not have look back and to only use 1 variable creation slot Automatic MaintenanceOn access a Fee payer burns (slot resolution)*RentFunc(tx accounts bytes) worth of sol, and each account's creation slot is bumped by slot resolution forward. This effectively prevents the account form being cleaned up just from normal access. Slot resolutionIf we use slot level resolution, then a 10mb account access fee to bump the creation slot into the future by 1 slot would be 0.00000085 sol at current prices. It might make sense to not charge the maintainer fee in the fee payer if the account.balance covers 2 years worth of rent already at current prices. Ideally this is some function just reduces the access fee gradually instead of on/off at some time boundary. Rent * K/(time) or something |
168 byte spl token airdrop account would get compressed into a 32 byte hash. So memory pressure will reduce. |
We actually want the bloom filter to be machine diversified.
If 3 is machine diversified then we see 1:1e6 spurious reads that attacker can't predict on the leader. the rent cost function would need to deterministcally estimate the size of the bloom filter if we want to track its overhead in ram and price the on disk index separately, but this is easy to guess with super low error. |
Super interested in hearing your means of making all currently deployed programs interact with a serialized input region they were not compiled for. |
proposals/0344-dynamic-state-rent.md
Outdated
| least 15 epochs worth of rent at the current rate. When an account is written | ||
| for the first time in an epoch it pays at least 1 epoch worth of rent. If an | ||
| account has not payed rent in a while, and the rent due is more than the rent | ||
| paid, the account becomes eligible for compression at which point the leader |
There was a problem hiding this comment.
This is going to be an absolute nightmare for users having to pay an additional 1 epoch of rent in an undeterministic way (would this be a new ix akin to TopUpRent or just rely on the account having sufficient lamports? The latter would surely result in double spends from users/dapps during blocks when cold state is needed?
Other than "hoping" valis/indexers store all the compressed data & never loose it (cough initial solana ledger blocks cough) this just seems like its pushing the issue down the road & will come back to bite given there is no incentive for the indexers to keep storing cold state data if they can make more off hot state
There was a problem hiding this comment.
This is going to be an absolute nightmare for users having to pay an additional 1 epoch of rent in an undeterministic way (would this be a new ix akin to TopUpRent or just rely on the account having sufficient lamports? The latter would surely result in double spends from users/dapps during blocks when cold state is needed?
No it won't. Just like priority fees, the rpc can return the expected amount charged at any given point in time.
At the current rent 0.002 sol for 168 bytes for 2 years would require a fee payer to pay 0.0000055 sol to bump the account by one epoch. Which is the same price as the current signature fee.
Since 80% of current accounts haven't been used in 6 months I'd expect the state size to drop by 80%, and the price to drop by more than 80% because the controller is super linear.
| - **last_write_epoch**: `u64` field tracking when account was last written | ||
| - Added to account metadata alongside existing fields (lamports, data, owner, | ||
| etc.) | ||
| - **rent_paid** is set to zero when account is compressed (rent is "consumed" |
There was a problem hiding this comment.
16 bytes on EVERY single account seems like a huge state bloat in of itself? Why does this need handling at account level?
This also breaks all existing indexer solutions & anything utilizing size/space calculations on accounts
There was a problem hiding this comment.
@MaxResnick Really wants a stateful controller. I like a simple stateless curve that only needs the accounts to store a creation slot, 5 or 6 bytes would be enough.
proposals/0344-dynamic-state-rent.md
Outdated
|
|
||
| ```c | ||
| // MUST pay at least 15 epochs worth of rent upfront | ||
| required_rent = account_data_size * current_rent_rate * 15; |
There was a problem hiding this comment.
Where did 15 come from? Any info around why it needs to be so long? If we are reverting back to non-exempt rent then why not just keep it to N+1 to ensure the lowest rental costs needed for account creation?
This would drastically impact the cost of token creation etc in DeFi and push more cost onto the user having to pay for 15 epochs when in reality most tokens only need a couple of epochs rent before they are totally cold.
Opening up shorter term rent options also enables more usecases around temp accounts if the cost is sufficiently low
There was a problem hiding this comment.
I agree that 15 epochs is excessive. It seems like a reasonable default so users aren't accidentally creating short-lived accounts that need to be decompressed constantly, but too high for the minimum requirement.
There was a problem hiding this comment.
On second thought, making the upfront cost higher makes sense if we compress upon delinquency rather than just delete. Compressed accounts aren't free, they still bloat the state with their pubkeys. Allowing a bunch of extremely cheap, short-lived accounts would be bad unless we add an exception where those accounts are deleted.
proposals/0344-dynamic-state-rent.md
Outdated
| } | ||
| ``` | ||
|
|
||
| ### Compression Reward |
There was a problem hiding this comment.
Compression should be outside of the core account model imho - If users want to utilize compression on accounts why not move this up the stack as a superset of the system program instead?
Users then have the option to pay for actual state size, or can opt for the cheaper compressed version. Given compression has consistently failed to get any traction on Solana despite the multiple attempts at it - Forcing users to compress seems like a loosing battle.
| **Modified RPC Methods:** | ||
|
|
||
| ```javascript | ||
| getAccountInfo() // Returns compression status in account metadata |
There was a problem hiding this comment.
Are you envisioning that the data returned from this call is in its compressed or decompressed state (client side)? This is a huge breaking change if so.
If compression is added to an account it should be opaque to the user that its compressed, rather there should be an intermediary step that ensures this doesnt break everything:
New Encoding param: Compressed - Returns the compressed state
All other encoding params should do in-flight decompression & return the decompressed state regardless (RPC's have more than enough capacity for this)
|
|
||
| ```c | ||
| // Begin rent collection on first write | ||
| // Existing accounts get grace period - no historical rent owed initially |
There was a problem hiding this comment.
ALL existing accounts are going to be forced into this new concept? Historical accounts have paid for rent exemption on state - i dont think they should be subjected to compression & account size increase.
I dont see how all accounts could feisably move to this new model without SF etc fronting the cost for the additional rent costs & opt-in migrations result in a fractured account ecosystem with older apps never upgrading.
There was a problem hiding this comment.
All accounts are compressed. No exemptions to historic accounts. It's always been permissionless to add lamports to accounts, so anyone that wants to can keep them hot.
|
|
||
| - Deploy updated RPC methods for compression and rent APIs | ||
| - Update transaction simulation to include rent costs | ||
| - Implement fee payer validation with rent collection |
There was a problem hiding this comment.
- Implement in-flight decompression to avoid breaking existing implementations
** Dapps **
- Implement dynamic rent estimations
- Implement split logic between hot & compressed state
- Additional intermediary logic for handling account decompression (dapps unlikely to want the user paying here - adds huge UX burden)
- Offset & Size calls all need adjusting to account for the new fields
- Scheduled/Cron to ensure any relevant account state is kept decompressed with sufficient rent (This is a huge PITA ops wise)
There was a problem hiding this comment.
This is a lower cost than the alternative. cost of off chain work vs onchain work is 1:1000
Triton/helius/metaplex will gladly take $ per account you want to keep incrementally warm
proposals/0344-dynamic-state-rent.md
Outdated
| The proposal introduces two new system variables (Sysvars): | ||
|
|
||
| - Dynamic Rent History (`sysvar: sol_get_dynamic_rent_history`, id TBD) | ||
| - Epoch State Size History (`sysvar: sol_get_epoch_state_size_history`, id TBD) |
There was a problem hiding this comment.
This identifier sol_get_dynamic_rent_history, is this part of the SDK or part of the protocol (e.g. syscall)
There was a problem hiding this comment.
It is a sysvar named like a syscall.
|
Can you give insights into how you came to conclusion that state rent is too high? From my personal perspective paying <1$ state rent for a token account that i can use my whole life feels fair. After all i can claim back the rent. Program deploys are in the range of 1-5 SOL maybe? I don't know a startup or project which has failed because of this cost. |
Competitors have lower state costs. So the foundation is being asked to subsidize companies that are choosing competitors, at a cost much greater than the engineering effort to land this.
Devs always want lower cost deploys |
the TXs would fail in the runtime. Accounts that are compressed cannot be executed unless passed to the decompress program. Transaction processor can check the account type and abort if this is violated. So program STF remains exactly the same. User needs to decompress the account back to the same state, and no other state can ever occupy the same index key. |
Yeah but they won't get it because a byte is a byte there's not much engineering to do here to help out. There's simply a disparity between the state and the throughput an optimized blockchain can provide. I'm personally fine if state costs gatekeep against programs that don't work in harmony with this disparity. |
|
on the topic of using a Fixed Rent Curve (RentFunc) instead of a PID controller, I do agree the simplicity of the former is desirable but the lack of an explicit target is concerning. If we assume a fixed SOL price, the network would eventually find an equilibrium state size where the number of allocations roughly matches the number of compressions/deallocations. This is essentially an implicit target, except we won't know what it is ahead of time or be able to control it. Furthermore, if we now assume (ha!) a volatile SOL price, this equilibrium (and therefore the implicit target) changes w/o a corresponding change in demand for state. So, not only do we have no control over what the target actually is, it's also constantly shifting. How serious this is depends a lot on the curve of course. There's a rough correspondence between the steepness of the curve at a certain point and the sensitivity to SOL price volatility. I'm not totally against going with the fixed curve for now and upgrading to full PID later, but if we do decide to go that route the curve params will need to be selected very carefully. |
That is not how this would work at all in practice. For an uncompressed account, This SIMD:
For compressed accounts, it:
Not like this, please. |
Quadratic or faster curve would be effectively the same thing. If it's 10x current price at 200gb and grows quadratically there is just not enough sol to double the snapshot. |
igor56D
left a comment
igor56D
left a comment
There was a problem hiding this comment.
Overall, I'm aligned with the general vision. Main critiques:
- awkwardness around epoch boundary: paying rent on first epoch write is tricky to plan around for users and devs.
- fuzzy miner incentives: it's unclear if the reward will be sufficient to incentivize compression. Ideally, the reward would be more predictable or roughly go up with the size of the delinquent account.
- questionable devex: devs need to plan around rent payments, epoch boundaries, etc. It would be preferable if they were given more control to determine their own rent payment strategies.
- maybe I missed it but I don't see read-only/read-mostly accounts (like program accounts) explicitly addressed. Because rent is paid on write-access only, these accounts need to be manually topped up.
Some suggestions:
- change
account.last_write_epochtoaccount.creation_epochand haveaccount.rent_paidrepresent the total rent paid over the account's lifetime rather than subtract the rent owed at unpredictable times. The sysvars can be used to determine the actual rent balance of the account. This reduces the significance of the epoch boundary and gives more options for incentives because the rent isn't burned. - Remove the in-protocol rent payment requirements. Devs could either directly enforce rent payments in their programs or a field can be added to accounts indicating how much rent each tx writing to it needs to pay.
proposals/0344-dynamic-state-rent.md
Outdated
|
|
||
| ### Account Rent Paid Field | ||
|
|
||
| Each account gets new fields: |
There was a problem hiding this comment.
We can be more stingy with bytes here. 4 bytes for last_write_epoch is more than enough. Even 2 bytes might be doable but we should leave some wiggle room in case we reduce epoch length in the future.
proposals/0344-dynamic-state-rent.md
Outdated
|
|
||
| ```c | ||
| // MUST pay at least 15 epochs worth of rent upfront | ||
| required_rent = account_data_size * current_rent_rate * 15; |
There was a problem hiding this comment.
I agree that 15 epochs is excessive. It seems like a reasonable default so users aren't accidentally creating short-lived accounts that need to be decompressed constantly, but too high for the minimum requirement.
proposals/0344-dynamic-state-rent.md
Outdated
|
|
||
| #### 3. First Write Per Epoch | ||
|
|
||
| On the FIRST write to an account each epoch: |
There was a problem hiding this comment.
Why is it necessary to single out the first write in an epoch? This change makes the epoch boundary more visible to users and devs. We can leave it up to devs/users to decide when to pay rent and how much to include per payment.
Also, the protocol/validators don't need to proactively check for delinquency like you're doing here, that's what the miner incentive is for. Transactions are welcome to access delinquent accounts while they're still available.
There was a problem hiding this comment.
will change to slots
|
|
||
| // Payment processing | ||
| account.rent_paid += transaction_rent_payment; | ||
| account.rent_paid -= rent_owed; |
There was a problem hiding this comment.
If an account isn't written to every epoch, couldn't account.rent_paid go negative?
proposals/0344-dynamic-state-rent.md
Outdated
| // Reward is capped at the minimum of one epoch's worth of current rent and | ||
| // the account's rent_paid. | ||
| // This second condition ensures that no new SOL is minted by this scheme. | ||
| reward = min(account.rent_paid, account.data_size * current_rent_rate); |
There was a problem hiding this comment.
maybe I'm missing something, but isn't it possible account.rent_paid is nearly 0 or even negative (for the reasons I mentioned in a previous comment)? In that case, there'd be no incentive to compress this account.
proposals/0344-dynamic-state-rent.md
Outdated
| **Compression Timing:** | ||
|
|
||
| - **Risk**: MEV extraction from compression reward opportunities | ||
| - **Mitigation**: Only the leader can compress transactions |
There was a problem hiding this comment.
Is MEV really an issue here? If only leaders can compress then how do they determine which accounts to compress? It's an implementation detail but a non-trivial one imo. Opening compression up to everyone is simpler because it moves that complexity outside the validator client software.
There was a problem hiding this comment.
How would you even enforce this anyway? A new LeaderSchedule Sysvar? sol_current_leader syscall?
There was a problem hiding this comment.
validation doesn't need to occur in the VM. This property can be enforced earlier in the transaction processing pipeline, though I'm not necessarily suggesting we do that.
There was a problem hiding this comment.
I still think we don't really need to enshrine this at all, but if we're going to, I'm really not a fan of moving lamports in the VM without also checking permissions in the VM.
proposals/0344-dynamic-state-rent.md
Outdated
| ```c | ||
| // Initialize new account fields for all existing accounts | ||
| for (account in all_accounts) { | ||
| account.rent_paid = 0; |
There was a problem hiding this comment.
if account.rent_paid is zero for old accounts then where does the mining incentive come from? I don't really see a way around this edge case w/o printing SOL but I think it's fine to lean on good samaritans (eg anza, foundation, etc) to submit these compression requests since this is a one-off.
|
I want to throw in an alternative model that works more similiar to how rent works in a hotel market:
The advantage here is that account lifetime is predictable for users and the rent system can adjust quickly to spikes in demand without affecting existing accounts. |
This is basically my original LSR proposal. RentFunc(current_slot - account.creation_slot, account.data.len) > account.balance; it's elegible for compression. Anyone can extend it by sending it lamports. |
I think focusing on a "deletion slot" gives some nice properties:
On the other hand it only works if rent is actually burned on payment. Since there's no notion of a creation slot it's not possible to track past rent and claim it back. |
In the simple LSR idea, rent is burned and claimed by the validator on compression. Extending the deletion slot can be done by anyone by depositing lamports into that account as a normal tx. Deletion slot has to be variable and based on the current cost to allocate otherwise the system can’t respond to memory pressure. If the usage goes up, rent price has to go up, and low priority data has to be reclaimed. Otherwise this is kind of pointless. |
If deletion slot isn't variable it can still respond to memory pressure. New allocations simply become (exponentially) more expensive when memory pressure becomes an issue. It's just a tradeoff between penalizing existing allocations or new allocations. |
|
@GustavAlbrecht we need the system to lower the cost asap to meet current developer demand, which ideally means removing old infrequently used state that no one has bothered to pay for. |
|
Any thoughts on how this will impact transaction stuffing too? At the moment, most DeFi swaps are close to the tx size limit If we then have to include 2/3 rent decompression calls, too the current tx size is going to be insufficient - Sure we can bundle but its just a patchy workaround around |
If the new rent_slot field can use the legacy rent_epoch field then a transition could look like
Probably it would compress a lot of token accounts so here's a related idea to safeguard against indexers messing up in transition phase:
The advantage is that users can decompress their account by brute forcing over amount and comparing SHA256 output with onchain hash. No need to rely on indexer companies to restore token account. In a super conservative approach the above mechanism of setting rent_slot to current_slot + grace period if rent_epoch == u64::max could be restricted to token program in a first step. Others would just have u64::max and are never elgibile for compression. They could follow later. |
|
I like this idea, but (as @aeyakovenko said above) the max offset would need to be low enough to allow the protocol to adjust to state size changes effectively. Only penalizing new allocations privileges existing accounts too much, which slows down eviction. A cap of < 10 epochs w/ a simple future rent premium formula might do the trick. We may need to go further and explicitly limit the amount of account data using this future rent feature and have them compete on premiums (idk just spitballing). This shouldn't be the sole mechanism though, since it requires a very active approach to maintaining account solvency due to the low offset cap. The risk of getting an important account compressed, rendering it temporarily unusable, is non-negligible (e.g. automation bug). Program accounts will likely be especially vulnerable because (1) users don't pay rent since accesses are read-only and (2) decompressing is a pain if the compiled program is large. A hybrid approach may be best:
This is very similar to LSR in spirit; the main difference being the option to pay future rent. (I use epoch granularity above but this can be adapted to slot granularity as well) |
|
I think a max offset definitely makes sense. What about simply bumping up the deletion slot on every write lock? If the primary goal is to remove dead accounts this should work. Accounts like token mint or token metadata would likely need monitoring and artifical write-lock transactions since they exist primarily for reference and reading. I think the current rent excempt reservere system wouldn't need to change. The reward for compression could simply come from claiming part of the rent excempt reserve after shrinking the account length. I think an implementation would be straightforward. The only question here is how much a single write-lock bumps up a token account (165 byte)? This would be the only new constant next to the max slot offset. |
I wouldn't say that's the primary goal -- necessary but not sufficient. Rent and eviction is needed for a complete and sustainable solution to state growth, otherwise we'd end up back here again. |
proposals/0344-dynamic-state-rent.md
Outdated
| least 15 epochs worth of rent at the current rate. When an account is written | ||
| for the first time in an epoch it pays at least 1 epoch worth of rent. If an | ||
| account has not payed rent in a while, and the rent due is more than the rent | ||
| paid, the account becomes eligible for compression at which point the leader |
There was a problem hiding this comment.
why make it so only the leader can submit the transactions?
There was a problem hiding this comment.
To avoid MEV spam trying to collect the compression rewards.
There was a problem hiding this comment.
I don't see how this is MEV. spam, sure. but I don't see the spam as an issue. if people are able to find gaps in accts not compressed by the default leader strategies and are willing to risk paying fees without collecting reward to do so that's a good thing! It leaders to faster compression of the state, and if participants view that as worthwhile then it is beneficial to the network by reducing state size at a faster rate.
I imagine most will be captured by the leader if there's spare block capacity. But putting this into consensus is extraordinarily weird in my view.
proposals/0344-dynamic-state-rent.md
Outdated
| account has not payed rent in a while, and the rent due is more than the rent | ||
| paid, the account becomes eligible for compression at which point the leader | ||
| can submit a transaction with compression instructions and earn a reward for | ||
| decompressing the accounts. This reward is set such that it is never more than |
There was a problem hiding this comment.
decompressing the accounts.
compressing not decompressing, right?
| **Dynamic Rent History** | ||
|
|
||
| - Array of `u64` values (8 bytes each) | ||
| - `[rent_rate_0, rent_rate_1, rent_rate_2, ...]` | ||
| - Index N = rent rate for epoch N (starting from activation epoch) |
There was a problem hiding this comment.
i mean it's small...but this just grows indefinitely? why do we need the history of rent forever?
There was a problem hiding this comment.
I don't really think this is a practical problem. The controler with historical rent is MUCH better than the one without it. I'll write up some proofs of this ig since it's non trivial.
| Lowering state rent therefore requires a plan to deal with the resulting state | ||
| growth. That plan, as outlined across SIMD 0329, SIMD 0341, and this SIMD | ||
| consists of 3 components: | ||
|
|
There was a problem hiding this comment.
The motivation doesn't describe why we're motivated to make this dynamic.
It talks only about how rent is too high. Why not just use a lower constant with the compression scheme described?
There was a problem hiding this comment.
Dynamic rent is necessary to respond faster to a spike in demand. More state allocated => price goes up => more accounts get evicted.
There was a problem hiding this comment.
faster
on an epoch basis? 2 days is NOT a fast response.
There was a problem hiding this comment.
2 days is much faster than manually aligning on and deploying an update to a consensus critical variable every time global state gets too large or small. Something faster would be nice but not necessary unless you expect massive (~50GB) state growth in an epoch. A sub-epoch interval might be reasonable but single slot is probably excessive.
There was a problem hiding this comment.
rent must be dynamic because we do not know the demand curve and in fact the demand curve may change overtime (particularly if it is denominated in solana, a volatile asset whose volatility is the reason rent is so high right now in the first place).
proposals/0344-dynamic-state-rent.md
Outdated
| Each account gets new fields: | ||
|
|
||
| - **rent_paid**: `u64` field storing total rent paid in lamports | ||
| - **last_write_epoch**: `u64` field tracking when account was last written |
There was a problem hiding this comment.
if we're gonna store as a u64 we might as well just store the last write slot instead? though reasonably neither of those will ever need close to 64 bits!
proposals/0344-dynamic-state-rent.md
Outdated
|
|
||
| ### Account Rent Paid Field | ||
|
|
||
| Each account gets new fields: |
There was a problem hiding this comment.
Is the plan here to use the now unused executable and rent_epoch bytes for these new fields? I don't think we want to make all accounts actually larger, right?
There was a problem hiding this comment.
Yeah that seems like a good idea
| 1. An economic mechanism to limit state growth (This SIMD) | ||
| 2. A predicate to determine which accounts can be compressed (This SIMD and | ||
| SIMD 0329) | ||
| 3. A compression scheme that removes accounts from the global state while |
There was a problem hiding this comment.
If state-growth is what we're worried about compression still has overhead and just delays the issue without solving it. With enough users this will still baloon over time to the point where we cannot store the compressed metadata and we end up back in the same situation, no?
It's unclear why we want compression under consensus at all.
If we want a consistent scheme and storage mechanism for apps to use, great, write that SIMD, make it as easy to use and encourage apps to adopt that scheme, but do not enforce it.
IF apps want to support compression, let them and it can be done in anyway they see fit - using some agreed upon scheme or something custom.
If we want to bring back some sort of rent, then let's do just that. No more exemption and get rid of one of the primary issues with the old rent system, that it was collected outside of transactions between slots. This caused large delays between slots because there was a ton of extra work needed to be done by scanning the accounts. If we add tracking fields to each account like described below, just track the owed rent, and if it's higher than current balance then ANYONE can just snipe the account for a reward. No compression in protocol, just delete it completely.
What's great here is that any (upgradeable) program can just add permissioned or non-permissioned ixs to compress or decompress accounts. We're worried about something like user token accts getting deleted? okay, add some ix on token-program to compress/decompress accounts, make it permissionless (with conditions ABOVE rent-deletion conditions), let compressing user get some reward for compressing. for something so central to solana's ecosystem like that, we can even probably get the foundation to run a service to make sure no token-accounts get fully deleted, though we probably don't need to do that since it'd likely be profitable given the reward!
There was a problem hiding this comment.
With enough users this will still baloon over time to the point where we cannot store the compressed metadata and we end up back in the same situation, no?
This SIMD only describes the compression condition, not the mechanism itself. Currently, SIMD-0341 describes v0 compression, which does indeed grow state linearly in the number of compressed and active accounts because the pubkeys and data hash are preserved and replicated. In a future version, all compressed account data/metadata will be replaced with a fixed size vector commitment, so only the active accounts will contribute to the state size.
If we want a consistent scheme and storage mechanism for apps to use, great, write that SIMD, make it as easy to use and encourage apps to adopt that scheme, but do not enforce it.
I think the term "compression" being used has caused some confusion because people associate it with other user-space features intended to give app developers cheaper storage options. To be clear, compression in the context of this proposal is not primarily intended to provide an alternative to on-chain storage. It's an alternative to deleting delinquent account altogether. Put bluntly, the lack of choice is precisely the point.
No compression in protocol, just delete it completely
Even compressing delinquent accounts is controversial, my guess is that outright deleting them would be a non-starter. I can see why deletion would eventually be necessary if we store all compressed account pubkeys, but using a fixed-size commitment solves that.
We're worried about something like user token accts getting deleted? okay, add some ix on token-program to compress/decompress accounts, make it permissionless (with conditions ABOVE rent-deletion conditions), let compressing user get some reward for compressing.
This sounds more complicated and dangerous than just doing compression. It introduces a race between deletion and compression. Will deletion also be incentivized? if not, when do we actually delete an account?
There was a problem hiding this comment.
This SIMD only describes the compression condition, not the mechanism itself.
Right, but that is already my issue. We're discussing when to compress, and haven't even landed on if we should compress.
In a future version, all compressed account data/metadata will be replaced with a fixed size vector commitment, so only the active accounts will contribute to the state size.
They only contribute to on-chain state size. But any hash tree we must store off-chain still grows with the number of compressed accts. I'd argue this is effectively under consensus because for whatever fixed-size root we store on chain to be usable this off-chain tree needs to be stored somewhere.
Am I understanding what you meant correctly here?
To be clear, compression in the context of this proposal is not primarily intended to provide an alternative to on-chain storage. It's an alternative to deleting delinquent account altogether.
I was under no such illusion. In #341 it describes that we store hash of data (plus some other meta), not a recoverable compression.
To me, it seems significantly simpler for apps if delinquent data is to be deleted.
If we go with 341 w/ hash stored instead of acct data, now apps need to handle that in their logic. They generally already handle accts not existing, because that can already happen!
Even compressing delinquent accounts is controversial, my guess is that outright deleting them would be a non-starter. I can see why deletion would eventually be necessary if we store all compressed account pubkeys, but using a fixed-size commitment solves that.
^as I argue above, I don't think it solves that. It pushes it off-chain and imo makes core more brittle.
This sounds more complicated and dangerous than just doing compression.
Complication is in put into the hands of users/devs not under consensus. The consensus requirements here can be extremely simple, which is what we want imo.
It introduces a race between deletion and compression. Will deletion also be incentivized? if not, when do we actually delete an account?
Deletion is incentivized:
If we add tracking fields to each account like described below, just track the owed rent, and if it's higher than current balance then ANYONE can just snipe the account for a reward.
There's 100% a race if the app doesn't incentive or enable compression soon enough. That's fine, their choice.
There was a problem hiding this comment.
They only contribute to on-chain state size. But any hash tree we must store off-chain still grows with the number of compressed accts. I'd argue this is effectively under consensus
I agree, the off-chain data will grow. Whether or not that's a problem depends on how that data is stored but I don't see why having it "under consensus" would be a problem (maybe I'm not understanding what you mean by that). My point was that v1 compressed accounts won't need to be stored on validators, so from the perspective of validator resource usage it's equivalent to deletion. This is not the case for v0 compression due to retention of pubkeys.
I was under no such illusion. In #341 it describes that we store hash of data (plus some other meta), not a recoverable compression.
My bad, I was unclear. I didn't mean that people are confusing compression here with traditional compression but with something like compressed NFTs. i.e. compression in this context isn't a feature for app developers but a protocol-level solution to state growth that avoids deletion. Of course, devs are welcome to use it but that isn't the motivation.
To me, it seems significantly simpler for apps if delinquent data is to be deleted
I'm not an app dev but compression seems strictly better than deletion because you can always just treat compressed accounts as deleted, but now you have the option to recover it as well. Only difference is compressed accounts still exist in the address space in the sense that new accounts can't share the same address. Is that the difficulty you're referring to? I can definitely see how needing to check if an address is already reserved for a compressed account would be annoying if your intention was to just recreate it.
Complication is in put into the hands of users/devs not under consensus. The consensus requirements here can be extremely simple, which is what we want imo
This is the main point I'm not grasping. In both cases the protocol needs to be able to track rent info, check delinquency, and preserve compressed account pubkey/data hash. The only difference AFAICT is that compression is opt-in, but how does that simplify consensus requirements? or are you saying that in your proposal apps would store the pubkey/hash for compressed accounts in some other account they manage instead of validators managing it?
There's 100% a race if the app doesn't incentive or enable compression soon enough. That's fine, their choice.
idk this seems more complicated than only doing compression if that's what the app wants anyway. If incentives are working perfectly, the compression reward is always higher so the account always gets compressed. If they're not working perfectly, the app dev has no other recourse to prevent the account from being deleted.
I'd support allowing the devs an option between marking an account compress_on_delinquency vs delete_on_delinquency then have only one incentivized eviction process that decides what to do based on that value. We could even require more lockup from compressible accounts so it isn't used frivolously.
Yea. Thats the LSR idea. Fee payer pays the cost the bump. So normal access just keeps extending the life of the account. |
What's the current acceptable state size? Make it really expensive to get above 2x of that. Make it super cheap to be below that. Change that value as hardware or implementation improves. Target can be updated in a patch release. There are always humans in the loop to be a controller. |
Where does it do this? Any ix touching a compressed account would abort unless it's the decompress ix. No deployed program could ever see a compressed account value. |
I liked that idea for that it allows autonomous programs to exist. However i realized that in practice autonomous programs aren't really a thing and there's always some operator. The operator earn fees from the program so it just makes sense that they also cover the rent costs via explicit payments. I now think active monitoring and explicit rent payments from app teams is probably more realistic. |
|
Fee payer needs to pay to keep memory hot. Since it's the fee payer that needs that state accessible. |
MaxResnick
force-pushed
the
simd-dynamic-state-rent-clean
branch
2 times, most recently
from
2b94490 to
fa0079b
Compare
Major changes: - Replace PID controller with simpler integral controller based on accumulated excess state - Switch from epoch-based to slot-based rent tracking using last_write_slot - Repurpose existing rent_epoch field to avoid account size increase - Clarify rent calculation for partial epochs and account size changes - Remove compression rewards to avoid SOL creation - Add detailed slot-based rent calculation helper function - Update syscall signatures and terminology throughout 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
MaxResnick
force-pushed
the
simd-dynamic-state-rent-clean
branch
from
fa0079b to
7002c22
Compare
- Change rent rate to be a direct function of accumulator - Remove addition of current_rent_rate from formula - new_rate = integral_gain * accumulator / 1e9 (instead of current_rate + adjustment) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
If this is the way to go then to clarify what i meant was a write-lock without a fee payment being sufficient. There would still be the concept of a rent excempt reserve but there's now an additional requirement that is frequent access. It's not perfect maybe but the least disruptive i can imagine. |
Assuming current rent price is for two years, bumping a 10mb account compression slot by 1 slot would be 0.00001 sol for the fee payer. If a dev really needs to keep an infrequently used account in memory they can run a cron to send lamports into that account once a week. But in general they can just do nothing. Let the users keep it alive. |
|
I think developers and users should have the choice between deletion and compression. There are tokens with limited lifetime e.g. SIMD vote tokens. Deletion seems like a feature here. This would also allow to make the initial payment on account creation even cheaper. There are also scenarios when dealing with PDAs where account (re-)creation which is part of the program is simpler than decompression which requires logic that isn't part of the program e.g. to decompress a liquidity pool and related accounts i need significant knowledge about the program storage logic. In contrast if the accounts were deleted it's just like creating a new liquidity pool via the existing interface. |
They do! Any user that wants to store the state can run a full node and store all the state, or just their own state. Any user that wants to have all the other full nodes store the state the user cares about can pay for it by depositing more lamports into that account. Compression doesn't delete the accounts. It just moves the data storage from the state that has guaranteed real time access for any transaction. The off chain storage is "cold", the on chain storage is "hot", there is a decompress/compress interface to move things in/out on demand. |
|
What is the current state here? From a user experience perspective i would currently prefer the following direction:
|
|
Some UX/devex suggestions:
|
|
After thinking about this a bit, I wonder if we should instead try and incentivise people to close accounts? Instead of incentivising compression, which only partially solves the state explosion problem, we should maybe design an incentive structure that still makes it cheap to create new accounts but discourages leaving stale accounts around forever without closing them. |
The key difference is that compression "mining" incentives can be open to anyone, not just the account owner, so they're not susceptible to the inactive owner problem. Also, how do we incentivize account closure beyond simply returning the bond balance to the owner? This is what the current system does, and it appears to be insufficient. |
12 participants
Summary
This PR adds a new SIMD proposal for implementing dynamic state rent in Solana based on account usage patterns.