SIMD-0341: v0 Account Compression

[igor56D]

No description provided.

[jacobcreech]

What authorization is required to compress/decompress an account?

[igor56D]

That depends on the "compression condition", which is intentionally set to false in this SIMD to focus discussion on the mechanism. SIMD-0344 extends the condition to include rent delinquency but other SIMDs can add account authority/owner checks to the condition as well. In any case, it's out of scope for this SIMD.

[jacobcreech]

Is it expected that this information would be stored offchain for recovery?

[igor56D]

yes, it will be off-chain but who stores the data and how isn't specified here. RPC providers and archival nodes are the more obvious options but wallets can cache account data for users, apps can subscribe to account update events and store the data, etc. It's pretty open ended and the best choice will likely depend on the account and its usage.

[jacobcreech]

If an app or wallet developer wanted to view the data of an account, would this mean that they would have to simulate a transaction with a decompress call to compression system program?

[igor56D]

If the account is compressed, the data isn't available on-chain. If that account needs to be used then it must be decompressed, which requires providing the data. The off-chain source of that data can vary, like I mentioned in a previous comment.

[jacobcreech]

What is the expected storage savings given an account being compressed with X bytes?

[igor56D]

Compressed accounts will take up 64 bytes of space (pubkey and data hash), so the space savings will be X - 64. Every active account must at the very least store the pubkey + owner pubkey + lamports + etc so space savings will always be positive. I believe token accounts are the most common and they're about 165 bytes, for reference.

[jacobcreech]

RPCs and good errors can provide the relevant info

This is incredibly important to get right. It we don't outline exactly all the edgecases and how this works for developers, it'll be a nightmare for anyone to integrate with. We should consider putting more detail either in this SIMD on how these edge cases and errors will be displayed, or have a separate document outlining how we can properly setup the right infrastructure to handle the expected behavior with compressed state.

[igor56D]

I added more to the SIMD regarding errors and RPC updates that will hopefully mitigate some of the concern. A separate doc on recommended infra setup + workflow changes is a good idea. I'll reach out to some of the RPC providers to see if we can put something like that together.

[wjthieme]

Would it be low-hanging fruit to allow programs to compress/decompress their own state? Either by allowing to syscall directly or to cpi into compression_system_program with the account_to_compress as signer?

[wjthieme]

Will an account appear as uninitialized if it is compressed? It would be useful to have programs be able to distinguish between uninitialized vs compressed. Or will the program a compressed account be assigned to be compressed_system_program vs normal system_program?

[brooksprumo]

Worthwhile to include lowering rent cost in this list too? It's a main motivator in the above paragraph. (Will be a separate SIMD, of course.)

[aeyakovenko]

My preference is that the data is stored in a secondary db that can be fetched by rpcs just like a snapshot. But it’s not in the hot path for a validator snapshot.

Rpc provides and indexing protocols can plug it into their systems

[igor56D]

Would it be low-hanging fruit to allow programs to compress/decompress their own state? Either by allowing to syscall directly or to cpi into compression_system_program with the account_to_compress as signer?

I wouldn't expect this to be too much of a lift -- would just require modifying the compression condition to allow owners/authorities to compress their accounts. The compression condition is intentionally made to be false in this SIMD (meaning no accounts are compressible) so the scope of discussion can be narrowed to just the compression mechanism itself. Modifications to the condition will be in separate SIMDs.

Will an account appear as uninitialized if it is compressed? It would be useful to have programs be able to distinguish between uninitialized vs compressed. Or will the program a compressed account be assigned to be compressed_system_program vs normal system_program?

The intention is that "compressed account : decompression" is functionally very similar to "uninitialized account : account creation". So, unless a compressed account is being decompressed, it should appear the same as an uninitialized account to users and programs. Main rationale is that this is just much simpler and less error prone implementation-wise.

I'm curious why being able to distinguish them within the runtime would be useful. If there's a good enough reason I'll think about how best to expose the distinction in the runtime.

[zfedoran]

sha_256(account_pubkey)[0..10]

Where did the design decision for a 10byte slice come from? Eg: why not 16 bytes?

Also, not sure it’s worth the trouble, using the full 32 byte pubkey is easier and reduces the trouble of having to check if an account can be created

[igor56D]

sha_256(account_pubkey)[0..10]

Where did the design decision for a 10byte slice come from? Eg: why not 16 bytes?

Also, not sure it’s worth the trouble, using the full 32 byte hash is easier and reduces the trouble of having to check if an account can be created

The intention was to use the minimum number of bytes to reduce the snapshot size as much as possible. Given what the pubkey_hash is used for, cryptographically strong collision resistance isn't necessary. That said, I discussed this with FD/Agave devs and the consensus is that the complexity isn't worth the extra space savings so I'll remove pubkey_hash and use the full pubkey instead.

[zfedoran]

I’m curious, the document seems to imply that only accounts that meet some TBD criteria can be compressed. Why not allow any account to be compressed as long as the signature or signing seeds are properly provided.

Some protocols want to be able to compress their program accounts, on demand, to recover account fees (as is done here).

[igor56D]

I’m curious, the document seems to imply that only accounts that meet some TBD criteria can be compressed. Why not allow any account to be compressed as long as the signature or signing seeds are properly provided.

Some protocols want to be able to compress their program accounts, on demand, to recover account fees (as is done here).

I don't see any reason we can't eventually support something like this. It'd be a fairly straightforward extension to the compression condition, but I intentionally left that and similar things out of this SIMD to minimize controversy and get alignment on the basics first.

[zfedoran]

I’m curious, the document seems to imply that only accounts that meet some TBD criteria can be compressed. Why not allow any account to be compressed as long as the signature or signing seeds are properly provided.
Some protocols want to be able to compress their program accounts, on demand, to recover account fees (as is done here).

I don't see any reason we can't eventually support something like this. It'd be a fairly straightforward extension to the compression condition, but I intentionally left that and similar things out of this SIMD to minimize controversy and get alignment on the basics first.

Awesome, fully understand and appreciate that. Just wanted to make sure it was seen as a potentially useful thing to do instead of hard-coding a “last_seen_epoch > 10” or something like that.

[wjthieme]

I'm curious why being able to distinguish them within the runtime would be useful. If there's a good enough reason I'll think about how best to expose the distinction in the runtime.

Imo there is functionally a difference between uninitialized and compressed. I think there is an important distinction:

• From client-side code it is important to be able to know the difference. How can I otherwise know to init or decompress an account.
• From program-runtime being able to distinguish might be more of a nice-to-have / future-proofing. I can imagine a scenario where you might want to decompress/init conditionally through a cpi or something. Could also just be over-optimization from my end tho

[igor56D]

Updates:

• removed CAM and pubkey_hash. Compressed accounts will instead be tracked in the regular accounts database.
• elaborated on updates to RPCs to give devs more clarity on how account management can be implemented to handle potentially compressed accounts.
• replaced SHA256 with BLAKE3 for data_hash calculation
• removed unnecessary implementation details

cc @brooksprumo @ripatel-fd @jacobcreech

[ripatel-fd]

Could you specify whether this check happens at program deploy-time or run-time?

[ripatel-fd]

sol_compress_account needs CU pricing

[ripatel-fd]

sol_decompress_account needs CU pricing

[ripatel-fd]

Syscalls unfortunately can only take 5 arguments (iirc).
Maybe we can use one of the account structs in the program SDK, such as the ones from the CPI syscalls.

[ripatel-fd]

I'm pretty sure that rent_epoch is not part of the lthash on any clusters anymore. I don't know which SIMD removed rent epoch from the hash though (cc @brooksprumo)

It's fine to remove from this syscall

[brooksprumo]

Yep, rent epoch was removed with the Accounts Lt Hash in SIMD-215. Let's remove it here as well.

[ripatel-fd]

I wonder if this syscall interface should be changed to improve user UX.
Correct me if I'm wrong but the implied flow here is as follows (for large accounts):

• User creates a dummy account that will hold a copy of the original data
• User uploads fragments of original account data to target account using multiple transactions
• User calls invokes the compression program, which invokes this syscall. Under the hood, the runtime allocates the target account and copies over data
• User cleans up the dummy account

I feel like we could combine some of these steps and skip the syscall copy

[igor56D]

Semantically, the compression program itself can delete the dummy account and transfer rent/balance/metadata to the new account. In that case, the copy vs move becomes an implementation detail. Is that sort of what you have in mind?

[igor56D]

rather, we can make it so the decompression syscall does the delete. If the program does it then move vs copy isn't an implementation detail.

[ripatel-fd]

the copy vs move becomes an implementation detail. Is that sort of what you have in mind?

@igor56D I feel like it would be very tricky to implement an account move in a syscall handler. Since all the sorrounding code (transaction executor, VM) assumes that accounts are pinned in memory.

EDIT: Would probably be easier if the move happens in the post-transaction cleanup after the program has finished executing.

[igor56D]

good call, I'm being pretty hand-wavy because I'm not as familiar with these low-level details. Conceptually, it seems feasible to take tmp_pubkey -> data, "alias" original_pubkey -> tmp_pubkey -> data at decompression time, then resolve/collapse that whenever it makes the most sense.

I'll need to do some digging into the execution env to formulate something more concrete.

[ripatel-fd]

Please note that the 'lattice hash' is 2048 bytes long, so the spec should mention how to go from 2048 bytes to 32 bytes hash size. Lattice hash is simply BLAKE3 with extended output (XOF). We can simply take the first 32 bytes here since BLAKE3 has the convenient property that, for a constant input, the low bytes of the hash output don't change as the XOF output length parameter is increased.

EDIT: blake3_2048(lthash_preimage)[:32] == blake3_32(lthash_preimage)

[ripatel-fd]

(What we should NOT do is something like blake3_32(blake3_2048(lthash_preimage)))

[igor56D]

I was using lthash.out as described in SIMD-0215:

out(a): blake3.fini( blake3.append( blake3.init(), a ) ), i.e. the 32-byte blake3 of the 2048-byte data.

^^ I'm confused by this but I assumed it was what you meant by 32-byte mode. I'll change the doc to explicitly use blake3_32 to avoid confusion.

[ripatel-fd]

Haha yeah we don't have definitions of these blake3 pseudocode functions

[brooksprumo]

(What we should NOT do is something like blake3_32(blake3_2048(lthash_preimage)))

Oops 🙃 I do blake3_32(blake3_2048(..)) for the checksum currently:
https://github.com/anza-xyz/agave/blob/7df820d832d2ae1152e7cdc13e495797600f4b3d/lattice-hash/src/lt_hash.rs#L52-L56

But, that's only used for logging purposes and can be changed. We should do the right thing for compression stuff.

[topointon-jump]

"Compression" is overloaded, especially in this space (see: compressed nfts) and as mentioned here, account compression isn't analogous to traditional compression. Using a completely new term might be clearer, maybe "Compacting"? Just a suggestion, feel free to ignore.

[brooksprumo]

I'd also love to not call it compression. I've been in favor of freeze/frozen/thaw/thawed personally.

[igor56D]

I also don't like the name but figured it had enough history that renaming would introduce more confusion.

[Sobeston]

Some alternative naming suggestions: "stubbed" accounts, "digest" accounts, "proxy" accounts

[brianlong]

Anything but "compression"! Frozen/thawed sounds better.

[topointon-jump]

rent_epoch is not an input to the hash function.

[topointon-jump]

What is the reason for doing compression/uncompression in syscalls, instead of adding two new system program instructions for doing this? Is there an advantage to these operations being syscalls rather than native program instructions?

[topointon-jump]

Similarly, why a new program? Compression/un-compression feels like it fits within the system programs purview.

[topointon-jump]

What if the original account data is larger than the max transaction size? There probably needs to be a mechanism for buffering the account data somewhere before uncompressing it.

[brooksprumo]

Yeah, we should do something akin to program deployment.

[ultd]

Agreed with the buffering but thinking out loud, would it actually be beneficial to limit the size of compressed accounts to the max transaction size? If an account is bigger than the tx size limit, maybe it should live uncompressed onchain to prevent tpu/tx bloat?

[igor56D]

this shouldn't materially impact tx bloat. If someone wants a large account with specific data on-chain, they need to upload the data in parts across multiple transactions. The same applies to a large account getting decompressed. When an account is decompressed, it persists on-chain like any other active account unless it gets compressed again so subsequent accesses don't need to re-upload.

[topointon-jump]

What happens if a user attempts to access a compressed account in the vm? Is this an execution error? I think it's simpler if it's required that all accounts that an instruction accesses (including through nested cpi calls) have already been decompressed before the vm invocation.

[igor56D]

During VM execution, attempts to access a compressed account are equivalent to attempts to access a non-existing account, and returns the same error

[roryharr]

What will happen to the lamports when the account is compressed from a global tracking perspective?
Currently (I think) if you sum up all the lamports in all the accounts it should be equal to the bank capitalization. With this change that will no longer work without decompressing all the accounts.

Maybe the get transferred to some kind of system account?

[roryharr]

I'm not sure what this paragraph is meant to address, or if I'm missing something. This is an implementation detail for the backing database that is irrelevant to this SIMD.

A validator could never compress the account in the database (this may be useful for an RPC provider) as long as they calculate the lt_hash correctly.

[igor56D]

I agree, this is bordering on an implementation detail. I'll remove it to avoid confusion.

[brooksprumo]

Yep, rent epoch was removed with the Accounts Lt Hash in SIMD-215. Let's remove it here as well.

[brooksprumo]

Seems like implementation details? We may or may not do exactly this. I agree we need the account's pubkey and data_hash, but I envision us doing it as the CompressedAccount struct shows below.

[brooksprumo]

(What we should NOT do is something like blake3_32(blake3_2048(lthash_preimage)))

Oops 🙃 I do blake3_32(blake3_2048(..)) for the checksum currently:
https://github.com/anza-xyz/agave/blob/7df820d832d2ae1152e7cdc13e495797600f4b3d/lattice-hash/src/lt_hash.rs#L52-L56

But, that's only used for logging purposes and can be changed. We should do the right thing for compression stuff.

[brooksprumo]

I don't think we need (nor want) to change the snapshot format.

We do need to modify the account storage file format though.

[Sobeston]

Could we end up in a situations where:
a) the original data of a compressed account is entirely lost
b) the data can be found by some validators, but not others (e.g. some validators are hooked up to a paid 3rd party for storage and some aren't)

What would you expect/want to happen in these scenarios? Can redundancy be truly guaranteed, and can it be fairly distributed to all validators (and quickly enough for a slot, globally)?

Is the expectation that a solana project would ensure that it does a good job at storing its data off-chain?

Let's say there's some trading happening which requires the decompression of an account. If that project turned their archive service off, and validators can't access that data, could that risk assets becoming un-tradeable? This feels like we could risk things becoming centralised. Let me know if I've misunderstood something.

[igor56D]

There is a theoretical possibility that compressed account data is completely lost, though it's extremely low. Even if every indexer, RPC provider, wallet provider, etc somehow loses the data (or refuses to release it), you can always fall back to replaying the ledger history from some snapshot before the account was compressed.

The protocol provides the guarantee that you can always recover a compressed account if you can provide the original data. Users/apps must decide for themselves how strong the data availability guarantee must be for their use case, and plan accordingly. The best guarantee comes from running your own full node that's synced with the network. Others may prefer to subscribe to a stream of compression events from an RPC provider and store the compressed data themselves. Less sophisticated users are likely to rely on wallet backups or indexers to keep the data available.

In any case, the complexity is moved out of the protocol and into off-chain infra.

If that project turned their archive service off, and validators can't access that data, could that risk assets becoming un-tradeable?

In reality, most full node indexers will have this data stored, though this isn't backed by clearcut incentives for now. Worst case scenario is needing to replay the ledger. There's also the option of storing the data in some distributed storage protocol like filecoin or IPFS. The availability concern is purely theoretical until the compressed state size grows dramatically, in which case we'll need to consider a more complete data availability system.

[brianlong]

There is a theoretical possibility that compressed account data is completely lost, though it's extremely low.

Guarantees of future availability are essential. This doesn't go far enough.

fall back to replaying the ledger history from some snapshot before the account was compressed

This is not as easy as it sounds (according to people who have done it and told me it was tough).

full node indexers will have this data stored

This is a weak economic assumption. There's not much incentive to store terabytes of cold data in a separate data store for the outside chance that someone will pay $0.000025 to read it. Keep in mind the accounts are primarily cold because people aren't using/reading them.

However, some of the discussion further above suggests a solution that comes closer to a guarantee of future availability with a data store that will always exist and does not require replay. The process for thawing a frozen account, which is similar to deploying a program using a buffer account, might also be used to freeze the account: write the account state to the ledger in one or more TXs before freezing it.

1. The theoretical possibility of losing the ledger history is almost nil as long as Solana exists.
2. The ledger history is replicated across many independent players who have a financial incentive to keep it. Also, the raw data is publicly available at no cost today, so anyone can load historical data if they prefer to roll their own. The economic incentives are aligned.
3. The process to thaw a frozen account will still require an off-chain actor to read the account data from the ledger and then use that as an input to the thawing process. However, you won't need to replay TXs; just read the ledger history.

Footnotes:

• The validators are disconnected from the full ledger history, so the entire process can't be done in-protocol. The external actor will be required to read from the ledger history & execute the thawing process.
• Using the ledger history might also give some extra cryptographic peace of mind since the archived account state can be verified within the block by using the hash. In comparison, data stored in an off-chain SQL database might become corrupted or incomplete.

A parting question: Since the validators are not connected to the ledger history, does that mean the ledger history is technically an "off-chain database"? (I'll see myself out) ;-)

[aeyakovenko]

So first of all, a solana "user" is assumed to have a full node, or is paying someone else to run a full node directly or indirectly. There is no way for the protocol to make any guarantees about users data without this assumption. The best any protocol can do is that if the users one full node stays consistent then the user is not at risk of any loss.

If a user has access to a full node they can always replicate just part of the compressed state that user cares about.

We need tools to automate the compressed state recovery from the ledger.

If you want weak assumption that there exists at least one full node in the network that will keep the user state, that user just needs to pay enough to keep it from being compressed. But this is also a weak economic incentive.

[brianlong]

We need tools to automate the compressed state recovery from the ledger.

The elegance of storing the frozen account state in the ledger is that the DB storage & retrieval tools already exist. getSignaturesForAddress() + getTransaction() or an Old Faithful gRPC stream will return the account state without a specialized SQL index or API. The new tooling will be related to the freeze/thaw processes.

[brianlong]

None that can offer a guarantee of completeness or future availability. For example, streaming accounts through Geyser to a database won't work because there's no guarantee that messages will be received or that the database will be complete. Losing a single message could have catastrophic consequences for the account holder.

I think all of your points about the tricky implementation are valid. Nonetheless, the guarantee is more important than the implementation challenge. Saving data in the ledger checks the boxes for storage & retrieval guarantees, universal availability, & economic alignment for the infrastructure providers.

[igor56D]

I can't figure out a way to make this work well enough. Even if we introduce the complexity I mentioned above, compression requests will still be prohibitively expensive. It's very important that compression is cheap -- the network needs to be able to promptly and sufficiently respond to growing state/rent prices by evicting delinquent state, which may require a lot of compression requests. Requiring all the compressed state to flow through the blocks will be a challenge for throughput.

You mentioned that replaying the ledger is difficult now. Would addressing that problem make more sense? Ledger history can tell you the slot the account was compressed at, so you'll know how far to replay. Not sure how much we can rely on historical snapshot retention, but reliable replay + snapshot fetching should make account reconstruction relatively doable.

[igor56D]

if that doesn't seem sufficient, I'd prefer @aeyakovenko's previous idea of maintaining a separate compressed state snapshot that's replicated on all validators over making compression really difficult.

[brianlong]

maintaining a separate compressed state snapshot that's replicated on all validators

This also works, and was my original suggestion at some earlier meetups. Hot & Cold account sets with separate snapshots & hashes. NVMe storage is affordable, and drive sizes keep getting bigger. JWash mentioned a concern about validator start times when the validator needs to validate two account DBs. That could be a more solvable problem and provide the guarantees.

[brianlong]

P.S. Validator startup times will be slower, but all good validators run hot spares, so there's minimal downtime during failover. Slow startup times are manageable.

[ultd]

Could we add a transaction-scoped (ephemeral) decompression syscall so a tx can read (and optionally commit a write to) a CompressedAccount without rehydrating it in AccountsDB? This would cut AccountsDB churn and still let programs operate over the account state.

We would intro a a cpl other new syscalls:

sol_transient_decompress_account(
    compressed_acct_pubkey: Pubkey, // aka index into accounts array
    current_state: &[u8]            // full account struct bytes 
);

sol_transient_commit_compressed_account(
    compressed_acct_pubkey: Pubkey, 
    new_state: &[u8],         
);

The runtime just verifies hash(current_state) == data_hash from the CompressedAccount. No modification/rehydration of AccountsDB. If succeeds, converts CompressedAccount to an ephemeral, read-only view of the account that is addressable during the tx (maybe stored directly in TransactionContext for other ix to use). The account must be listed in the tx’s account metas per usual, locks still ensure consistent access ordering if multiple txs target it. At the tx's end, ephemeral memory is discarded.

Example pseudo-code of how used in a tx:

compressed_account_state = [...]
compressed_account_pubkey = compressedAccount1A3b5D7f9GhKm2pQ4rStVxYz6uL8

TX(a):
    accounts: [owner, compressed_account_pubkey, new_owner, SYSTEM_PROGRAM]
    signers: [owner]
    instructions:
         0: sol_transient_decompress_account(compressed_account_pubkey, compressed_account_state)
         1: transfer_asset_ownership(compressed_account_pubkey, new_owner); sol_transient_commit_compressed_account(compressed_account_pubkey, new_state)

The other sol_transient_commit_compressed_account syscall would allow one to optionally commit new account state to a CompressedAccount. This would work with sol_transient_decompress_account syscall which would need to be called first to acquire the compressed account. On commit, the sol_transient_commit_compressed_account syscall would update the CompressAccount entry’s data_hash field with new computed hash without ever storing full bytes in AccountsDB.

This covers the common pattern where an authority needs to read or locally authorize effects (NFTs/RWAs/hotspots, etc.) but doesn’t need the account to remain "active" after the tx. It avoids touching AccountsDB entirely in the read-only case, and avoids the rehydrate/recompress loop in the write-through case. It also keeps snapshots small since the compressed entry never expands.

This wouldn't work well with Accounts meant to be "global", shared state (DEX orderbooks, AMMs) that needs a consistent, persistent view across txs. Those should just remain in active set. Also, this probably only works well for small account sizes. Probably should limit to Accounts that are max N bytes as you don't want each tx submitting excess ix bytes causing TPU bloat.

EDIT:

Maybe we don't need sol_transient_commit_compressed_account as the runtime should probably commit the new hash to a CompressedAccount's data_hash field upon tx success implicitly. In fact, it's probably necessary.

[igor56D]

@ultd IIUC this isn't incompatible with the existing proposal -- no new syscalls would need to be introduced. This proposal intentionally leaves the compression condition itself stubbed out to focus discussion on the compression mechanism rather than all the potential ways to use it (ephemeral compressed accounts, rent eviction, etc).

For example, if there was a subsequent SIMD that expanded that compression condition to include something like ... || caller == account.owner it would give the owner of an account the authority to compress it. With this they can use the existing syscalls to compress/decompress at will.