paritydb breaks on upgrade to v0.9.18

[tugytur]

Both our paritydb-experimental sync nodes for Kusama and Polkadot failed on the startup with the release v0.9.18.

RUST_BACKTRACE=full on both gave the same output.

WARNING: "paritydb-experimental" database setting is deprecated and will be removed in future releases. Please update your setup to use the new value: "paritydb".
2022-03-21 12:06:51 Parity Polkadot
2022-03-21 12:06:51 ✌️  version 0.9.18-99cd17ddb2-x86_64-linux-gnu
2022-03-21 12:06:51 ❤️  by Parity Technologies <admin@parity.io>, 2017-2022
2022-03-21 12:06:51 📋 Chain specification: Polkadot
2022-03-21 12:06:51 🏷  Node name: dot1paritydb
2022-03-21 12:06:51 👤 Role: FULL
2022-03-21 12:06:51 💾 Database: ParityDb at /var/blockchain/chains/polkadot/paritydb/full
2022-03-21 12:06:51 ⛓  Native runtime: polkadot-9180 (parity-polkadot-0.tx12.au0)
2022-03-21 12:06:54 Essential task `txpool-background` failed. Shutting down service.
2022-03-21 12:06:54 Essential task `transaction-pool-task-0` failed. Shutting down service.
2022-03-21 12:06:54 Essential task `transaction-pool-task-1` failed. Shutting down service.
Error:
   0: Error at calling runtime api: Runtime code error: `:code` hash not found
   1: Runtime code error: `:code` hash not found
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                 ⋮ 1 frame hidden ⋮                               
   2: polkadot::main::he63b6329d0694b27
      at <unknown source file>:<unknown line>
   3: std::sys_common::backtrace::__rust_begin_short_backtrace::h0b52984c10824fba
      at <unknown source file>:<unknown line>
   4: main<unknown>
      at <unknown source file>:<unknown line>
   5: __libc_start_main<unknown>
      at <unknown source file>:<unknown line>
   6: _start<unknown>
      at <unknown source file>:<unknown line>
Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.

both on Kusama and Polkadot the DBs were synced from scratch with the release 0.9.17-rc4. Same for the rocksdb servers, but those had no issues updating. Sync servers have the following relevant arguments --unsafe-pruning \ --pruning 1000 \

The bricked DBs are available here:

SHA256 URL:

• https://parasnaps.io/polkadot_paritydb-experimental_bricked.tar.sha256
• https://parasnaps.io/kusama_paritydb-experimental_bricked.tar.sha256

S3 URI:

• s3://euw1.parasnaps.io/polkadot_paritydb-experimental_bricked.tar
• s3://euw1.parasnaps.io/kusama_paritydb-experimental_bricked.tar

HTTP URL:

• http://euw1.parasnaps.io/polkadot_paritydb-experimental_bricked.tar
• http://euw1.parasnaps.io/kusama_paritydb-experimental_bricked.tar

The working DB that breaks during upgrade are the ones currently available at https://parasnaps.io . These are the exact DBs that broke during upgrade from ~12hours ago.

[bkchr]

Cc @arkpar @cheme

[cheme]

Yes it is software bug, the database version in metadata is being upgraded while it should not have, I am looking at it.

[cheme]

Re-synching is safe. Also manually setting the db_version in metadata to 5 seems to fix the issue. (the fact that it upgrade to 6 is probably from a parsing error and metadata getting rewritten, setting manually the rewritten metadata to 5 on the rebuild metadatat work then).

[cheme]

Ok, that's it, https://github.com/paritytech/substrate/blob/1bd5d786fda99d5a0c27c339226789ac047ea4ae/client/db/src/utils.rs#L279 should not be allowed.
Here what did happen is that the columns definition of transaction storage change, and there was also a new column, so validation did fail resulting in rewrite of metadata with 'db_version=6', which breaks the state access (trying to use salt on key as done with v6).

@tugytur , sorry for the bug.
You don't need to resynch the snaps, editing "metadata" file and setting db_version to 5 should fix this.
Resynching is not useless though (db will be version 6), but really not necessary (a migration could be written at some point to update the db).

[tugytur]

@cheme thanks!

Can confirm that changing db_version to 5 fixes it.

I like to sync from scratch every time there are any changes to the db.
We will provide version 6 snapshots as soon as it's synced.

[PolkachuIntern]

I confirm that changing the db version from 6 to 5 on metadata file has fixed the issue

[tugytur]

@cheme
I've upgraded one of our Kusama Validators and implemented the db_version fix.
CPU usage was ~30-50% higher than usual. Heartbeat was sent every epoch.

But we were assigned as a Para Validator twice and earned 0 points.

Inbound and outbound request failures (timeout) increased from ~0 to couple thousands request per seconds.
libp2p_out_events_notifications_sizes were constantly growing as well.

rpc call babe_epochAuthorship was hanging as well. Canceled the call after 5mintues of waiting.

In the meantime I finished synching db_version 6.
I've migrated the affected Validator to new db_version snapshot and everything is back to normal. I'll make the snapshot tomorrow publicly available if the Validator runs stable through the night.

[cheme]

Thanks for the feedback.
I got an hypothesis the high usage is not from using paritydb version 5, but from a switch of the polkadot parachain db between 0.9.17 and 0.9.18 (which IIUC requires some additional synching).
I know we ran this on validator node before, but it may have been synched from scratch.
CC @ordian , could the describe issue be related?

[tugytur]

@cheme last update for tonight.
There seems to be a issue with db_version 6 as well. I'm not sure how much this would affect the Validator but I'm switching back to rocksdb for the time being.

After roughly 20 minutes of running mpsc_finality_notification_stream and mpsc_import_notification_stream suddenly keeps growing (grows to over 500 in under 20 mintues) and during that time the Validator doesn't respond to any rpc calls but it keeps syncing. When stopping the service it happens via SIGKILL after the service runs into a timeout. I did one restart and waited to be sure that this isn't a random bug.

Edit: Maybe I jumped to a conclusion here. After the second restart this did not happen again. I'll not switch to rocksdb for now and hope we have more data by the morning.

Edit2: It happened again. I've switched to rocksdb. I've kept the paritydb in case you would like to have a look at it.

[cheme]

@tugytur , if you have log available, it can be interesting (even if I have little hope to see something, I am currently running node without much logged).
I will push the version fix and then ask to rerun the tests we did on validator previously, will also prepare a branch which keep using rocksdb for as parachain db: could be a simple way so spot if issue is related to the switch of db at this level.