Fix plutus list serialization

[kodemill]

#227

• serialize PlutusList as indefinite length array instead of fixed length array, with the exception of 0-length arrays
• serialize the PlutusData (PlutusList) in the TransactionWitnessSet as fixed length arrays

[SebastienGllmt]

This is the same root cause as #217 where the root cause is actually a bug the cardano-ledger. Doesn't mean we can't merge PRs to unblock people on this in the meantime though

[Quantumplation]

@SebastienGllmt how does changing the wire format for cardano at this point play out? Not sure what that process looks like or how it would effect downstream tooling

[alessandrokonrad]

This is the same root cause as #217 where the root cause is actually a bug the cardano-ledger. Doesn't mean we can't merge PRs to unblock people on this in the meantime though

If the plutus data serialization is correct in the serialization-lib and just wrong in the cli, I would leave it as it is. #217 was necessary to change because the LanguageViews encoding is created on-chain. The plutus data on the other hand are just off-chain and I personally rely on the serialization-lib already with my dApp.

[rooooooooob]

If the plutus data serialization is correct in the serialization-lib and just wrong in the cli, I would leave it as it is.

Do the specs for encoding Plutus data mention that it should be in canonical CBOR form (and thus definite), or is it ambiguous?

[JaredCorduan]

Our hope in the ledger is to fix the serialization bug in the next hard fork IntersectMBO/cardano-ledger#2512

[BlakeBrown]

Just a casual observer here, how do we know if our datum hash is correct if it differs between the cli and the serialization lib? Is there no way to test this until the issue gets resolved in the next hard fork?

[rooooooooob]

Just a casual observer here, how do we know if our datum hash is correct if it differs between the cli and the serialization lib? Is there no way to test this until the issue gets resolved in the next hard fork?

Unfortunately that's why hashing is so useful - small changes in the input cause huge changes in the output so you can't predict it. This makes debugging not very nice about why the hash is wrong. You just know that something is different/wrong. If you can export the data that the cardano-node hashes that can help a lot since we can manually examine that and compare it with the cardano-serialization-lib's structure. Any difference is probably something small like the CBOR encoding differences (e.g. definite vs indefinite list length encoding) here or in #217 for example. Or even just having any similar sample data from the node vs ours can help since you can look out for small differences in the CBOR encoding like that. That's how I figured it out in #217 by looking at sample data from the cardano-node and noticing the indefinite CBOR list/map length encoding instead of definite.

[vsubhuman]

Our hope in the ledger is to fix the serialization bug in the next hard fork input-output-hk/cardano-ledger#2512

Am I getting it right that if we merge this change right now for the hardfork we will have to change it back?

[JaredCorduan]

Our hope in the ledger is to fix the serialization bug in the next hard fork input-output-hk/cardano-ledger#2512

Am I getting it right that if we merge this change right now for the hardfork we will have to change it back?

Plutus V1 is domed to have to always use this sad script integrity hash, but our hope is to have Plutus V2 use what we had intended.

[vsubhuman]

@kodemill , can you plz pull latest master and also run the npm run rust:test and resolve any test failures?

I am getting this failure when checking your branch locally, and it does not happen on master:

failures:

---- utils::tests::correct_script_data_hash stdout ----
language_views = a141005901d59f1a000302590001011a00060bc719026d00011a000249f01903e800011a000249f018201a0025cea81971f70419744d186419744d186419744d186419744d186419744d186419744d18641864186419744d18641a000249f018201a000249f018201a000249f018201a000249f01903e800011a000249f018201a000249f01903e800081a000242201a00067e2318760001011a000249f01903e800081a000249f01a0001b79818f7011a000249f0192710011a0002155e19052e011903e81a000249f01903e8011a000249f018201a000249f018201a000249f0182001011a000249f0011a000249f0041a000194af18f8011a000194af18f8011a0002377c190556011a0002bdea1901f1011a000249f018201a000249f018201a000249f018201a000249f018201a000249f018201a000249f018201a000242201a00067e23187600010119f04c192bd200011a000249f018201a000242201a00067e2318760001011a000242201a00067e2318760001011a0025cea81971f704001a000141bb041a000249f019138800011a000249f018201a000302590001011a000249f018201a000249f018201a000249f018201a000249f018201a000249f018201a000249f018201a000249f018201a00330da70101ff
thread 'utils::tests::correct_script_data_hash' panicked at 'assertion failed: `(left == right)`
  left: `"4415e6667e6d6bbd992af5092d48e3c2ba9825200d0234d2470068f7f0f178b3"`,
 right: `"57240d358f8ab6128c4a66340271e4fec39b4971232add308f01a5809313adcf"`', src/utils.rs:2100:9


failures:
    utils::tests::correct_script_data_hash

test result: FAILED. 109 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out

[kodemill]

@vsubhuman I've fixed the failing test and added a couple more to cover the changes introduced in this PR.

[vsubhuman]

I've fixed the failing test and added a couple more to cover the changes introduced in this PR.

Thank you, @kodemill !