CIP-0010: schema & process updates

[Godspeed-exe]

In my spare time I'm working on a tool that should make it easy for non-native-Cardano entities to (i) create assets, (ii) submit metadata (both transaction and inline datums) without having to build a transaction processing engine, learn how UTXO's work, how to balance a transaction, etc. Just one-click-deploy and push data to the API.

For sanitation purposes I wanted to integrate it with CIP10, to:

• check if a metadata label was registered: reason was to force them to at least register a new label instead of just pushing records without documenting it
• verify per record that it's format was valid according to the documented standard.

In theory it's pretty easy, but in practice we would want a specific metadata label to point to some sort of json schema or cddl file.

A CIP10 entry today does not enforce / require linking to something like this, and I want to suggest to correct this.
Today, there is no way for a new person to know that metadata label for 721 schema is available / documented in CIP25.
Please correct me if I'm wrong.

So my proposal would be to:

• backfill the old CIP10 records based on the records that exist on-chain today
• make the below schema adjustment and require a schema to exist before merging

{ "transaction_metadatum_label": 721, "description": "CIP-0025 - NFT Token Standard", "schema": [ "CIP-0025/cddl/version_1.cddl", "CIP-0025/cddl/version_2.cddl" ] }

Of course, making it required is a preference.. But let's discuss! Happy to brainstorm, clarify or improve!

WDYT @rphair ?

[rphair]

@Godspeed-exe I think the system you're describing would provide useful tooling around what CIP-0010 originally established... and therefore I don't think this needs to change or update CIP-0010 itself any more than wallets that handle CIP-0025 or CIP-0068 tokens need to modify those standards.

A metadata utility like the one you're describing, if adopted well enough by the community, might generate the best analytics we could get about what's missing (or wrong, or redundant, or never used) in the CIP-0010 database.

As I understand CIP-0010, it doesn't require project representatives themselves to submit only their own project's registry entry... only that the new entries be verifiable somehow. So at intervals of analysing the transactions in your planned system, you could set it to do a JSON dump of the labels in use and create from it a PR to update the entire database accordingly.

That would leave a huge amount of verification to the editors, though: since without first hand access to those analytics we'd never be able to authenticate the many metadata label additions, deletions, and changes that would be submitted at least in the beginning.

We might already be in a paradox over CIP-0010: the protocol that was intended to be decentralised & impersonal, to support a realistically broad current vocabulary of metadata labels, would have to be centralised one way or another... either

• editors reviewing mass updates from tools such as the one you're working on, through the CIP-0010 mandated centralised manual review process, or
• manage the CIP-0010 database automatically in a full-blown version of your idea above: a centralised, autonomous agent whose behaviour could be fully documented and peer reviewed.

Because the latter approach could be legitimised through the CIP process, I think we have a scope of a CIP + a CPS that could, when ready, "take over" the database that was created through CIP-0010:

• CPS: to characterise this problem, likely including some updates to the metadata tag schema that would suit automation and verification;
• CIP: for an open source implementation of your "transaction processing engine" and process by which its analytics would (continuously or periodically) produce the metadata label DB updates.

Those are all the thoughts I have about how this could be standardised & I'm tagging this Triage to introduce at our next CIP meeting (https://hackmd.io/@cip-editors/111). I think at least @Ryun1 and @Crypto2099 have thought about CIP-0010 automation before and perhaps they also have some approaches to consider.

[Godspeed-exe]

Hello @rphair

Thanks for the feedback.

My tool would not be creating new metadata label registrations in an automated way. Entities would still be required to come to this repo and register their metadata label.

It would allow Web2 entities to post transactions (with or without metadata) on chain. In case there is metadata I wanted to verify the metadata is in the required format and for this we'll need to:

1. have a schema validator
2. have a logical way of going from CIP10 to that schema validator

Ideally we would have (let's say) a 1000 new metadata labels for 1000 new use cases, but we're not there yet (thank god). So adding a logical way to go from number to format does make sense.

But I'm happy to jump in on the call to listen in / clarify!

[rphair]

thanks @Godspeed-exe for the clarification. My #1034 (comment) was imagining that the previously unregistered metadata tags would be accumulated by the system & could periodically be assembled into lists of commonly used tags that weren't already defined.

I do think that would be of service the community in "crowdsourcing" a better tag vocabulary but it certainly wouldn't be a requirement... rather a voluntary standard that more such services might participate in, and hence my suggestion that it would make a good CIP.

Any updates to the (likely outdated) CIP-0010 schema (and the CIP document in sync) would be welcome in the meantime and I think at the meeting we could quickly get an outline together of changes that would suit everyone: then you could submit such a PR afterward.

[Ryun1]

Do you think this would work best as an extension to CIP-10 and adjust the existing standard?
or perhaps it would be better that this is its own standard, to avoid changing CIP10 as it is widely understood and likely implemented?

A new CIP could create a new metadatum database, using the same primary key
but define ways that people can extend and add new fields/information, like schema as you'd suggested

[Godspeed-exe]

Wellllll, CIP10 is the current metadata label registration schema, which currently lacks a machine readable / understandable way of linking numbers to formats. So it would be an update to the current CIP10 registry. We'll have to see what makes sense in terms of doing this upgrade. There's several approaches:

• big bang
• phased

I'm a fan of:

• updating the workflow to enforce new entries to have a schema
• updating the existing entries to either existing schema's or point to a dummy one and ping the owners / editors with the request to add this or generate one ourselves.

It's a little clean up effort, but in the long term, if we have more automated tools that push data on chain these schema validators would surely try to prevent malformed data from being pushed.

[Godspeed-exe]

hey guys,

Any update on this proposal? I'm happy to contribute in terms of format validators / schema validators to achieve this for existing metadata labels. I've gotten to the point that I would be needing this information.

@rphair @Ryun1

[Godspeed-exe]

This is the service that I'm using CIP10 in: https://github.com/Godspeed-exe/heron-cardano/blob/master/heron_app/utils/registry_loader.py

Right now I'm just checking if a certain metadata label is used: https://github.com/Godspeed-exe/heron-cardano/blob/master/heron_app/api/transactions.py#L55

But I would also LOVE to be able to validate submitted metadata to the standards - preventing "shit" to be submitted.

[rphair]

@Godspeed-exe apologies that a number of weighty proposals to adjust the CIP process have also come in over the last few weeks: personally I was waiting to see how these developed to see if they would also guide how this issue might go. It does seem, as of recently (this week's CIP meeting a couple days ago), that the two areas are completely independent.

---

I would say we are facing a binary decision, about which 2 editorial opinions so far are split:

1 - Since it seems to me like a new schema field would simply be ignored by existing CIP-0010 parsers that weren't looking for it — especially if it's defined as optional — I would take the somewhat reckless approach and declare a v2 update to the CIP which would simultaneously, when merged:

• put a schema field into registry.schema.json: with type definition either to accept an elaborated schema or a null value (or an optional parameter)
• update CIP-0010 with this v2 specification, adding:
  • a new Versioning section (for the behaviour)
  • a new Changelog section (for the document)
  • probably yourself @Godspeed-exe as a co-author
• populate all existing entries in registry.json with that null value (if it's required rather than optional)

I think this change would be much smoother with an optional schema value, which would make modification of existing CIP-0010 entries unnecessary.

• Unfortunately, a required schema value — since it would have to be settable to null (or [] for backward compatibility of existing entries — wouldn't "force" metadata label contributors to provide an elaborated schema.

2 - @Ryun1 — with a time-honoured conservative approach to tooling compatibility — is cautious about such a modification breaking existing tooling. I would defer to his broader knowledge & experience about this but also think it's been vital to involve other developers in determining exactly what would actually happen to existing tooling: and then put in place any migration plans that might be needed.

At the CIP meeting 2 weeks + 2 days ago I put forward my "reckless" plan, not really expecting the conservative voices to endorse it. @gufmar @gitmachtl @Crypto2099 @AndrewWestberg what do you think we could "get away with" here?

• Should we take a moderated, phased approach, or would this be "overthinking it"?
• Would the community demand that we walk away from CIP-0010 into a new metadata standard & leave the legacy metadata behind in a separate construct?

@Godspeed-exe I would welcome a PR to do define & execute the v2 as above but it's not my place to approve it without a community consensus behind it. Since you have been working in this field for a while & have your own tooling on the horizon please by all means tag some more people regarding the specific design features + compatibility questions... so you can build them into a PR we'd know we can accept.

Tip

I do think we have a chance of getting a better quality review from the tagged reviewers & stakeholders if you produce a PR with whatever implementation details you've chosen... and then certainly tag them there (vs. this hypothetical issue).

re: #1034 (comment) I do believe GitHub automation for CIP-0010 is a secondary issue here, since the editors could just manage the new requirements by hand (with a few seconds per pull request) while integrating your plan to automate the important part: which I assume would be JSON syntax checking for any newly contributed entries with an elaborated (not null) schema.

[rphair]

p.s. another design question I thought of when walking away from #1034 (comment):

What about versioning of the schema variables themselves? Will the schema for schema handle version branches that metadata label providers can use it to avoid incompatibilities when a schema evolves, without having to use a new label?

[Ryun1]

Thanks for keeping this one moving @rphair @Godspeed-exe

Happy to see a PR to CIP-10 as described by Robert via option 1

if there is any major push back from the community and CIP10 implementors, then I think we can safely move the content into its own CIP as captured by Robert's comment option 2

[rphair]

(a bit more detail I had in draft) @Godspeed-exe discussion at the biweekly CIP meeting today confirmed that we would welcome a PR from you on this, including the following feedback (most of which confirms #1034 (comment):

• We should use the existing CIP-0010 database and just add a v2 to the CIP that adds an optional "schema" field... so as not to require a format migration for this file and probably not require any immediate metadata tooling changes.
• Each metadata label will simply be "dumb" (or proprietary: which metadata architects should feel free to specify) if a schema is not provided.
• There is no advantage to a null schema field for labels without schemas vs. making the schema an optional field.

Once that PR is submitted then we can re-tag the metadata & tooling stakeholders and subject matter experts to see if they can find any flaws or anticipate any tooling incompatibilities.

If those incompatibilities were found, we would either agree upon a migration process or declare that CIP-0010 should be deprecated by a new CIP with a new schema-capable database.

p.s. with this recommendation this issue is definitively triaged: so removing the label.