feat!: Use binary envelopes for operation lower_func encoding #2447

aborgna-q · 2025-07-17T12:56:46Z

~~Blocked by #2445 due to errors with the hugr-py model encoding.~~

Changes the format used to encode the LowerFunc::FixedHugrs in an extension's operation from string-encoded envelopes to base64-encoded binary envelopes.

Adds an envelope::serde_with::AsBinaryEnvelope wrapper similar to AsStringEnvelope, that lets us include base64-encoded envelopes in a serde definition.

hugr-py does not yet support loading binary envelopes, so we ignore lowering functions when loading extensions.
The deserializer can still load older extensions. We check for the envelope magic numbers on the encoded string and skip the base64 deserialization if found.
Older hugr versions will not be able to read extensions encoded this way, and will fail with an "invalid magic" error.
The schema does not change, since the hugr field is still a string.

drive-by: Adds a custom deserializing function for LowerFunc so errors found while decoding the fixed hugr get communicated back up. The default serde derive produced an opaque "no variants match" error due to using #[serde(untagged)].

BREAKING CHANGE: Lowering functions in extension operations are now encoded as binary envelopes. Older hugr versions will error out when trying to load them.

codecov · 2025-07-17T13:00:03Z

Codecov Report

Attention: Patch coverage is 84.25197% with 20 lines in your changes missing coverage. Please review.

Project coverage is 82.42%. Comparing base (1a301eb) to head (152507a).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
hugr-core/src/envelope/serde_with.rs	83.63%	9 Missing and 9 partials ⚠️
hugr-core/src/extension/op_def.rs	91.66%	0 Missing and 1 partial ⚠️
hugr-py/src/hugr/_serialization/extension.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2447      +/-   ##
==========================================
+ Coverage   82.03%   82.42%   +0.39%     
==========================================
  Files         247      247              
  Lines       46018    46185     +167     
  Branches    41608    41764     +156     
==========================================
+ Hits        37749    38070     +321     
+ Misses       6250     6076     -174     
- Partials     2019     2039      +20

Flag	Coverage Δ
python	`87.74% <80.00%> (+2.47%)`	⬆️
rust	`81.86% <84.42%> (+0.17%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

hugr-core/src/envelope/serde_with.rs

ss2165 · 2025-07-18T11:15:29Z

hugr-core/src/envelope/serde_with.rs

+
+                let str = source
+                    .store_str_with_exts(
+                        $crate::envelope::EnvelopeConfig::text(),


Suggested change

$crate::envelope::EnvelopeConfig::text(),

$crate::envelope::EnvelopeConfig::binary(),

? maybe I've misunderstood

ss2165 · 2025-07-18T11:21:46Z

Would be nice to have a bit more coverage

ss2165

beautiful test

This release fixes multiple inconsistencies between the serialization formats and improves the error messages when loading unsupported envelopes. We now also support nodes with up to `2^32` connections to the same port (up from `2^16`). --- ## 🤖 New release * `hugr-model`: 0.21.0 -> 0.22.0 (✓ API compatible changes) * `hugr-core`: 0.21.0 -> 0.22.0 (✓ API compatible changes) * `hugr-llvm`: 0.21.0 -> 0.22.0 (✓ API compatible changes) * `hugr-passes`: 0.21.0 -> 0.22.0 (✓ API compatible changes) * `hugr-persistent`: 0.1.0 -> 0.2.0 (✓ API compatible changes) * `hugr`: 0.21.0 -> 0.22.0 (✓ API compatible changes) * `hugr-cli`: 0.21.0 -> 0.22.0 (✓ API compatible changes) <details><summary>Changelog</summary> ## `hugr-model` <blockquote> ## [0.22.0](hugr-model-v0.21.0...hugr-model-v0.22.0) - 2025-07-24 ### New Features - Names of private functions become `core.title` metadata. ([#2448](#2448)) - include generator metatada in model import and cli validate errors ([#2452](#2452)) - Version number in hugr binary format. ([#2468](#2468)) - Use semver crate for -model version, and include in docs ([#2471](#2471)) </blockquote> ## `hugr-core` <blockquote> ## [0.22.0](hugr-core-v0.21.0...hugr-core-v0.22.0) - 2025-07-24 ### Bug Fixes - Ensure SumTypes have the same json encoding in -rs and -py ([#2465](#2465)) ### New Features - Export entrypoint metadata in Python and fix bug in import ([#2434](#2434)) - Names of private functions become `core.title` metadata. ([#2448](#2448)) - [**breaking**] Use binary envelopes for operation lower_func encoding ([#2447](#2447)) - [**breaking**] Update portgraph dependency to 0.15 ([#2455](#2455)) - Detect and fail on unrecognised envelope flags ([#2453](#2453)) - include generator metatada in model import and cli validate errors ([#2452](#2452)) - [**breaking**] Add `insert_region` to HugrMut ([#2463](#2463)) - Non-region entrypoints in `hugr-model`. ([#2467](#2467)) </blockquote> ## `hugr-llvm` <blockquote> ## [0.21.0](hugr-llvm-v0.20.2...hugr-llvm-v0.21.0) - 2025-07-09 ### New Features - [**breaking**] No nested FuncDefns (or AliasDefns) ([#2256](#2256)) - [**breaking**] Split `TypeArg::Sequence` into tuples and lists. ([#2140](#2140)) - [**breaking**] More helpful error messages in model import ([#2272](#2272)) - [**breaking**] Merge `TypeParam` and `TypeArg` into one `Term` type in Rust ([#2309](#2309)) - Add `MakeError` op ([#2377](#2377)) </blockquote> ## `hugr-passes` <blockquote> ## [0.22.0](hugr-passes-v0.21.0...hugr-passes-v0.22.0) - 2025-07-24 ### New Features - ReplaceTypes allows linearizing inside Op replacements ([#2435](#2435)) - Add pass for DFG inlining ([#2460](#2460)) </blockquote> ## `hugr-persistent` <blockquote> ## [0.2.0](hugr-persistent-v0.1.0...hugr-persistent-v0.2.0) - 2025-07-24 ### New Features - [**breaking**] Update portgraph dependency to 0.15 ([#2455](#2455)) </blockquote> ## `hugr` <blockquote> ## [0.22.0](hugr-v0.21.0...hugr-v0.22.0) - 2025-07-24 ### Bug Fixes - Ensure SumTypes have the same json encoding in -rs and -py ([#2465](#2465)) ### New Features - ReplaceTypes allows linearizing inside Op replacements ([#2435](#2435)) - Add pass for DFG inlining ([#2460](#2460)) - Export entrypoint metadata in Python and fix bug in import ([#2434](#2434)) - Names of private functions become `core.title` metadata. ([#2448](#2448)) - [**breaking**] Use binary envelopes for operation lower_func encoding ([#2447](#2447)) - [**breaking**] Update portgraph dependency to 0.15 ([#2455](#2455)) - Detect and fail on unrecognised envelope flags ([#2453](#2453)) - include generator metatada in model import and cli validate errors ([#2452](#2452)) - [**breaking**] Add `insert_region` to HugrMut ([#2463](#2463)) - Non-region entrypoints in `hugr-model`. ([#2467](#2467)) </blockquote> ## `hugr-cli` <blockquote> ## [0.22.0](hugr-cli-v0.21.0...hugr-cli-v0.22.0) - 2025-07-24 ### New Features - include generator metatada in model import and cli validate errors ([#2452](#2452)) </blockquote> </details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/). --------- Co-authored-by: Agustín Borgna <[email protected]>

🤖 I have created a release *beep* *boop* --- ## [0.13.0rc1](hugr-py-v0.12.2...hugr-py-v0.13.0rc1) (2025-07-24) ### ⚠ BREAKING CHANGES * Lowering functions in extension operations are now encoded as binary envelopes. Older hugr versions will error out when trying to load them. * **py:** `EnvelopeConfig::BINARY` now uses the model binary encoding. `EnvelopeFormat.MODULE` is now `EnvelopeFormat.MODEL`. `EnvelopeFormat.MODULE_WITH_EXTS` is now `EnvelopeFormat.MODEL_WITH_EXTS` * hugr-model: Symbol has an extra field * Renamed the `Any` type bound to `Linear` * The model CFG signature types were changed. * Added `TypeParam`s and `TypeArg`s corresponding to floats and bytes. * `TypeArg::Sequence` needs to be replaced with `TypeArg::List` or `TypeArg::Tuple` * FuncDefns must be moved to beneath Module. `Container::define_function` is gone, use `HugrBuilder::module_root_builder`; similarly in hugr-py `DefinitionBuilder` (`define_function` -> `module_root_builder().define_function`). In hugr-llvm, some uses of ### Features * Add `BorrowArray` extension ([#2395](#2395)) ([782687e](782687e)) * Add `MakeError` op ([#2377](#2377)) ([909a794](909a794)), closes [#1863](#1863) * add toposort to HUGR-py ([#2367](#2367)) ([34eed34](34eed34)) * Add Visibility to FuncDefn/FuncDecl. ([#2143](#2143)) ([5bbe0cd](5bbe0cd)) * Added float and bytes literal to core and python bindings. ([#2289](#2289)) ([e9c5e91](e9c5e91)) * **core, llvm:** add array unpack operations ([#2339](#2339)) ([a1a70f1](a1a70f1)), closes [#1947](#1947) * Detect and fail on unrecognised envelope flags ([#2453](#2453)) ([5e36770](5e36770)) * Export entrypoint metadata in Python and fix bug in import ([#2434](#2434)) ([d17b245](d17b245)) * Expose `BorrowArray` in `hugr-py` ([#2425](#2425)) ([fdb675f](fdb675f)), closes [#2406](#2406) * include generator metatada in model import and cli validate errors ([#2452](#2452)) ([f7cedb4](f7cedb4)) * Names of private functions become `core.title` metadata. ([#2448](#2448)) ([4bc7f65](4bc7f65)) * No nested FuncDefns (or AliasDefns) ([#2256](#2256)) ([214b8df](214b8df)) * Non-region entrypoints in `hugr-model`. ([#2467](#2467)) ([7b42da6](7b42da6)) * Open lists and tuples in `Term` ([#2360](#2360)) ([292af80](292af80)) * **py:** enable Model as default BINARY envelope format ([#2317](#2317)) ([f089931](f089931)) * **py:** Helper methods to get the neighbours of a node ([#2370](#2370)) ([bb6fa50](bb6fa50)) * **py:** Use SumValue serialization for tuples ([#2466](#2466)) ([f615037](f615037)) * Rename 'Any' type bound to 'Linear' ([#2421](#2421)) ([c2f8b30](c2f8b30)) * Split `TypeArg::Sequence` into tuples and lists. ([#2140](#2140)) ([cc4997f](cc4997f)) * Standarize the string formating of sum types and values ([#2432](#2432)) ([ec207e7](ec207e7)) * Use binary envelopes for operation lower_func encoding ([#2447](#2447)) ([2c16a77](2c16a77)) ### Bug Fixes * Ensure SumTypes have the same json encoding in -rs and -py ([#2465](#2465)) ([7f97e6f](7f97e6f)) * Escape html-like labels in DotRenderer ([#2383](#2383)) ([eaa7dfe](eaa7dfe)) * Export metadata in Python ([#2342](#2342)) ([7be52db](7be52db)) * Fix model export of `Opaque` types. ([#2446](#2446)) ([3943499](3943499)) * Fixed bug in python model export name mangling. ([#2323](#2323)) ([041342f](041342f)) * Fixed bugs in model CFG handling and improved CFG signatures ([#2334](#2334)) ([ccd2eb2](ccd2eb2)) * Fixed export of `Call` and `LoadConst` nodes in `hugr-py`. ([#2429](#2429)) ([6a0e270](6a0e270)) * Fixed invalid extension name in test. ([#2319](#2319)) ([c58ddbf](c58ddbf)) * Fixed two bugs in import/export of function operations ([#2324](#2324)) ([1ad450f](1ad450f)) * map IntValue to unsigned repr when serializing ([#2413](#2413)) ([26d426e](26d426e)), closes [#2409](#2409) * Order hints on input and output nodes. ([#2422](#2422)) ([a31ccbc](a31ccbc)) * **py:** correct ConstString JSON encoding ([#2325](#2325)) ([9649a48](9649a48)) * StaticArrayVal payload encoding, improve roundtrip checker ([#2444](#2444)) ([1a301eb](1a301eb)) * stringify metadata before escaping in renderer ([#2405](#2405)) ([8d67420](8d67420)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Release-As: 0.13.0rc1 --------- Co-authored-by: Agustín Borgna <[email protected]>

## 🤖 New release * `hugr-model`: 0.22.0 -> 0.22.1 * `hugr-core`: 0.22.0 -> 0.22.1 * `hugr-llvm`: 0.22.0 -> 0.22.1 * `hugr-passes`: 0.22.0 -> 0.22.1 (✓ API compatible changes) * `hugr`: 0.22.0 -> 0.22.1 (✓ API compatible changes) * `hugr-cli`: 0.22.0 -> 0.22.1 * `hugr-persistent`: 0.2.0 -> 0.2.1 <details><summary>Changelog</summary> ## `hugr-model` <blockquote> ## [0.22.0](hugr-model-v0.21.0...hugr-model-v0.22.0) - 2025-07-24 ### New Features - Names of private functions become `core.title` metadata. ([#2448](#2448)) - include generator metatada in model import and cli validate errors ([#2452](#2452)) - Version number in hugr binary format. ([#2468](#2468)) - Use semver crate for -model version, and include in docs ([#2471](#2471)) </blockquote> ## `hugr-core` <blockquote> ## [0.22.0](hugr-core-v0.21.0...hugr-core-v0.22.0) - 2025-07-24 ### Bug Fixes - Ensure SumTypes have the same json encoding in -rs and -py ([#2465](#2465)) ### New Features - Export entrypoint metadata in Python and fix bug in import ([#2434](#2434)) - Names of private functions become `core.title` metadata. ([#2448](#2448)) - [**breaking**] Use binary envelopes for operation lower_func encoding ([#2447](#2447)) - [**breaking**] Update portgraph dependency to 0.15 ([#2455](#2455)) - Detect and fail on unrecognised envelope flags ([#2453](#2453)) - include generator metatada in model import and cli validate errors ([#2452](#2452)) - [**breaking**] Add `insert_region` to HugrMut ([#2463](#2463)) - Non-region entrypoints in `hugr-model`. ([#2467](#2467)) </blockquote> ## `hugr-llvm` <blockquote> ## [0.21.0](hugr-llvm-v0.20.2...hugr-llvm-v0.21.0) - 2025-07-09 ### New Features - [**breaking**] No nested FuncDefns (or AliasDefns) ([#2256](#2256)) - [**breaking**] Split `TypeArg::Sequence` into tuples and lists. ([#2140](#2140)) - [**breaking**] More helpful error messages in model import ([#2272](#2272)) - [**breaking**] Merge `TypeParam` and `TypeArg` into one `Term` type in Rust ([#2309](#2309)) - Add `MakeError` op ([#2377](#2377)) </blockquote> ## `hugr-passes` <blockquote> ## [0.22.1](hugr-passes-v0.22.0...hugr-passes-v0.22.1) - 2025-07-28 ### New Features - Include copy_discard_array in DelegatingLinearizer::default ([#2479](#2479)) - Inline calls to functions not on cycles in the call graph ([#2450](#2450)) </blockquote> ## `hugr` <blockquote> ## [0.22.1](hugr-v0.22.0...hugr-v0.22.1) - 2025-07-28 ### New Features - Include copy_discard_array in DelegatingLinearizer::default ([#2479](#2479)) - Inline calls to functions not on cycles in the call graph ([#2450](#2450)) </blockquote> ## `hugr-cli` <blockquote> ## [0.22.0](hugr-cli-v0.21.0...hugr-cli-v0.22.0) - 2025-07-24 ### New Features - include generator metatada in model import and cli validate errors ([#2452](#2452)) </blockquote> ## `hugr-persistent` <blockquote> ## [0.2.0](hugr-persistent-v0.1.0...hugr-persistent-v0.2.0) - 2025-07-24 ### New Features - [**breaking**] Update portgraph dependency to 0.15 ([#2455](#2455)) </blockquote> </details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/).

aborgna-q added 2 commits July 16, 2025 18:04

Better lower func serialization errors

172f85e

better error reporting on for serde_with envelopes

7f61088

aborgna-q added 2 commits July 17, 2025 14:00

Update schema comment

7acf296

Fix doclink visibility

4e9e1ee

aborgna-q marked this pull request as ready for review July 18, 2025 08:28

aborgna-q requested a review from a team as a code owner July 18, 2025 08:28

aborgna-q requested review from acl-cqc and ss2165 and removed request for acl-cqc July 18, 2025 08:28

ss2165 reviewed Jul 18, 2025

View reviewed changes

aborgna-q added 2 commits July 18, 2025 13:33

encode binary hugrs...

cda40c0

cross-compat tests

152507a

aborgna-q requested a review from ss2165 July 18, 2025 14:43

ss2165 approved these changes Jul 18, 2025

View reviewed changes

aborgna-q added this pull request to the merge queue Jul 18, 2025

Merged via the queue into main with commit 2c16a77 Jul 18, 2025
26 checks passed

aborgna-q deleted the ab/extension-binary-hugrs branch July 18, 2025 15:00

This was referenced Jul 18, 2025

chore(main): release hugr-py 0.13.0rc1 #2302

Merged

chore: release #2443

Closed

chore: release #2470

Closed

chore: release #2476

Merged

hugrbot mentioned this pull request Jul 28, 2025

chore: release #2483

Merged

This was referenced Jul 29, 2025

chore: release #2492

Closed

chore: release #2493

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat!: Use binary envelopes for operation lower_func encoding #2447

feat!: Use binary envelopes for operation lower_func encoding #2447

Uh oh!

aborgna-q commented Jul 17, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jul 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

ss2165 Jul 18, 2025

Uh oh!

ss2165 commented Jul 18, 2025

Uh oh!

ss2165 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	$crate::envelope::EnvelopeConfig::text(),
	$crate::envelope::EnvelopeConfig::binary(),

feat!: Use binary envelopes for operation lower_func encoding #2447

feat!: Use binary envelopes for operation lower_func encoding #2447

Uh oh!

Conversation

aborgna-q commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

ss2165 Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

ss2165 commented Jul 18, 2025

Uh oh!

ss2165 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aborgna-q commented Jul 17, 2025 •

edited

Loading

codecov bot commented Jul 17, 2025 •

edited

Loading