feat(compartment-mapper): add "preload" option to captureFromMap()

[boneskull]

Prior art:

Ref: #2856
Ref: #2864
Ref: #2876

Description

This adds option preload to captureFromMap(). This is an optional string[] of keys of the CompartmentMapDescriptor also provided to captureFromMap(). It can also be an Array<{compartment: string, entry: string}> to preload a specific module other than the entry of that compartment.

After loading the entry Compartment (and attenuators Compartment, if applicable), the Compartments having names (practically speaking, these will be locations) mentioned in preload will then be loaded if they were not already.

This option can be used to support dynamic requires and imports which would otherwise be omitted from the captured CompartmentMapDescriptor via digestion.

• Refactored capture-lite.js; stuffed all of the Compartment-loading business into a function.
• Added test and fixture that shows how a Compartment which would otherwise be omitted from the captured Compartment Map is included when used with preload.

Alternatives Considered

Besides the above references, I considered adding a boolean field load to CompartmentDescriptor. But I don't see how @endo/compartment-mapper would want to set this flag for any reason other than in captureFromMap() on the entry and attenuators compartments. Thus, it seemed specific to captureFromMap() and so shouldn't be in CompartmentDescriptor.

Supplying an array of file URL strings is kind of a pain for the end-user, but they're right there in the CompartmentMapDescriptor, so it seems workable.

Open to naming suggestions.

Security Considerations

Just the usual disclaimers about being careful what you put in the compartment map.

Scaling Considerations

There's a tradeoff: use of preload has performance implications, since it causes (potentially many) module sources to be read and analyzed which would not otherwise be.

Testing Considerations

There are probably some conditionals I've left uncovered.

Compatibility Considerations

Backwards-compatible

Upgrade Considerations

This is a user-facing change which should be described in NEWS.md.

[boneskull]

Is a CompartmentDescriptor not having a ModuleDescriptor referencing itself considered malformed?

[boneskull]

besides the refactors, I renamed:

• parameter powers ➡️ readPowers
• internal name of option Compartment ➡️ CompartmentOption
• module-scoped defaultCompartment ➡️ DefaultCompartment
• internal entry Compartment compartment ➡️ entryCompartment
• internal Record<string, Compartment> post-linking compartments ➡️ linkedCompartments

[boneskull]

Without this change, the generated policy.d.ts has no reference to SomePackagePolicy and is thus broken. I would expect lint:types to report this error, but it doesn't and I don't know why.

[boneskull]

ref: #2921

[naugtur]

With outside-facing compartment map transforms the use of a marker in compartment map indicating what to load looks appealing again.
Instead of loading entry by default, the approach would be to load everything marked as a linkingStartNode, which would be true for entry by default.

As a result adding missing links and linking start nodes based on policy overrides from lavamoat could happen in one transform provided by lavamoat.

I like the idea that compartment-mapper's linking and loading would be an engine to do whatever compartment map says, with as little configurability as possible. Configurability made more sense before transforms, but if we land transforms they could replace a lot of it.

[boneskull]

@naugtur Yes, I may have mentioned above that I considered a similar design. I'm happy to go that way if we feel it's a better fit. Still, there will need to be an option somewhere because how else is mapNodeModules going to know which CompartmentDescriptor to flag for loading?

[kriskowal]

I’m sold on the need for the feature and mostly sold on the implementation.

I think forceLoad should be framed as a set of {compartment, module} duples, where module may default to ".".

I was confused during my review by the verb “load” being applied to compartments, since we capture compartments and load modules. This is indeed loading modules, as synecdoche for the main module of a compartment. That could be clearer in choices of names.

Consider some more names for forceLoad:

• preload (I think “force” is dissatisfying because we’re not overriding a safety measure.)
• entries (If this is an array of additional entry modules with the same type as the entry in compartment-map.json)

Consider, in fact, instead of a option, a way to add an entries field to the compartment map input to the argument. That might be threaded from an option higher up.

I had envisioned a feature of extra entries in compartment-map.json such that app.import(name) would produce one of the extra modules. This would be useful for staged evaluation of modules in a bundle.

{
  "entry": { "module": ".", "compartment": "x" },
  "entries": {
    "y": { "module": ".", "compartment": "y"}
  }
}

[kriskowal]

It might be good to say globalThis.Compartment on the RHS so it’s not a reference error, in cases where it’s intended to be injected.

[boneskull]

globalThis being undefined per ESLint seems like a misconfiguration. SES also does not assign its globals to globalThis, which incurs a TS error. I just wanted to note this.

Regardless, your suggestion is correct, so I will just squelch the TS error for now.

[naugtur]

This is only used in capture-lite. Would an application work at runtime when its policy generation situation depended on this? I'm assuming it'd go and do a dynamic import for what's not been preloaded. But wanted to point at it and say I'm not sure if using it only in one place is enough.