feat(ses-ava): Multiplex configurations #2987

kriskowal · 2025-10-09T22:39:11Z

Preface

There are a number of ways an Endo environment can be configured. We consider it table stakes that being able to move between these configurations should have as little friction as possible. Code written that works in one configuration should work in others.

The important configuration is the strict, default post-lockdown environment. This is where most of our test coverage occurs today. However, we also support a lockdown mode hardenTaming: unsafe, through @endo/init/unsafe-fast.js, which lets single-tenant vats like the Agoric chain supervisor run fast and rely on pinky-swear immutability of hardened data and interfaces. We wish to have test coverage for this mode, and to that end, we wish to run our existing test suites in multiple modes.

Description

In order to better cover multiple environment configurations, this change introduces a ses-ava command and machinery to let us reuse tests regardless of whether lockdown is called. Then, adopts this feature in every applicable Endo package, in every configuration for which tests currently pass.

The new ses-ava command passes most flags through to ava, and runs ava once for each of the configurations in package.json, including the default ava if present, then other configuration files named in a sesAvaConfigs section. The ses-ava command then goes on to add --only-config-${x} and --no-config-${x} to include or exclude configurations in a test run, to help facilitate isolation of test failures.

In this change, we update every applicable workspace’s package.json and provide a bank of common AVA configuration files at the root of the repository for each workspace to use. Without modification, we adopt all the configurations that are currently working, with the expectation that we will support 2 or 3 such configurations for most packages going forward.

Security Considerations

None.

Scaling Considerations

Tests will run longer.

Documentation Considerations

See ses-ava/README and ses-ava/NEWS.

Testing Considerations

In entire.

Compatibility Considerations

This will increase test coverage for supported scenarios.

Upgrade Considerations

None.

gibson042 · 2025-10-09T22:55:18Z

packages/ses-ava/README.md

+```
+
+This relies on `@endo/ses-ava/prepare-endo-config.js` to initialize an
+Endo envrionment, including the SES shims and Eventual Send shim, and also


Suggested change

Endo envrionment, including the SES shims and Eventual Send shim, and also

Endo environment, including the SES shims and Eventual Send shim, and also

gibson042 · 2025-10-09T23:00:23Z

packages/ses-ava/test.js

+export const register = newTest => {
+  test = newTest;
+};


I think register is a bit too generic.

Suggested change

export const register = newTest => {

test = newTest;

};

/**

* @template [Context=unknown]

* @param {import('ava').TestFn<Context>} newTestFn

*/

export const replaceAvaTestFn = newTestFn => {

test = newTestFn;

};

gibson042 · 2025-10-09T23:02:33Z

packages/ses-ava/README.md

+The `ses-ava` command consumes the `"ava"` and (new) `"avaConfigs"` properties
+in `package.json` to discover and name the supported configurations and infer
+flags like `--only-configname` and `--no-configname` for each, where the `"ava"`
+configuration is the `default`, if present.


We should rename "avaConfigs" to something that makes clear that the functionality is not being provided by ava itself.

Suggested change

The `ses-ava` command consumes the `"ava"` and (new) `"avaConfigs"` properties

in `package.json` to discover and name the supported configurations and infer

flags like `--only-configname` and `--no-configname` for each, where the `"ava"`

configuration is the `default`, if present.

The `ses-ava` command consumes the `"ava"` and (new) `"sesAvaConfigs"` properties

in `package.json` to discover and name the supported configurations and infer

flags like `--only-configname` and `--no-configname` for each, where the `"ava"`

configuration is the `default`, if present.

Also note that such a wrapper is probably useful for more than just ses-ava... should we use "multiAvaConfigs" in anticipation of possible generalization?

I considered sesAvaConfigs to avoid name-squatting in Ava’s domain, so I think that’s two votes. We can update our packages if ava upstream’s the feature.

gibson042 · 2025-10-09T23:05:12Z

packages/ses-ava/README.md

+With this configuration, `ses-ava ...args --no-lockdown` and `ses-ava ...args
+--only-unsafe` would both just run the `unsafe` configuration.
+Using `ses-ava` under `c8` allows all configurations to cover used code.


What controls which arguments are interpreted by ses-ava vs. passed along? Argument hockey against a moving target gets dicey.

I’m open to other options. My intuition is consistent with yours until I considered the ergonomics. I’m convinced, given the absence of better alternatives, that we should just use a more elaborate pair of prefixes to avoid collisions.

I went with fully virtualizing the ava flags. The new code intercepts the common flags and passes them down. -- allows for pass-thru of arbitrary additional flags, but isn’t necessary in any of the common cases. I went with --no-config- and --only-config- since they don’t collide, are unlikely to collide, and we can compensate for collisions later without breaking anything.

gibson042 · 2025-10-09T23:21:39Z

packages/ses-ava/src/command.js

+    const noKey = noFlags.get(arg);
+    const onlyKey = onlyFlags.get(arg);
+    if (noKey) {
+      no.add(noKey);
+    } else if (onlyKey) {
+      only.add(onlyKey);
+    } else {
+      args.push(arg);
+    }


Yeah, this dynamic sensitivity can make for awful debugging, and hinders future evolution. Suggestion: just claim everything that starts with - up to the first --.

// Claim all options up to the first `--` (if any), but pass through every // positional argument. const args = []; const argsIterator = process.argv.slice(2)[Symbol.iterator](); for (const arg of argsIterator) { if (arg === '--') { args.push(...argsIterator); break; } else if (!arg.startsWith('-')) { args.push(arg); continue; } if (arg.startsWith('--no-')) { no.add(arg.slice('--no-'.length)); } else if (arg.startsWith('--only-')) { only.add(arg.slice('--only-'.length)); } else { throw Error(`Unsupported argument: ${arg}`); } } for (const key of no) { if (Object.hasOwn(avaConfigs, key)) continue; throw Error(`Cannot find excluded configuration: ${key}`); } for (const key of only) { if (Object.hasOwn(avaConfigs, key)) continue; throw Error(`Cannot find exclusive configuration: ${key}`); }

Downside: You then need to yarn test -- -m case.

Also downside: You cannot yarn test -m 'case name' --only-lockdown and iterate on the final argument.

gibson042 · 2025-10-09T23:24:46Z

packages/ses-ava/src/command.js

+      args.push(arg);
+    }
+  }
+  const configs = (only.size > 0 ? only : all).difference(no);


--only-foo with --no-foo should probably be rejected, but if not then letting --no-foo win seems odd.

Rewrote with safeties.

packages/ses-ava/src/command.js

gibson042 · 2025-10-23T14:25:19Z

packages/ses-ava/README.md

+In the root of the Endo repository, look the the `ava-*.config.mjs` modules
+for example configurations.


Suggested change

In the root of the Endo repository, look the the `ava-*.config.mjs` modules

for example configurations.

In the root of the Endo repository, look the at the `ava-*.config.mjs` modules

for example configurations.

gibson042 · 2025-10-23T14:32:03Z

packages/ses-ava/README.md

 a regular dependency, so it you include @endo/ses-ava as a regular
 dependency, bundlers might bundle your code with all of Ava.

 SES-Ava rhymes with Nineveh.


Nit (throughout the documentation): https://github.com/avajs/ava/tree/main#how-is-the-name-written-and-pronounced

Suggested change

SES-Ava rhymes with Nineveh.

SES-AVA rhymes with Nineveh.

gibson042 · 2025-10-23T14:41:25Z

packages/ses-ava/src/command.js

+ * The ses-ava command will by default run all tests in every mode but allows
+ * the user to pass --only-* and --no-* at any argument position for any of
+ * the named configurations to filter.


Suggestion: rather than dynamically defining flags from configuration, use repeatable argument-bearing options.

Suggested change

* The ses-ava command will by default run all tests in every mode but allows

* the user to pass --only-* and --no-* at any argument position for any of

* the named configurations to filter.

* The ses-ava command will by default run all tests in every mode but allows

* the user to pass --only <name> and --exclude <name> (or shorthands -o <name>

* and -x <name>) at any argument position for any of the named configurations

* to filter.

(and if accepted, remember to also update the test descriptions and documentation in e.g. README.md)

I am sure I will regret this, but in the interest of using time efficiently, I’m going to merge without addressing this one, though it is absolutely a good idea.

Captured as #2998.

Eh, I just went ahead and submitted a fix: #2999

gibson042 · 2025-10-23T14:54:38Z

packages/ses-ava/src/command.js

+const singlePassThrough = new Set([
+  '-t',
+  '--tap',
+  '-s',
+  '--serial',
+  '-u',
+  '--update-snapshots',
+  '--no-worker-threads',
+  '--verbose',
+  '--no-verbose',
+]);
+
+const doublePassThrough = new Set(['-m', '--match', '--node-arguments']);


Composite suggestions (format short/long aliases on the same line; include options for ava debug; rename these two variables):

Suggested change

const singlePassThrough = new Set([

'-t',

'--tap',

'-s',

'--serial',

'-u',

'--update-snapshots',

'--no-worker-threads',

'--verbose',

'--no-verbose',

]);

const doublePassThrough = new Set(['-m', '--match', '--node-arguments']);

// We pass these flags directly to ava.

const passThroughFlags = new Set([

...['-s', '--serial'],

...['-t', '--tap'],

...['-T', '--timeout'],

...['-u', '--update-snapshots'],

...['-v', '--verbose'],

'--no-worker-threads',

'--no-verbose',

// Specific to ava debug.

'--break',

]);

// We read a single argument for each of these options and pass the pair

// directly to ava.

const passThroughArgOptions = new Set([

...['-m', '--match'],

'--node-arguments',

// Specific to ava debug.

'--host',

'--port',

]);

gibson042 · 2025-10-23T14:57:46Z

packages/ses-ava/src/command.js

+    const onlyKey = onlyFlags.get(arg);
+    if (singlePassThrough.has(arg)) {
+      passThroughArgs.push(arg);
+    } else if (doublePassThrough.has(arg)) {


Suggested change

} else if (doublePassThrough.has(arg)) {

} else if (doublePassThrough.has(arg)) {

// Note that we don't support GNU-style `--name=value` syntax.

gibson042 · 2025-10-23T15:00:44Z

packages/ses-ava/src/command.js

+      foundNextArg: {
+        for (const nextArg of argsIterator) {
+          passThroughArgs.push(arg, nextArg);
+          break foundNextArg;
+        }
+        throw new Error(`Expected argument after ${arg}`);
+      }


This seems like overkill in comparison to just calling next().

Suggested change

foundNextArg: {

for (const nextArg of argsIterator) {

passThroughArgs.push(arg, nextArg);

break foundNextArg;

}

throw new Error(`Expected argument after ${arg}`);

}

const { done, value } = argsIterator.next();

if (done) throw new Error(`Expected argument after ${arg}`);

passThroughArgs.push(arg, value);

gibson042 · 2025-10-23T15:07:07Z

packages/ses-ava/src/command.js

+        if (failFast) {
+          process.exit();
+        }


Suggested change

if (failFast) {

process.exit();

}

if (failFast && process.exitCode) {

process.exit();

}

gibson042 · 2025-10-23T15:09:29Z

packages/ses-ava/src/command.js

+
+    // Debug will only apply to the first matching configuration.
+    if (debug) {
+      break;
+    }


I think this will be misleading, and also frustrating in cases where I cared about debugging against multiple configurations (or just not the first one). The child processes are run serially, so we might as well just apply debug to all of them.

Suggested change

// Debug will only apply to the first matching configuration.

if (debug) {

break;

}

gibson042 · 2025-10-23T15:20:10Z

packages/ses-ava/package.json

+  "sesAvaConfigs": {
+    "one": "test/_ava1.config.js",
+    "two": "test/_ava2.config.js",
+    "raw": "test/_ses-ava-is-ava.config.js"
+  },


These config names seem unnecessarily generic; can we make the naming (and thus comprehension of how this file expects testing to proceed) more clear? A possibility:

Suggested change

"sesAvaConfigs": {

"one": "test/_ava1.config.js",

"two": "test/_ava2.config.js",

"raw": "test/_ses-ava-is-ava.config.js"

},

"sesAvaConfigs": {

"custom-env": "test/_custom-env.ava-config.js",

"must-ignore": "test/_must-ignore.ava-config.js",

"raw": "test/_raw.ava-config.js"

},

gibson042 · 2025-10-23T15:29:01Z

packages/ses-ava/src/command.js

+  }
+
+  // Execute configurations serially.
+  for (const config of configs) {


Suggestion: tack on our own detail.

Suggested change

for (const config of configs) {

for (const config of configs) {

console.warn(`[ses-ava] config:`, config);

(or similar)

kriskowal requested review from erights, gibson042 and mhofman October 9, 2025 22:40

kriskowal force-pushed the kriskowal-ses-ava-multi-config branch 2 times, most recently from 756468c to eb857e3 Compare October 9, 2025 23:06

gibson042 reviewed Oct 9, 2025

View reviewed changes

kriskowal force-pushed the kriskowal-ses-ava-multi-config branch from eb857e3 to 226bd4f Compare October 9, 2025 23:26

gibson042 reviewed Oct 9, 2025

View reviewed changes

kriskowal force-pushed the kriskowal-ses-ava-multi-config branch 6 times, most recently from 546fbed to 84e6784 Compare October 10, 2025 04:36

kriskowal requested a review from gibson042 October 10, 2025 16:33

kriskowal force-pushed the kriskowal-ses-ava-multi-config branch 8 times, most recently from 87d00cd to 342ac49 Compare October 11, 2025 03:10

gibson042 approved these changes Oct 23, 2025

View reviewed changes

kriskowal force-pushed the kriskowal-ses-ava-multi-config branch 2 times, most recently from ecec978 to df3ba90 Compare October 28, 2025 01:37

kriskowal enabled auto-merge October 28, 2025 01:39

kriskowal added 3 commits October 27, 2025 18:43

feat(ses-ava): Support multiple configurations, with or without lockdown

0a80ede

test: Adopt multi-config ava testing

452b00f

fix(skel): Skip placeholder test

e7eaa64

chore: Update yarn.lock

2b0119f

kriskowal force-pushed the kriskowal-ses-ava-multi-config branch from df3ba90 to 2b0119f Compare October 28, 2025 01:43

kriskowal merged commit e914ed8 into master Oct 28, 2025
19 checks passed

kriskowal deleted the kriskowal-ses-ava-multi-config branch October 28, 2025 01:50

This was referenced Oct 28, 2025

[ses-ava] Dynamic --{no,only}-config-$name should be replaced with repeatable --{only,exclude} $name #2998

Closed

refactor(ses-ava): Remove use of ambient process from main #3002

Merged

	Endo envrionment, including the SES shims and Eventual Send shim, and also
	Endo environment, including the SES shims and Eventual Send shim, and also

-export const register = newTest => {
-  test = newTest;
-};
+/**
+ * @template [Context=unknown]
+ * @param {import('ava').TestFn<Context>} newTestFn
+ */
+export const replaceAvaTestFn = newTestFn => {
+  test = newTestFn;
+};

		In the root of the Endo repository, look the the `ava-*.config.mjs` modules
		for example configurations.

-const singlePassThrough = new Set([
-  '-t',
-  '--tap',
-  '-s',
-  '--serial',
-  '-u',
-  '--update-snapshots',
-  '--no-worker-threads',
-  '--verbose',
-  '--no-verbose',
-]);
-const doublePassThrough = new Set(['-m', '--match', '--node-arguments']);
+// We pass these flags directly to ava.
+const passThroughFlags = new Set([
+  ...['-s', '--serial'],
+  ...['-t', '--tap'],
+  ...['-T', '--timeout'],
+  ...['-u', '--update-snapshots'],
+  ...['-v', '--verbose'],
+  '--no-worker-threads',
+  '--no-verbose',
+  // Specific to ava debug.
+  '--break',
+]);
+// We read a single argument for each of these options and pass the pair
+// directly to ava.
+const passThroughArgOptions = new Set([
+  ...['-m', '--match'],
+  '--node-arguments',
+  // Specific to ava debug.
+  '--host',
+  '--port',
+]);

	} else if (doublePassThrough.has(arg)) {
	} else if (doublePassThrough.has(arg)) {
	// Note that we don't support GNU-style `--name=value` syntax.

	for (const config of configs) {
	for (const config of configs) {
	console.warn(`[ses-ava] config:`, config);

feat(ses-ava): Multiplex configurations #2987

feat(ses-ava): Multiplex configurations #2987

Uh oh!

Conversation

kriskowal commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Preface

Description

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Compatibility Considerations

Upgrade Considerations

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kriskowal commented Oct 9, 2025 •

edited

Loading