Plugin to generate webmanifest

[JakobJingleheimer]

Possibly prescriptive, but I'm trying to create a plugin to generate a webmanifest. My working design is to include the webmanifest file in entryPoints, use the plugin to filter on that file-type (.webmanifest, which is actually just a json file), and do the behind the scenes work within the plugin.

I'm thinking rather than hard-coding file-paths into the json source file (which tooling does not connect to the FS), it's better to inject them via the plugin based on some assumed FS structure like:

src/
  webmanifest/
    icons/
      192.png
      512.png
    app.webmanifest

I've got the basic plugin working to handle the app.webmanifest file.

I think the simplest way is to add those dependencies into esbuild's inputs/outputs so they're processed normally. But I can't figure out how to add the icons to esbuild's inputs as if they were "imported" by app.webmanifest so they get appropriately hashed etc.

I thought maybe including them in OnLoadResult['watchFiles'] or "looking them up" via PluginBuild['resolve'] would cause that to happen, but no.

Note that the outputted result for the manifest file must be a json file, not a javascript file (it "works" if esbuild is allowed to transform the manifest into js, but that isn't suitable because a manifest file is a json file).

Also, I have a different plugin (as a workaround for esbuild not supporting html as an entry-point: #31) that generates an HTML file and injects <link>s and <script>s based on data in the build.metafile, so that data needs to be accurate (so generating a fake manifest.js will create a problem with the html generator plugin).

[JakobJingleheimer]

Perhaps something clever. It doesn't need to start as json, only end as json. So maybe the entry-point is a js/ts file that actually imports those icons. Then I just need a way to get esbuild to output that input as json (and it correctly reflect that in the metafile).

Ah, perhaps not: #2560

It seems esbuild is trying very hard to make this very difficult 😬

[JakobJingleheimer]

@hyrious you usually have a clever idea. Any nuggets you can share?

[hyrious]

First of all, not everything should be implemented in the esbuild's build pipeline. There are fs.readFile / fs.writeFile. You can write a few scripts to do the job.

Here is the way to "hack" into the esbuild pipeline and consumes / outputs anything as you want:

For inputs, just put them in the entry points (like { entryPoints: ['src/app.webmanifest'] }). Use the onResolve plugin API to resolve it to the dest file path.

For outputs, use the onLoad plugin API to load the file path to meaningful data, and return the data with loader: 'copy' so esbuild will then write the data to the dest file path.

Here's a working example, that consumes a template HTML file, builds and runs the svelte SSR module, then output the SSR-ed HTML file: https://github.com/hyrious/esbuild-repl/blob/main/scripts/plugins/svelte-ssr.ts The advantage of this approach is that it doesn't actually write the output to the file system in serve mode. Instead the browser can fetch the output file from memory.

[JakobJingleheimer]

Thanks!

Sure, not everything should go through esbuild. But 98% of what I want esbuild already does: file watch, content-hashing, dep crawling, metafile. I can write the resulting json file myself if esbuild won't. But I think this is a fairly legitimate (albeit small) use-case for JavaScript → JSON.

Is the nested esbuild.build how the svelte file's deps get processed?

I see you're hacking the module graph with a dynamic import + timestamp cache-buster. I thought of that, but was hoping to avoid that 🤔

[hyrious]

Is the nested esbuild.build how the svelte file's deps get processed?

Yes. The svelte SSR simply works by running the svelte component in Node.js. To get the Node.js code it requres a separate build to compile & bundle App.svelte → App-bundle.js.

The dynamic import in the plugin is only needed for this case as Node.js will cache every module by their file path. So I need the timestamp to get the fresh svelte module in every build in watch mode.

[JakobJingleheimer]

it requres a separate build to compile & bundle App.svelte → App-bundle.js

Why though? I would expect esbuild to already do that since (at least in my case) App.svelte is an entry-point in the parent like

import { fileURLToPath } from 'node:url';

import { build } from 'esbuild';

import { compileIndexEJSPlugin } from './compile-index-ejs.ts';
import { compileWebmanifestPlugin } from './compile-webmanifest.ts';

export const outdir = fileURLToPath(import.meta.resolve('../dist'));

build({
  bundle: true,
  entryPoints: [
    // …
    fileURLToPath(import.meta.resolve('../src/webmanifest/webmanifest.ts')),
    fileURLToPath(import.meta.resolve('../src/service-worker.ts')),
  ],
  format: 'esm',
  loader: {
    '.png': 'copy',
  },
  metafile: true,
  plugins: [
    // …
    compileWebmanifestPlugin(),
    compileIndexEJSPlugin(
      fileURLToPath(import.meta.resolve('../src/index.ejs')),
    ),
  ],
  outdir,
  splitting: true,
});

I'm thinking the better design/option is for the webmanifest to start as a js/ts file with imports, like

import type { WebAppManifest } from 'web-app-manifest';

import icon192 from './icons/192.png';
import icon512 from './icons/512.png';

export default {
  name: "…",
  // …
  icons: [
    {
      src: icon192,
      // …
    },
    // …
  ],
} satisfies WebAppManifest;

And then the plugin just needs to capture when esbuild tries to output the webmanifest file and redirect it (changing the output filename and use the default export as the json contents for the output file's contents). That way, the imported images etc get copied as normal and included in the metafile.

[hyrious]

Why though? I would expect esbuild to already do that since (at least in my case) App.svelte is an entry-point in the parent...

For simple es modules yes, but svelte actually compiles to different things on browser mode and on SSR mode. You can try that yourself on https://svelte.dev/playground and select the "JS output" panel on the right and toggle the option generate: "client" <-> "server" at below. It also requires esbuild to run in the platform: 'node' context, and it is not possible in esbuild to reuse the main build with different options.

[JakobJingleheimer]

Ah, okay. So I think that svelte plugin is not comparable to what I'm trying to do then?

I haven't been able to figure out how to get the contents esbuild compiles for the webmanifest.ts file so I can pluck the default export to provide as the contents for the output file.

Surely I don't need an extra build to get that contents—it's already an entry-point, so esbuild is already doing it.

On paper, this plugin should be extremely simple, like

onLoad({ filter: /webmanifest/ }, async (…) => ({
  contents: (await import(`data:text/javascript,${compiledContents}`)).default,
  path: filepath.replace(extname(filepath), '.json'),
}));

[hyrious]

I haven't been able to figure out how to get the contents esbuild compiles for the webmanifest.ts file so I can pluck the default export to provide as the contents for the output file.

I guess it is impossible to get something before it was being made.

And then the plugin just needs to capture when esbuild tries to output the webmanifest file and redirect it (changing the output filename and use the default export as the json contents for the output file's contents).

That reminds me of using the onEnd callback to modify the bundle result. It does work:

import * as esbuild from 'esbuild'

let result = await esbuild.build({
	entryPoints: [
		'src/index.js',
		'src/manifest.js',
	],
	bundle: true,
	format: 'esm',
	outdir: 'dist',
	write: false,
	plugins: [{
		name: 'manifest',
		setup({ initialOptions, onEnd }) {
			onEnd(async result => {
				// Will be undefined if write: true
				if (result.outputFiles) {
					let entry = result.outputFiles.find(e => e.path.endsWith('manifest.js'))
					let code = entry.text
					// https://2ality.com/2019/10/eval-via-import.html
					// Note: doesn't work if the code contains 'import.meta.url'
					let mod = await import(`data:text/javascript;charset=utf-8,${encodeURIComponent(code)}`)
					let manifest = JSON.stringify(mod.default, null, 2)
					entry.path = entry.path.replace('.js', '.json')
					entry.contents = new TextEncoder().encode(manifest)
				}
				// Modify the result on the disk
				if (initialOptions.write !== false) {
					// TODO: read & write changes to disk
				}
			})
		}
	}],
	logLevel: 'info'
}).catch(() => process.exit(1))

if (result.outputFiles) {
	for (let item of result.outputFiles) {
		console.log(item.path)
		console.log(item.text)
	}
} else {
	console.log('Changes already on disk.')
}

// --- Output ---

  dist/manifest.js  99b
  dist/index.js     32b

⚡ Done in 2ms
/path/to/dist/index.js
// src/index.js
console.log(1);

/path/to/dist/manifest.json
{
  "foo": 42
}

The only gotcha in this approach, is that if you do not enable write: false, the build output will be written to the disk directly before onEnd run. So in that case you will have to touch the file system as I was commented on the example code above.

[JakobJingleheimer]

Thanks! I've got it nearly working in stream-mode (after falling down a rabbit-hole from accidentally setting '.png': 'copy' instead of '.png': 'file' 🫠).

There is strangely a chunk import at the top of the file:

import "../chunk-JSBRDJBE.js";

// src/webmanifest/icons/192.png
var __default = "../192-BDXN7UWJ.png";

// src/webmanifest/icons/512.png
var __default2 = "../512-TDFJLFP5.png";

// src/webmanifest/webmanifest.ts
var webmanifest_default = {
  name: "Test",
  icons: [
    {
      src: __default,
      type: "impage/png",
      sizes: "192x192"
    },
    {
      src: __default2,
      type: "impage/png",
      sizes: "512x512"
    }
  ],
  start_url: "/",
  prefer_related_applications: false
};
export {
  webmanifest_default as default
};

The chunk contains some polyfills from esbuild to facilitate cjs ↔︎ esm (but this doesn't need/use them at all). That chunk import causes the dynamic import via data-url to blow up (trying to resolve that chunk).

---

For the write: false route: why is outputFiles absent when write: false? It seems like it would be extremely useful for plugins, and I can't think of any harm it would cause to surface it (esbuild surely has it already).

[hyrious]

The import "../chunk-JSBRDJBE.js"; is caused by splitting: true, which is to say, esbuild collects shared codes across chunks and put them together in a separate chunk. Unfortunately there's no way to control this behavior more precisely like other bundlers' "manual chunks".

My suggestion for now is to turn off the splitting feature and do code layering/chunking by yourself (example, build script). Although it might be not that straight forward for an existing project.

---

I assume the behavior of write: false is for the speed. Since esbuild doesn't need to run & wait plugins (it calls fs from the Go side).

[JakobJingleheimer]

Mm, I figured it was caused by code splitting. I was surprised to see the chunk there since it's not applicable to that file. That kinda seems like a bug in esbuild 🤔

Thanks, I'll try the manual way.

---

I'll do some digging, but I think esbuild always has that info, so there would be no speed difference. If it is always there (just not exposed) do you think a request to always expose it would be accepted?

And if it isn't, you think a request would be accepted to add a config option to always enable outputFiles?

[hyrious]

...but I think esbuild always has that info, so there would be no speed difference

I guess sending this info from the Go side to the JS side needs some time.

...do you think a request to always expose it would be accepted?

I vote for this request too.

[JakobJingleheimer]

I got it working 😀

Turns out I also have to manually update/fix metafile too 😞

I reported that cjs ↔ esm polyfill chunk bug.

I'll try to look into potential perf considerations and propose always exposing outputFiles.

Once that's sorted, I'll polish the plugin and publish it.

Thanks for all your help & insight @hyrious 🙏