EMFILE "too many files open" on Windows due to un‑awaited archiver streams (descriptor leak & uncontrolled concurrency)

[joroaf]

vropkg: EMFILE "too many files open" on Windows due to un-awaited archiver streams (descriptor leak & uncontrolled concurrency)

Summary

Running vropkg on Windows for packages with many elements fails with:

Error: EMFILE: too many open files

Archive write streams are created and finalized without awaiting their completion. High parallelism (Promise.all across elements) causes many simultaneous open file descriptors until the OS limit is hit (Windows defaults are low). Also introduces a race where signing may read files before archives are flushed.

Affected Code

• typescript/vropkg/src/serialize/flat.ts
  • initializeContext.bundle returns the result of .finalize() (void) but is typed/used like a Promise<void>.
  • serializeFlatElements launches many async tasks per element concurrently.
• typescript/vropkg/src/serialize/util.ts
  • zipbundle starts an archiver pipeline and returns immediately; no Promise on stream close.
• typescript/vropkg/src/packaging.ts
  • archive() creates and pipes an archiver but callers don't wait for the destination stream to finish.

Impact

• Frequent failure on Windows with medium/large packages (e.g. ≥512 elements).
• Potential race: signatures may be generated before archive contents are fully written.

Environment (example)

• OS: Windows 10 / Windows Server 2019
• Node.js: 18.x / 20.x / 22.x
• Branch: main (as of 2025-08-27)

Steps to Reproduce

1. Prepare a project producing a vRO package with 100+ elements (some with bundles/resources).
2. Run vropkg to build the package.
3. Observe eventual failure with EMFILE.

Expected

Packaging completes; file descriptors are reused; signatures generated after all archive writes complete.

Actual

Multiple archives in progress simultaneously; write streams not awaited; EMFILE thrown.

Root Cause

archiver.finalize() is not asynchronous (returns void). Code assumes it can be awaited. Without listening to finish/close on the output stream, promises resolve too early (or are not real promises), allowing unbounded concurrency and exhausting file handles.

Proposed Fix

1. Wrap each archiving operation in a real Promise resolving on output stream close (or finish) and rejecting on error.
2. Correct return types (ensure functions actually return Promise<void>).
3. Await archive promises (e.g., await context.save(...)).
4. Add simple concurrency control for element serialization (e.g., p-limit with a reasonable cap or process elements sequentially).
5. Ensure signature generation runs only after all archive writes have completed.
6. Add an integration test packaging a large synthetic set of elements to assert success without EMFILE.

Patch Sketch (conceptual)

// In serialize/util.ts
export const zipbundle = (target: string) => {
  fs.mkdirsSync(target);
  return (file: string) => (sourcePath: string, isDir: boolean): Promise<void> => {
    const absoluteZipPath = path.join(target, file);
    if (!isDir) {
      fs.copySync(sourcePath, absoluteZipPath);
      return Promise.resolve();
    }
    return new Promise((resolve, reject) => {
      const output = fs.createWriteStream(absoluteZipPath);
      const archive = archiver('zip', { zlib: { level: 9 }});
      output.on('close', resolve).on('error', reject);
      archive.on('error', reject);
      archive.directory(sourcePath, false);
      archive.pipe(output);
      archive.finalize();
    });
  };
};

(Apply similar pattern for final package bundle and resource element bundles.)

Workarounds

• Reduce elements per run (split package).
• Increase Windows file handle limit (not always feasible).
• Run on Linux/macOS (higher default limits) — mitigates but doesn’t fix root cause.

Acceptance Criteria

• Large package builds on Windows without EMFILE.
• Signatures are created only after archive completion.
• No regression in output structure.

Additional Notes

Fix is low risk and localized to I/O lifecycle management; may slightly reduce parallel throughput but improves stability and determinism.

---

Attached Log:

[INFO] info: Using certificate file target/keystore-1.0.4/private_key.pem
[INFO] info: Using certificate file target/keystore-1.0.4/cert.pem
[ERROR] node:fs:562
[ERROR]   return binding.open(
[ERROR]                  ^
[ERROR]
[ERROR] Error: EMFILE: too many open files, open 'C:\Users\txdcy\Documents\projects\di\workflows\target\vropkg\elements\758fd262-5974-497d-9a6a-4a4223697fc4\content-signature'
[ERROR]     at Object.openSync (node:fs:562:18)
[ERROR]     at Object.writeFileSync (node:fs:2440:35)
[ERROR]     at C:\Users\txdcy\Documents\projects\di\workflows\node_modules\@vmware-pscoe\vropkg\dist\serialize\util.js:39:12
[ERROR]     at Generator.next (<anonymous>)
[ERROR]     at C:\Users\txdcy\Documents\projects\di\workflows\node_modules\@vmware-pscoe\vropkg\dist\serialize\util.js:8:71
[ERROR]     at new Promise (<anonymous>)
[ERROR]     at __awaiter (C:\Users\txdcy\Documents\projects\di\workflows\node_modules\@vmware-pscoe\vropkg\dist\serialize\util.js:4:12)
[ERROR]     at Object.contentSignature (C:\Users\txdcy\Documents\projects\di\workflows\node_modules\@vmware-pscoe\vropkg\dist\serialize\util.js:35:51)
[ERROR]     at C:\Users\txdcy\Documents\projects\di\workflows\node_modules\@vmware-pscoe\vropkg\dist\serialize\flat.js:220:20
[ERROR]     at Generator.next (<anonymous>)
[ERROR]     at fulfilled (C:\Users\txdcy\Documents\projects\di\workflows\node_modules\@vmware-pscoe\vropkg\dist\serialize\flat.js:5:58) {
[ERROR]   errno: -4066,
[ERROR]   code: 'EMFILE',
[ERROR]   syscall: 'open',
[ERROR]   path: 'C:\\Users\\txdcy\\Documents\\projects\\di\\workflows\\target\\vropkg\\elements\\758fd262-5974-497d-9a6a-4a4223697fc4\\content-signature'
[ERROR] }
[ERROR]
[ERROR] Node.js v22.16.0
[INFO] Running vropkg... finished

[vgudzhev]

const serializeFlatSignatures = (context, pkg) => __awaiter(void 0, void 0, void 0, function* () {
const target = context.target;
const files = glob.sync((target + "/**/*").replace(/[/]+/gm, path.posix.sep), { nodir: true });
const results = [];
for (const file of files) {
const location = path.normalize(file).replace(target, "");
const fileData = yield fs.promises.readFile(file);
const data = s.sign(fileData, pkg.certificate);
const signature = yield context.signatures(location)(data);
results.push(signature);
}
return results;
});

[vgudzhev]

const zipbundle = (target) => {
fs.mkdirsSync(target);
return (file) => (sourcePath, isDir) => __awaiter(void 0, void 0, void 0, function* () {
const absoluteZipPath = path.join(target, file);
if (isDir) {
return new Promise((resolve, reject) => {
const output = fs.createWriteStream(absoluteZipPath);
const archive = archiver('zip', { zlib: { level: 9 } });
output.on('close', resolve);
output.on('error', reject);
archive.on('error', reject);
archive.directory(sourcePath, false);
archive.pipe(output);
archive.finalize();
});
}
else {
fs.copySync(sourcePath, absoluteZipPath);
return Promise.resolve();
}
});
};

[vgudzhev]

const serialize = (target) => {
fs.mkdirsSync(target);
return (file) => (data, options, fileName) => __awaiter(void 0, void 0, void 0, function* () {
const absolutePath = path.join(target, (fileName || file));
fs.mkdirsSync(path.dirname(absolutePath));
(0, logger_1.default)().debug(Writing ${absolutePath});
return new Promise((resolve, reject) => {
fs.writeFile(absolutePath, data, options || {}, (err) => {
if (err)
reject(err);
else
resolve();
});
});
});
};

[vgudzhev]

"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
return new (P || (P = Promise))(function (resolve, reject) {
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
function rejected(value) { try { step(generator"throw"); } catch (e) { reject(e); } }
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
step((generator = generator.apply(thisArg, _arguments || [])).next());
});
};
Object.defineProperty(exports, "_esModule", { value: true });
exports.serializeFlat = void 0;
const fs = require("fs-extra");
const glob = require("glob");
const path = require("path");
const logger_1 = require("../logger");
const t = require("../types");
const xmlbuilder = require("xmlbuilder");
const uuid_1 = require("uuid");
const s = require("../security");
const p = require("../packaging");
const util_1 = require("../util");
const util_2 = require("./util");
function queue(concurrency, tasks) {
return new Promise((resolve, reject) => {
const results = [];
let running = 0;
let index = 0;
let finished = 0;
const total = tasks.length;
function runNext() {
if (finished === total) {
resolve(results);
return;
}
while (running < concurrency && index < total) {
const current = index;
running++;
taskscurrent.then(result => {
results[current] = result;
running--;
finished++;
runNext();
}).catch(err => {
reject(err);
});
index++;
}
}
runNext();
});
}
const encoding_1 = require("../encoding");
const xmlDoc = require("xmldoc");
const constants_1 = require("../constants");
const DUNES_META_INF = "dunes-meta-inf", CERTIFICATES = "certificates", ELEMENTS = "elements", CATEGORIES = "categories", CONTENT_SIGNATURE = "content-signature", DATA = "data", BUNDLE = "bundle", DATA_ALLOWEDOPERATIONS = "attribute_allowedOperations", DATA_DESCRIPTION = "attribute_description", DATA_ID = "attribute_id", DATA_MIMETYPE = "attribute_mimetype", DATA_NAME = "attribute_name", DATA_VERSION = "attribute_version", INFO = "info", TAGS = "tags", SIGNATURES = "signatures", INPUT_FORM = "input_form", INPUT_FORM_ITEM = "input_form_item";
const initializeContext = (target) => {
fs.pathExistsSync(target) && fs.removeSync(target);
const bundle = (name) => {
const arch = p.archive(path.join(target, name));
[CERTIFICATES, ELEMENTS, SIGNATURES].forEach(folder => arch.directory(path.join(target, folder), folder));
return arch.append(fs.createReadStream(path.join(target, DUNES_META_INF)), { name: DUNES_META_INF }).finalize();
};
const store = (0, util_2.serialize)(target);
return {
target: target,
dunesMetaInf: store(DUNES_META_INF),
certificates: (subjectName) => store(path.join(CERTIFICATES, subjectName)),
elements: (elementId) => initializeElementContext(path.join(target, ELEMENTS, elementId)),
signatures: (signature) => store(path.join(SIGNATURES, signature)),
save: bundle
};
};
const initializeElementContext = (target) => {
const store = (0, util_2.serialize)(target);
const storezipped = (0, util_2.zipbundle)(target);
return {
target,
categories: store(CATEGORIES),
contentSignature: store(CONTENT_SIGNATURE),
data: store(DATA),
bundle: storezipped(BUNDLE),
dataComplex: serializeFlatElementData(path.join(target, DATA)),
info: store(INFO),
tags: store(TAGS),
form: store(INPUT_FORM),
formItems: store(INPUT_FORM_ITEM)
};
};
const serializeFlatElementData = (target) => {
let bundle;
const initBundle = () => {
bundle = p.archive(target);
};
const append = (name) => (data) => {
if (data) {
bundle.append(Buffer.from(data, 'utf8'), { name: ${constants_1.VSO_RESOURCE_INF}/${name} });
}
else {
(0, logger_1.default)().debug(Element not available ${constants_1.VSO_RESOURCE_INF}/${name});
}
};
return {
target: target,
initBundle: initBundle,
allowedOperations: append(DATA_ALLOWEDOPERATIONS),
description: append(DATA_DESCRIPTION),
id: append(DATA_ID),
mimetype: append(DATA_MIMETYPE),
name: append(DATA_NAME),
version: append(DATA_VERSION),
data: append(DATA),
save: () => bundle.finalize()
};
};
const serializeFlatCertificate = (context, pkg) => {
const future = [];
pkg.certificate.chain.forEach((pem, subject) => {
future.push(context.certificates(${subject}.cer)(s.pemToDer(pem), { encoding: "binary" }));
});
return future;
};
const serializeFlatDunesMetadata = (context, pkg) => __awaiter(void 0, void 0, void 0, function* () {
let node = xmlbuilder.create("properties", util_2.xmlOptions);
node.dtd("", "http://java.sun.com/dtd/properties.dtd");
node.ele("comment").text("UTF-16").up()
.ele("entry").att("key", "pkg-id").text((0, uuid_1.v5)('http://' + (0, util_2.getPackageName)(pkg), uuid_1.v5.URL)).up()
.ele("entry").att("key", "pkg-name").text([pkg.groupId, pkg.artifactId + "-" + pkg.version].join(".")).up()
.ele("entry").att("key", "pkg-description").text(pkg.description || "Built by Build Tools for VMware Aria").up()
.ele("entry").att("key", "pkg-signer").text(pkg.certificate.subject).up()
.ele("entry").att("key", "pkg-owner").text(pkg.certificate.subject);
return context.dunesMetaInf(node.end(util_2.saveOptions));
});
const serializeFlatElements = (context, pkg) => __awaiter(void 0, void 0, void 0, function* () {
const concurrency = 8;
return queue(concurrency, pkg.elements.map(element => () => __awaiter(void 0, void 0, void 0, function* () {
const elementContext = context.elements(element.id);
const inputFormItems = yield serializeFlatElementInputFormItems(elementContext, element);
yield Promise.all([
serializeFlatElementInfo(elementContext, pkg, element),
serializeFlatElementCategory(elementContext, element),
serializeFlatElementContent(elementContext, pkg, element),
serializeFlatElementBundle(elementContext, element),
serializeFlatElementTags(elementContext, element),
serializeFlatElementInputForm(elementContext, element),
...(Array.isArray(inputFormItems) ? inputFormItems : [])
]);
return exportPackageElementContentSignature(elementContext, pkg);
})));
});
const serializeFlatElementInfo = (context, pkg, element) => __awaiter(void 0, void 0, void 0, function* () {
let node = xmlbuilder.create("properties", util_2.infoOptions);
node.dtd("", "http://java.sun.com/dtd/properties.dtd");
node.ele("comment").text("Exported from PSCoE o11n-convert").up()
.ele("entry").att("key", "type").text(element.type.toString()).up()
.ele("entry").att("key", "signature-owner").text(pkg.certificate.subject).up()
.ele("entry").att("key", "id").text(element.id);
return context.info(node.end(util_2.saveOptions));
});
const serializeFlatElementInputForm = (context, element) => __awaiter(void 0, void 0, void 0, function* () {
var _a, _b;
if ((_a = element.form) === null || _a === void 0 ? void 0 : _a.data) {
const buffer = Buffer.from('\ufeff' + JSON.stringify((_b = element.form) === null || _b === void 0 ? void 0 : _b.data), "utf16le").swap16();
return context.form(buffer, constants_1.DEFAULT_ENCODING);
}
});
const serializeFlatElementInputFormItems = (context, element) => __awaiter(void 0, void 0, void 0, function* () {
const promises = [];
if (element.formItems && Array.isArray(element.formItems)) {
element.formItems.forEach((formItem) => {
const buffer = Buffer.from('\ufeff' + JSON.stringify(formItem.data), "utf16le").swap16();
const formName = formItem.name || "";
const fileName = constants_1.FORM_ITEM_TEMPLATE.replace("{{formName}}", formName);
const promise = context.formItems(buffer, constants_1.DEFAULT_ENCODING, fileName);
promises.push(promise);
});
}
return promises;
});
const serializeFlatElementCategory = (context, element) => __awaiter(void 0, void 0, void 0, function* () {
const categories = xmlbuilder.begin().ele(CATEGORIES);
(element.type == t.VroElementType.ScriptModule
? [element.categoryPath.join(".")]
: element.categoryPath).forEach((path) => {
const realName = path.replace(///g, "");
categories.ele("category").att("name", realName).ele("name").dat(realName);
});
return context.categories(Buffer.from('\ufeff' + categories.end(util_2.saveOptions), "utf16le").swap16(), constants_1.DEFAULT_ENCODING);
});
const serializeFlatElementBundle = (context, element) => __awaiter(void 0, void 0, void 0, function* () {
var _a;
if (!((_a = element === null || element === void 0 ? void 0 : element.action) === null || _a === void 0 ? void 0 : _a.bundle)) {
return;
}
if (!(0, util_1.exist)(element.action.bundle.contentPath)) {
throw new Error(Source script bundle "${element.action.bundle.contentPath}" does not exist for element
+ ${element.name}" of type "${element.type}", category "${element.categoryPath}", id: "${element.id}".
+ Cannot save script bundle.);
}
let isDir = (0, util_1.isDirectory)(element.action.bundle.contentPath);
return context.bundle(element.action.bundle.contentPath, isDir);
});
const serializeFlatElementTags = (context, element) => __awaiter(void 0, void 0, void 0, function* () {
var _a;
if (!((_a = element.tags) === null || _a === void 0 ? void 0 : _a.length)) {
(0, logger_1.default)().debug(Element does not have tags ${element.name});
return;
}
let node = xmlbuilder.create("tags", util_2.xmlOptions);
element.tags.forEach(name => {
node.ele("tag").att("name", name).att("global", true).up();
});
const contentBuffer = Buffer.from('\ufeff' + node.end(util_2.saveOptions), "utf16le").swap16();
return context.tags(contentBuffer, constants_1.DEFAULT_ENCODING);
});
const serializeFlatElementContent = (context, pkg, element) => _awaiter(void 0, void 0, void 0, function* () {
const pkgVersion = pkg.version.split("-SNAPSHOT")[0];
let content = null;
switch (element.type) {
case t.VroElementType.ResourceElement: {
const data = context.dataComplex;
const value = (name) => element.attributes[name.replace('attribute', '')];
data.initBundle();
data.id(value(DATA_ID));
data.name(value(DATA_NAME));
data.version(pkgVersion);
data.description(value(DATA_DESCRIPTION));
data.mimetype(value(DATA_MIMETYPE));
data.allowedOperations(value(DATA_ALLOWEDOPERATIONS));
data.data(Buffer.from(fs.readFileSync(element.dataFilePath)));
return data.save();
}
case t.VroElementType.ScriptModule: {
content = (0, util_2.getActionXml)(element.id, element.name, element.description, element.action);
break;
}
case t.VroElementType.ActionEnvironment: {
return context.data(fs.readFileSync(element.dataFilePath));
}
default: {
content = (0, encoding_1.decode)(fs.readFileSync(element.dataFilePath));
}
}
let xmlContent = new xmlDoc.XmlDocument(content);
xmlContent.attr["version"] = pkgVersion;
const contentBuffer = Buffer.from('\ufeff' + xmlContent.toString({ compressed: true }), "utf16le").swap16();
return context.data(contentBuffer, constants_1.DEFAULT_ENCODING);
});
const exportPackageElementContentSignature = (context, pkg) => __awaiter(void 0, void 0, void 0, function* () {
let data = yield fs.readFile(path.join(context.target, DATA));
return context.contentSignature(s.sign(data, pkg.certificate));
});
const serializeFlatSignatures = (context, pkg) => __awaiter(void 0, void 0, void 0, function* () {
const promises = [];
const target = context.target;
glob.sync((target + "/**/").replace(/[\/]+/gm, path.posix.sep), { nodir: true }).forEach(file => {
const location = path.normalize(file).replace(target, "");
const data = s.sign(fs.readFileSync(file), pkg.certificate);
const signature = context.signatures(location)(data);
promises.push(signature);
});
return promises;
});
const serializeFlat = (pkg, targetPath) => __awaiter(void 0, void 0, void 0, function () {
const context = initializeContext(targetPath);
yield Promise.all([
...serializeFlatCertificate(context, pkg),
serializeFlatDunesMetadata(context, pkg)
]);
yield serializeFlatElements(context, pkg);
yield serializeFlatSignatures(context, pkg);
return context.save((0, util_2.getPackageName)(pkg));
});
exports.serializeFlat = serializeFlat;

[vgudzhev]

// index.js
const ffi = require('ffi-napi');

// Load the MSVC runtime library (Windows only)
const msvcrt = ffi.Library('msvcrt', {
'_getmaxstdio': ['int', []]
});

// Call the function and display the result
const maxStdio = msvcrt._getmaxstdio();
console.log("Maximum number of simultaneously open files (stdio-level):", maxStdio);

[vgudzhev]

const runWithConcurrency = (tasks, max) => __awaiter(void 0, void 0, void 0, function* () {
const results = [];
let i = 0;
const executing = [];
function runNext() {
return __awaiter(this, void 0, void 0, function* () {
if (i >= tasks.length)
return;
const current = i++;
const p = taskscurrent.then(r => { results[current] = r; });
executing.push(p);
p.then(() => executing.splice(executing.indexOf(p), 1));
if (executing.length >= max) {
yield Promise.race(executing);
}
yield runNext();
});
}
yield Promise.all(Array(Math.min(max, tasks.length)).fill(0).map(runNext));
return results;
});

const serializeFlatSignatures = (context, pkg) => __awaiter(void 0, void 0, void 0, function* () {
const target = context.target;
const files = glob.sync((target + "/**/").replace(/[/]+/gm, path.posix.sep), { nodir: true });
const tasks = files.map(file => () => __awaiter(void 0, void 0, void 0, function () {
const location = path.normalize(file).replace(target, "");
const data = s.sign(fs.readFileSync(file), pkg.certificate);
return context.signatures(location)(data);
}));
return runWithConcurrency(tasks, MAX_CONCURRENT_ELEMENTS);
});

const serializeFlatElements = (context, pkg) => {
return pkg.elements.map((element) => __awaiter(void 0, void 0, void 0, function* () {
const elementContext = context.elements(element.id);
yield Promise.all([
serializeFlatElementInfo(elementContext, pkg, element),
serializeFlatElementCategory(elementContext, element),
serializeFlatElementContent(elementContext, pkg, element),
serializeFlatElementBundle(elementContext, element),
serializeFlatElementTags(elementContext, element),
serializeFlatElementInputForm(elementContext, element),
serializeFlatElementInputFormItems(elementContext, element)
]);
return exportPackageElementContentSignature(elementContext, pkg);
}));
};