Use fs.Dirent objects, and extract metadata in the background #3
Description
Currently, the app freeze while performing the first phase of the import (listing all the files recursively).
A large portion of this delay in phase 1 is due to stating all the files to see if they are directories. Alternatively using fs.Dirent objects, the following code would be much better:
// Recursively walk through a directory compiling ImageElements
const walkSync = (dir, filelist) => {
const files = fs.readdirSync(dir, {encoding: 'utf8', withFileTypes: true});
files.forEach(function (file) {
if (file.name) {
if (!fileSystemReserved(file.name)) {
// if the item is a _DIRECTORY_
if (file.isDirectory()) {
filelist = walkSync(path.join(dir, file.name), filelist);
} else {
const extension = file.name.split('.').pop();
if (acceptableFiles.includes(extension.toLowerCase())) {
// before adding, remove the redundant prefix: sourceFolderPath
const partialPath = dir.replace(sourceFolderPath, '');
// fil finalArray with 3 correct and 5 dummy pieces of data
finalArray[elementIndex] = [partialPath, file.name, cleanUpFileName(file.name), '', 0, '', 0, 0];
elementIndex++;
}
}
}
}
});
See https://nodejs.org/api/fs.html#fs_fs_readdirsync_path_options
This will be a drastic improvement in speed, especially over networked file systems. Unfortunately, this is only added in node v10.10.0, which will need to be updated first.
Additionally, it would be much better if the second phase (metadata extraction) was performed in the background like phase 3 (thumbnail generation). This is especially important with the addition of file hashes, which will require reading every file.