'close' vs 'finish' events?

[d4hines]

Hello,
Thank you for making this library - it's really helpful to us. I'm having trouble understanding the difference between the 'close' and 'finish' events - could you explain the difference?

The reason I ask is that we're hitting an intermittent error where sometimes not all of the folder is extracted, causing our program to fail later on. Looking at this post on SO, they listened to the 'finish' event, whereas we listen to the 'close' event, and I'm wondering if that might be connected.

Thanks!

[blackmad]

I'm having the opposite problem - I'm not getting a complete extraction of a file when I listen on 'finish,' but I do when I listen on 'close'

[blackmad]

Here's a repro

this is the file:
https://www2.census.gov/geo/tiger/TIGER2016/ADDRFEAT/tl_2016_36005_addrfeat.zip

I wrote two little programs, one which extracts, waits on close and then exits, one which waits on finish - the extract in ~/Downloads is from using the built in mac unzip-on-doubleclick
https://gist.github.com/blackmad/a033c6ee245bfe9dba7683853f5731cf

• node test-close.js
• node test-finish.js
• md5sum /Users/blackmad/Downloads/tl_2016_36005_addrfeat/tl_2016_36005_addrfeat.shx
  1fe6e3946b391144adf64dd63d138276 /Users/blackmad/Downloads/tl_2016_36005_addrfeat/tl_2016_36005_addrfeat.shx
• md5sum ./tmp-close/tl_2016_36005_addrfeat.shx
  1fe6e3946b391144adf64dd63d138276 ./tmp-close/tl_2016_36005_addrfeat.shx
• md5sum ./tmp-finish/tl_2016_36005_addrfeat.shx
  943e48cccea536eb59b6d2c73684b838 ./tmp-finish/tl_2016_36005_addrfeat.shx

the one that listens/exits on close appears to get this file (the last one in the archive) correct, the one that listens on finish doesn't

[ZJONSSON]

Thanks you for the report @d4hines and @blackmad really appreciate the report and the repro in particular, nicely done!!

Worth noting that I get the same checksum on all files on v12.13.0 on ubuntu so this might be platform specific error.

So the only difference between finish and close is one tick:

node-unzipper/lib/extract.js

Lines 43 to 45 in 9e38162

	.on('finish',function() {
	extract.emit('close');
	});

But it appears that one tick is really needed. I think the fix might be to wrap the callbacks earlier within setImmediate to ensure all the i/o has truly flushed. This is still a little bit weird since the callbacks are only called when the writers emitted finished (might be problem with fstream)

[ZJONSSON]

@blackmad can you run the repro using the extract-setimmediate branch?

npm install zjonsson/node-unzipper#extract-setimmediate

[blackmad]

seemingly that doesn't fix it :-/ ... made a new directory just to doublecheck I was on the right version (base) ➜ /tmp mkdir fresh (base) ➜ /tmp cd fresh (base) ➜ fresh cp ~/RadarCode/interpolation/test* . (base) ➜ fresh mkdir -p tiger/downloads (base) ➜ fresh cp ~/RadarCode/interpolation/tiger/downloads/tl_2016_36005_addrfeat.zip tiger/downloads (base) ➜ fresh (base) ➜ interpolation git:(master) ✗ npm install zjonsson/node-unzipper#extract-setimmediate + unzipper@0.10.10 updated 1 package and audited 869 packages in 3.527s 19 packages are looking for funding run `npm fund` for details found 0 vulnerabilities (base) ➜ fresh sh test.sh + node test-close.js + node test-finish.js + md5sum /Users/blackmad/Downloads/tl_2016_36005_addrfeat/tl_2016_36005_addrfeat.shx 1fe6e3946b391144adf64dd63d138276 /Users/blackmad/Downloads/tl_2016_36005_addrfeat/tl_2016_36005_addrfeat.shx + md5sum ./tmp-close/tl_2016_36005_addrfeat.shx 1fe6e3946b391144adf64dd63d138276 ./tmp-close/tl_2016_36005_addrfeat.shx + md5sum ./tmp-finish/tl_2016_36005_addrfeat.shx 943e48cccea536eb59b6d2c73684b838 ./tmp-finish/tl_2016_36005_addrfeat.shx

…

On Tue, Mar 10, 2020 at 8:29 AM Ziggy Jonsson ***@***.***> wrote: @blackmad <https://github.com/blackmad> can you run the repro using the extract-setimmediate <#185> branch? npm install zjonsson/node-unzipper#extract-setimmediate — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165?email_source=notifications&email_token=AADMZMAQCA4OGQNNTNA6QYLRGYXDXA5CNFSM4KE2UEW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOLGUMI#issuecomment-597060145>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADMZMGMNFJVQ66OAWYGTHDRGYXDXANCNFSM4KE2UEWQ> .

-- *David Blackman* creative technologist & wandering help me find my purpose <http://purpose.blackmad.com>

[blackmad]

here are the various versions & platform numbers that might be useful to you (base) ➜ fresh uname -a Darwin nts2.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64 (base) ➜ fresh node --version v13.6.0 (base) ➜ fresh npm --version 6.13.4 (base) ➜ fresh nvm --version 0.35.2

…

On Tue, Mar 10, 2020 at 10:12 AM David Blackman ***@***.***> wrote: seemingly that doesn't fix it :-/ ... made a new directory just to doublecheck I was on the right version (base) ➜ /tmp mkdir fresh (base) ➜ /tmp cd fresh (base) ➜ fresh cp ~/RadarCode/interpolation/test* . (base) ➜ fresh mkdir -p tiger/downloads (base) ➜ fresh cp ~/RadarCode/interpolation/tiger/downloads/tl_2016_36005_addrfeat.zip tiger/downloads (base) ➜ fresh (base) ➜ interpolation git:(master) ✗ npm install zjonsson/node-unzipper#extract-setimmediate + ***@***.*** updated 1 package and audited 869 packages in 3.527s 19 packages are looking for funding run `npm fund` for details found 0 vulnerabilities (base) ➜ fresh sh test.sh + node test-close.js + node test-finish.js + md5sum /Users/blackmad/Downloads/tl_2016_36005_addrfeat/tl_2016_36005_addrfeat.shx 1fe6e3946b391144adf64dd63d138276 /Users/blackmad/Downloads/tl_2016_36005_addrfeat/tl_2016_36005_addrfeat.shx + md5sum ./tmp-close/tl_2016_36005_addrfeat.shx 1fe6e3946b391144adf64dd63d138276 ./tmp-close/tl_2016_36005_addrfeat.shx + md5sum ./tmp-finish/tl_2016_36005_addrfeat.shx 943e48cccea536eb59b6d2c73684b838 ./tmp-finish/tl_2016_36005_addrfeat.shx On Tue, Mar 10, 2020 at 8:29 AM Ziggy Jonsson ***@***.***> wrote: > @blackmad <https://github.com/blackmad> can you run the repro using the > extract-setimmediate <#185> > branch? > > npm install zjonsson/node-unzipper#extract-setimmediate > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#165?email_source=notifications&email_token=AADMZMAQCA4OGQNNTNA6QYLRGYXDXA5CNFSM4KE2UEW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOLGUMI#issuecomment-597060145>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AADMZMGMNFJVQ66OAWYGTHDRGYXDXANCNFSM4KE2UEWQ> > . > -- *David Blackman* creative technologist & wandering help me find my purpose <http://purpose.blackmad.com>

-- *David Blackman* creative technologist & wandering help me find my purpose <http://purpose.blackmad.com>