IndexOutOfBoundsException when running a Zip64 jar file larger than 4,294,967,295 bytes

[jfarjona]

When assembling a big jar file, SpringBoot breaks with a IndexOutOfBoundsException.

I have put the example that creates this bug at:

https://github.com/jfarjona/bugs-spring-boot-1

It breaks with different versions.

Exception details:

Exception in thread "main" java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.IndexOutOfBoundsException
        at org.springframework.boot.loader.PropertiesLauncher.<init>(PropertiesLauncher.java:158)
        at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:465)
Caused by: java.lang.IllegalStateException: java.lang.IndexOutOfBoundsException
        at org.springframework.boot.loader.PropertiesLauncher.getHomeDirectory(PropertiesLauncher.java:167)
        at org.springframework.boot.loader.PropertiesLauncher.<init>(PropertiesLauncher.java:152)
        ... 1 more
Caused by: java.lang.IndexOutOfBoundsException
        at org.springframework.boot.loader.jar.AsciiBytes.<init>(AsciiBytes.java:73)
        at org.springframework.boot.loader.jar.CentralDirectoryFileHeader.load(CentralDirectoryFileHeader.java:96)
        at org.springframework.boot.loader.jar.CentralDirectoryParser.parseEntries(CentralDirectoryParser.java:68)
        at org.springframework.boot.loader.jar.CentralDirectoryParser.parse(CentralDirectoryParser.java:57)
        at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:137)
        at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:123)
        at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:109)
        at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:100)
        at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:73)
        at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:69)
        at org.springframework.boot.loader.Launcher.createArchive(Launcher.java:163)
        at org.springframework.boot.loader.PropertiesLauncher.getProperty(PropertiesLauncher.java:442)
        at org.springframework.boot.loader.PropertiesLauncher.getPropertyWithDefault(PropertiesLauncher.java:403)
        at org.springframework.boot.loader.PropertiesLauncher.getHomeDirectory(PropertiesLauncher.java:164)
        ... 2 more

Juan

[philwebb]

The sample application cannot be built on OSX but I did manage to replicate the problem on Linux. The jar file in question is 8.4G 😱

[philwebb]

I think the problem is out zip64 detection logic. We currently check to see if the "number of records" entry contains 0xFFFF. With the jar from your sample we have an EOCD that looks like this:

504B0506 signature
0000     number of disk
0000     disk where cd starts
CA01     number of records
CA01     total num of rec
D3AE0000 size of CD
FFFFFFFF offset of start of central directory
0000     comment length

The number of records in this file is 0xCA01 not 0xFFFF. Only the offset has a 0xFFFFFFFF value.

Looking at https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT I think this is valid:

The number of this disk, which contains central directory end record. If an archive is in ZIP64 format and the value in this field is 0xFFFF, the size will be in the corresponding 4 byte zip64 end of central directory field.

The important part being "if ... the value in this field is 0xFFFF". I think this means that it doesn't need to be 0xFFFF, but it can be 0xFFFF.

I think we need to refine CentralDirectoryEndRecord.isZip64.

[wilkinsona]

With a fix for the Zip64-detection in place, another failure occurs:

Exception in thread "main" java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.IndexOutOfBoundsException
	at org.springframework.boot.loader.PropertiesLauncher.<init>(PropertiesLauncher.java:158)
	at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:465)
Caused by: java.lang.IllegalStateException: java.lang.IndexOutOfBoundsException
	at org.springframework.boot.loader.PropertiesLauncher.getHomeDirectory(PropertiesLauncher.java:167)
	at org.springframework.boot.loader.PropertiesLauncher.<init>(PropertiesLauncher.java:152)
	... 1 more
Caused by: java.lang.IndexOutOfBoundsException
	at org.springframework.boot.loader.data.RandomAccessDataFile.getSubsection(RandomAccessDataFile.java:83)
	at org.springframework.boot.loader.jar.CentralDirectoryEndRecord$Zip64End.getCentralDirectory(CentralDirectoryEndRecord.java:188)
	at org.springframework.boot.loader.jar.CentralDirectoryEndRecord$Zip64End.access$300(CentralDirectoryEndRecord.java:149)
	at org.springframework.boot.loader.jar.CentralDirectoryEndRecord.getCentralDirectory(CentralDirectoryEndRecord.java:118)
	at org.springframework.boot.loader.jar.CentralDirectoryParser.parse(CentralDirectoryParser.java:55)
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:139)
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:123)
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:109)
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:100)
	at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:73)
	at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:69)
	at org.springframework.boot.loader.Launcher.createArchive(Launcher.java:163)
	at org.springframework.boot.loader.PropertiesLauncher.getProperty(PropertiesLauncher.java:442)
	at org.springframework.boot.loader.PropertiesLauncher.getPropertyWithDefault(PropertiesLauncher.java:403)
	at org.springframework.boot.loader.PropertiesLauncher.getHomeDirectory(PropertiesLauncher.java:164)
	... 2 more

[wilkinsona]

And another one after that:

Exception in thread "main" java.lang.IllegalStateException: Failed to get nested archive for entry BOOT-INF/lib/cuda-11.2-8.1-1.5.5-windows-x86_64-redist.jar
	at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchive(JarFileArchive.java:120)
	at org.springframework.boot.loader.archive.JarFileArchive$NestedArchiveIterator.adapt(JarFileArchive.java:274)
	at org.springframework.boot.loader.archive.JarFileArchive$NestedArchiveIterator.adapt(JarFileArchive.java:265)
	at org.springframework.boot.loader.archive.JarFileArchive$AbstractIterator.next(JarFileArchive.java:226)
	at org.springframework.boot.loader.PropertiesLauncher$ClassPathArchives.addNestedEntries(PropertiesLauncher.java:651)
	at org.springframework.boot.loader.PropertiesLauncher$ClassPathArchives.<init>(PropertiesLauncher.java:542)
	at org.springframework.boot.loader.PropertiesLauncher.getClassPathArchivesIterator(PropertiesLauncher.java:458)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:55)
	at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)
Caused by: java.io.IOException: Unable to open nested jar file 'BOOT-INF/lib/cuda-11.2-8.1-1.5.5-windows-x86_64-redist.jar'
	at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:295)
	at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:281)
	at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchive(JarFileArchive.java:116)
	... 8 more
Caused by: java.io.IOException: Unable to find ZIP central directory records after reading 65792 bytes
	at org.springframework.boot.loader.jar.CentralDirectoryEndRecord.<init>(CentralDirectoryEndRecord.java:68)
	at org.springframework.boot.loader.jar.CentralDirectoryParser.parse(CentralDirectoryParser.java:51)
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:139)
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:123)
	at org.springframework.boot.loader.jar.JarFile.createJarFileFromFileEntry(JarFile.java:326)
	at org.springframework.boot.loader.jar.JarFile.createJarFileFromEntry(JarFile.java:303)
	at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:292)
	... 10 more

[wilkinsona]

I have a fix for this. I'm a little bit concerned about the increased memory footprint due to JarFileEntries.centralDirectoryOffsets changing from int[] to long[]. That'll consume in additional 4 bytes per entry (top-level and nested).