Remove abstractions of MMapDirectory and its "fake" MR-JAR support and move it to main sourceSet (vectors untouched)

[uschindler]

This PR moves all classes of MMapDirectory and the MemorySegment supporting classes from the java21 sourceset to the main sourceset.

It also updates the MR-JAR sourceset for vectorization to the Java 23 API extraction and removes all stub classes from the APIJAR which refer to memory segments, so only the vector API is extracted and saved in the API-JAR. This change makes merging vector code a bit harder as the file paths changes, but at some point we have to live with this.

The MMapDirectory abstractions (attachments, provider classes,..) were removed from code and all implementations are statically initialized by MMapDirectory (native access). The MemorySegmentIndexInput is now a 1st class citizen.

We may remove further abstractions in future and maybe also move other code away from ByteBuffer to MemorySegment to allow mapping vectors and directly access index parts also from other directories (NIOFSDir may also return vector views)...

[uschindler]

This PR updates the ASM dependcy to 9.8 because its needed for the APIJAR extraction. Due to our locking code, the expressions module therefore also updates.

But after a discussion with Robert, I am already working on a move away from ASM for expressions with Java 24. The ASM is then no longer a dependency of Lucene artifacts.

[rmuir]

looks good to me, this is a good step.

[uschindler]

I also removed some workaround code for Java 21 that prevented thread safe status check on closed arena to propagate AlreadyClosedException: 9925e5e

P.S.: with this check removed we ensure that the JDK 23+ version is correct! If we see errors in tests that close open files we have to reopen issues @ JDK.

[uschindler]

I applied some further code cleanups to get rid of more abstractions in private methods. I also added changes entry.

I think it's ready.

[uschindler]

I updated the code to Java 24 (generation of APIJAR).

I also tested vectorization, seems to compile and work:

> Task :lucene:core:test
WARNING: Using incubator modules: jdk.incubator.vector
  2> Apr. 29, 2025 9:47:35 PM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider <init>
  2> INFORMATION: Java vector incubator API enabled; uses preferredBitSize=512
  2> Apr. 29, 2025 9:47:35 PM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider <init>
  2> INFORMATION: Java vector incubator API enabled; uses preferredBitSize=512
  1> filesystem: ExtrasFS(HandleLimitFS(LeakFS(ShuffleFS(DisableFsyncFS(VerboseFS(sun.nio.fs.WindowsFileSystemProvider@292241db))))))
  1> FS 0 [2025-04-29T19:47:35.248226100Z; SUITE-TestVectorUtil-seed#[B55333409746887B]-worker]: createDirectory: ..\tests-tmp (FAILED: java.nio.file.FileAlreadyExistsException: C:\Users\Uwe Schindler\Projects\lucene\lucene\lucene\core\build\tmp\tests-tmp)
  2> Apr. 29, 2025 9:47:35 PM org.apache.lucene.internal.vectorization.VectorizationProvider lookup
  2> WARNUNG: Vector bitsize and/or integer vectors enforcement; using default vectorization provider outside of testMode
  1> Loaded codecs: [MinimalCodec, MinimalCompoundCodec, Lucene103, Asserting, CheapBastard, DeflateWithPresetCompressingStoredFieldsData, FastCompressingStoredFieldsData, FastDecompressionCompressingStoredFieldsData, HighCompressionCompressingStoredFieldsData, LZ4WithPresetCompressingStoredFieldsData, DummyCompressingStoredFieldsData, ConfigurableMCodec, SimpleText]
  1> Loaded postingsFormats: [Lucene103, MockRandom, RAMOnly, LuceneFixedGap, LuceneVarGapFixedInterval, LuceneVarGapDocFreqInterval, TestBloomFilteredLucenePostings, Asserting, UniformSplitRot13, STUniformSplitRot13, BlockTreeOrds, BloomFilter, Direct, FST50, UniformSplit, SharedTermsUniformSplit]
  1> NOTE: Created shared ExecutorService with 1 threads
  2> NOTE: test params are: codec=Asserting(Lucene103): {}, knn_vectors:{}, docValues:{}, maxPointsInLeafNode=1005, maxMBSortInHeap=6.190502464272589, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=xog, timezone=America/Martinique
  2> NOTE: Windows 11 10.0 amd64/Oracle Corporation 24 (64-bit)/cpus=1,threads=1,free=278614240,total=324534272
  2> NOTE: All tests run in this JVM: [TestVectorUtil]
:lucene:core:test (SUCCESS): 30 test(s)

[uschindler]

I added a small change to the JAR task so the MR-JAR part is only build if the target version is > minimum version. This was basically a bug in the previous main branch: it added a Java 21 MRJAR part. I backported the last commit to 10.x, although the issue won't appear there.

[uschindler]

Damn: "Create file open hints on IOContext to replace ReadAdvice (#14482)" was merged yesterday..... Trying to figure out what's going on. @thecoop

[ChrisHegarty]

Thanks @uschindler - LGTM

[uschindler]

I am still verifyying that the conflicting changes due to readAdvice are merged in here in a correct way.

I will merge this very soon tro avoid conflicts like this. My head is burning....

[thecoop]

The changes around ReadAdvice look ok to me

[uschindler]

Thanks @thecoop. About the other PR: It was only for main branch, right? So it won't be backported, so all changes to readAdvice will be a feature for Lucene 11. To me those changes look to invasive for a minor release.

I am just asking this because due to the changes here merging of MemorySegmentIndexInput and Native access code will be more complicated du to the new location of files and removed abstractions. If the code is only developed for main branch it's not a problem.

I would hope that Directory and MMapDirectoy are in "feature freeze" for 10.x.

EDIT: DAMN it was backported. Happy merging in future!