Fix INT8 and BINARY vector quantization offset bug in LSMVectorIndex page loading

[Copilot]

What does this PR do?

Fixes the "Variable length (70) quantity is too long" error that made both INT8 and BINARY vector quantization completely unusable. The bug was in loadVectorsFromFile() which failed to skip quantized vector data when reading entry metadata from pages, causing misaligned offsets that interpreted vector bytes as VInt-encoded numbers.

Motivation

INT8 and BINARY quantization were broken across all dimensions (4-1536), failing with either:

• IndexOutOfBoundsException with negative indices (dims < 16)
• IllegalArgumentException: Variable length quantity is too long (dims ≥ 16)
• Data loss and 0% search recall

This blocked adoption of quantization for production embeddings.

Additional Notes

Changes:

• Core fix: Modified LSMVectorIndex.loadVectorsFromFile() to skip quantized data when loading vector locations:

  • INT8: Skip 1 byte (type) + 4 bytes (length) + N bytes (quantized) + 8 bytes (min/max)
  • BINARY: Skip 1 byte (type) + 4 bytes (length) + packed bytes + 4 bytes (median)

• Test coverage: Added comprehensive quantization test suite covering dimensions 4-128, persistence, and search functionality for both INT8 and BINARY

Status:

• ✅ INT8 quantization: Fully working across all tested dimensions (4-128)
• ✅ BINARY quantization: Fully working across tested dimensions (4-16 in parameterized tests, 32 in persistence tests, 64 in basic search, 128 in large dimensions test)

Testing:

• 48 total tests pass (17 new quantization tests: 9 INT8 + 8 BINARY)
• Original LSMVectorIndexTest suite unaffected
• All BINARY quantization tests enabled and passing with appropriate assertions for lossy compression behavior

Checklist

• I have run the build using mvn clean package command
• My unit tests cover both failure and success scenarios

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Bug: Vector Quantization (INT8/BINARY) is fundamentally broken across all dimensions</issue_title>
<issue_description># Bug: Vector Quantization (INT8/BINARY) is fundamentally broken across all dimensions

Summary

Both INT8 and BINARY quantization in LSM_VECTOR indexes are currently unusable. They fail with critical errors ranging from storage overflows and negative index exceptions to severe data loss, even at extremely low dimensions (e.g., 4).

• INT8: Fails with IndexOutOfBoundsException (negative indices) for dims < 16, and IllegalArgumentException (storage overflow) for dims >= 16.
• BINARY: Successfully builds the index but drops ~50% of vectors and yields ~0% recall due to read-time errors.

Environment

• Component: LSM_VECTOR Index (JVector integration)
• Quantization: INT8, BINARY
• Dimensions: Tested on 4, 8, 16, 32, 64, 100

Symptoms

INT8 Symptoms

1. Dims 4-8: Fails with IndexOutOfBoundsException accessing negative indices (e.g., -8, -64).
2. Dims >= 16: Fails with IllegalArgumentException: Variable length (70) quantity is too long (must be <= 63).
3. Dims >= 32: Fails with IllegalArgumentException: vector dimensions differ.

BINARY Symptoms

1. All Dims: Logs Filtered out X vectors with deleted/invalid documents (indicating data loss).
2. Search: Returns near-zero results or fails with NullPointerException.

Analysis

The error message Variable length (70) quantity is too long (must be <= 63) strongly suggests an overflow in the variable-length integer (VInt) encoding used by the underlying storage engine (likely in com.arcadedb.database.Binary or related serialization logic).

It appears that when INT8 quantization is active, the serialized size of the graph node (or a specific field within it) grows beyond the capacity of the variable-length encoding field being used.

• Dimensions 16: Fails (Recall 0%, Storage Overflow).
• Dimensions 32: Fails (Length 70 > 63).
• Dimensions 64: Fails.
• Dimensions 100: Fails.

This prevents the use of INT8 quantization for any practical vector dimensionality (e.g., 768, 1536) used in modern embeddings.

Note on BINARY Quantization

I also tested BINARY quantization. It behaves differently but is equally broken:

1. Index Creation: Succeeded for all dimensions (up to 100). It does not hit the "Variable length" storage limit.
2. Search: Fails immediately with IndexOutOfBoundsException and NullPointerException.

While INT8 fails at write time (storage overflow), BINARY fails at read time (incorrect offset calculation or data corruption during retrieval). Both are unusable for high-dimensional vectors.

Accuracy & Correctness (Dim=4, 8, 16, 32)

Further testing across multiple dimensions confirms that quantization is fundamentally broken. To validate the test harness, we also measured the recall of the unquantized (NONE) index finding itself (Ground Truth). The NONE index achieved 100% recall in all cases, proving the test data and search logic are correct.

Benchmark Results (N=1,000, K=10):

Dim	NONE (Self)	INT8 (vs NONE)	BINARY (vs NONE)	Notes
4	100.00%	7.00%	0.00%	INT8: Index -8 out of bounds
8	100.00%	3.50%	4.50%	INT8: Index -64 out of bounds
16	100.00%	1.50%	3.00%	INT8: Variable length (70) > 63
32	100.00%	0.00%	3.00%	INT8: Variable length (70) > 63

Logs Analysis

INT8 Errors:

• Dim 4: Index -8 out of bounds for length 3 (Negative index access).
• Dim 8: Index -64 out of bounds for length 3 (Negative index access).
• Dim 16+: IllegalArgumentException - Variable length (70) quantity is too long (Storage overflow).
• Dim 32: IllegalArgumentException: vector dimensions differ: 65536!=32 (Severe serialization mismatch).

BINARY Errors:

• All Dims: Filtered out X vectors with deleted/invalid documents (Data loss during indexing).
• Dim 8+: Error reading vector from offset ...: null (Read failure).

This confirms that INT8 suffers from severe offset calculation errors (negative indices) at very low dimensions and storage overflow at slightly higher dimensions. BINARY consistently fails to retrieve vectors correctly, often reading null or dropping data.

Performance Benchmark (Dim=16)

Warning: These performance numbers are for an index that produces incorrect results (0% recall). They are provided only to show the potential speedup if the feature were working.

Benchmark Results (Dim=16, N=10,000):

Metric	NONE	BINARY	Speedup
Insert Time	0.28s	0.15s	~1.9x
Index Time	0.12s	0.04s	~2.7x
Search Latency (Avg)	9....

• Fixes Bug: Vector Quantization (INT8/BINARY) is fundamentally broken across all dimensions #3052

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[lvca]

@copilot can you also take care of the remaining issue with BINARY quant?

[Copilot]

@copilot can you also take care of the remaining issue with BINARY quant?

Done! BINARY quantization is now fully working. The same offset bug that affected INT8 was also breaking BINARY quantization.

The fix (skipping quantized data during page loading) resolved both issues. I've re-enabled all BINARY tests:

• Added BINARY quantization to parameterized tests (dims 4, 8, 16)
• Re-enabled testBinaryQuantizationPersistence
• Re-enabled testLargeDimensionsBinary
• Added testBinaryQuantizationBasicSearch

All 17 quantization tests now pass (9 INT8 + 8 BINARY). ✅

Commit: 58e8e0c

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for 58e8e0c.

🟢 All jobs passed!

But CI Insights is watching 👀