Scalar Quantizer to support signed int8 vectors

[naveentatikonda]

Summary

As of today, we have three different 8 bit quantizers QT_8bit_direct, QT_8bit_uniform and QT_8bit but all three of them only support uint8 vectors ranging from [0 to 255]. If we try to ingest signed 8 bit vectors ([-128 to 127]) using these quantizers then during encoding it will be casted as uint8 values which will change the sign and magnitude of values outside of uint8 range. There are few use cases where customers want to use models like Cohere Embed that generates signed int8 embeddings ranging from [-128 to 127]. To support such use cases we need a new signed 8 bit scalar quantizer.

Solution

To solve this problem, we can add a new signed 8 bit quantizer something similar to QT_8bit_direct where during encoding it adds 128 to each dimension of the vector to bring it into uint8 range to store in uint8_t* code. Similarly, during decoding or while reconstructing the components 128 will be subtracted from each dimension to retrieve the actual signed int8 vector back before computing the distance.

[naveentatikonda]

@mdouze @alexanderguzhva Can you please take a look and provide your comments on the approach mentioned above ?

I have already made the code changes on my local with AVX2 and NEON intrinsics and tested it with some datasets like cohere, ms_marco, sift, etc. The recall and other performance benchmarks looks good.

[mdouze]

Are you using max inner product search or L2 search?
for L2 you can just use QT_8bit_direct and add 128 to the vector components (that are passed in a floats anyways).
I am not saying that we should not support signed int8 it but the workaround seems simple.

[alexanderguzhva]

@naveentatikonda so, the plan is to use cvtepi8 instead of cvtepu8 for x86 and s8 instead of u8 for ARM, correct? Well, why not, the new code is trivial

[naveentatikonda]

Are you using max inner product search or L2 search? for L2 you can just use QT_8bit_direct and add 128 to the vector components (that are passed in a floats anyways). I am not saying that we should not support signed int8 it but the workaround seems simple.

@mdouze Initially, we had this approach in our mind. But, it doesn't work for InnerProduct and we want to support both L2 and IP space types.

[naveentatikonda]

@naveentatikonda so, the plan is to use cvtepi8 instead of cvtepu8 for x86 and s8 instead of u8 for ARM, correct? Well, why not, the new code is trivial

@alexanderguzhva I think replacing cvtepu8 with cvtepi8 for x86 will not decode it back to signed int8 properly. Here is an example

Input vector -> -10, -12, -14, -16, -18, 20, -22, -128
Encoded vector -> 118, 116, 114, 144, 110, 148, 106, 0
Decoding with cvtepu8 -> 118, 116, 114, 144, 110, 148, 106, 0
Decoding with cvtepi8 -> 118, 116, 114, -112, 110, -108, 106, 0

I was trying to subtract 128 from all dimensions after loading it with cvtepu8. Not sure if there is a better way of doing this with intrinsics

[alexanderguzhva]

@naveentatikonda I mean that adding a new int8_t quantizer is a reasonable idea for your use case.