Bypass total frequency check if field uses custom term frequency [LUCENE-10048]

[asfimport]

For all fields whose index option is not IndexOptions.NONE. There is a check on per field total token count (i.e. field-length) to ensure we don't index too many tokens. This is done by accumulating the token's TermFrequencyAttribute.

 

Given that currently Lucene allows custom term frequency attached to each token and the usage of the frequency can be pretty wild. It is possible to have the following case where the check fails with only a few tokens that have large frequencies. Currently Lucene will skip indexing the whole document.

"foo|<very large number> bar|<very large number>"

 

What should be way to inform the indexing chain not to check the field length?

A related observation, when custom term frequency is in use, user is not likely to use the similarity for this field. Maybe we can offer a way to specify that, too?

---

Migrated from LUCENE-10048 by Tony Xu (@Tony-X), 1 vote, resolved Aug 13 2021

[asfimport]

Tony Xu (@Tony-X) (migrated from JIRA)

My two-cents here:

is it a good idea to encode this information as a method boolean useCustomTermFreq() that defaults to false in FieldType? With that we can provide an extra constructor in TextField with a boolean argument to indicate if custom term frequency is in use.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

It has been brought up before, I don't think we should allow bypassing the check. If you use custom term frequencies (expert feature), then it is your responsibility to have reasonable values.

Otherwise, things can easily overflow, break scoring, cause failures from checkindex. There are aggregate statistics that assume values are in-bound for a field in the doc (e.g. less than int32).

[asfimport]

Ankur (@goankur) (migrated from JIRA)

@[~rcmuir]

Consider the case where these term-document level scoring factors are computed in an offline process, indexed in Lucene and accessed at query time by a ranking function that does not rely on Lucene's Scorer and Similarity abstractions.

What is considered reasonable is up to the offline process that serves the needs of the ranking function and is outside our control. A single term-document scoring factor can still be less than Integer.MAX_VALUE but the sum of all such factors for a document could easily exceed the Integer.MAX_VALUE range.

Without this our only option (I think) is to use BinaryDocValues and implement mechanisms to serialize/deserialize term-document level scoring factors at indexing and searching time ourselves. With this we don't get the space efficiencies that come with the use of highly optimized terms dictionary and the integer compression techniques used to encode postings data (at least not without significant work).

Maybe we can keep the restriction on the custom term frequency to be less than Integer.MAX_VALUE but relax the check on per field total token count for the expert use case ?
 

 

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Whether or not you use the similarity or scorer is irrelevant. I'm referring to term/field statistic values stored in the segment itself.
It doesn't matter what parts you do/don't use of lucene here, overflowing these values will essentially behave as corruption.

That's why I repeat myself every time a JIRA issue is opened to try to bypass these checks.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

btw, there are a lot of alternatives you can look into to avoid having > 2^32 tokens inside a single document's field. For example, you could use more fields, you could encode the thing in the payload, etc.

But I don't understand what you are doing, to me it sounds like lucene may not be the right solution at all honestly. Or maybe instead it is worth looking at lucene 9 vector format or something very different.

[asfimport]

Ankur (@goankur) (migrated from JIRA)

Thanks for your response @rmuir.

Let me try to explain the use case at a high level

1. An offline (map-reduce style) batch process consumes a set of indexable documents.
2. The process also consumes terms and metadata information from external data sources.
3. For each indexable document, the batch-process computes a set of term-doc scores and add this set to a document field (to be indexed later). 
4. A document will only have a small number of such terms in a field, less than 10K.
5. There could be many such fields in a single document populated by different offline processes, all of which scale these values arbitrarily (due to historical reasons) but still make sure a single value fits in 4-bytes.
6. The document also has usual textual fields (title, description etc) for which Lucene computes term/field statistics and produces BM25 scores.
7. All of these scores are used by a ranking method.

You are referring to Payloads right? It is a viable option but less space efficient (no delta compression) compared to storing these values directly as term-frequencies.

So only for fields that are populated by an external process, I am hoping we can come up with a mechanism to ignore the overflow checks on term/field statistics.

 

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

this doesn't explain any use case to me. it sounds like hogwash. you aren't using scoring, similarity, etc, you're making documents with more than 2^32 tokens in a single document's field, and crying out that you should be able to bypass checks.

sorry, again, sounds like maybe you shouldn't be using lucene for this. doesn't sound like search to me at all. It isn't viable to have an option that says "allow me to corrupt my index". -1 from me on that.

[asfimport]

Adrien Grand (@jpountz) (migrated from JIRA)

@goankur I wonder if encoding the scores more efficiently would be an option, e.g. using a bfloat16?

[asfimport]

Michael Sokolov (@msokolov) (migrated from JIRA)

> Ankur I wonder if encoding the scores more efficiently would be an option, e.g. using a bfloat16?

Or perhaps imposing a constraint on suppliers that scores must be < some bound, maybe 2^16 (still encoded as an int) and then (clipping, throwing an error, rejecting terms) when they are out of bounds

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Some brief historical context: #9990 (and this PR) is where we discussed this (and decided against it) last time.  Please read the discussion on the PR – that is when @rmuir convinced me removing this limit is indeed dangerous.

You are referring to Payloads right? It is a viable option but less space efficient (no delta compression) compared to storing these values directly as term-frequencies.

Note that term frequencies are also not delta-coded (the way Lucene docids are), because they are not in any sorted order.  Rather they are written in blocks of 128 ints, encoded with PFOR.  Though we might subtract the minimum value (and separately store it once) across that block and encode the deltas?  Not sure.  We do efficiently encode the case when all 128 values are the same value.

Payloads would work here, but I suspect that'd be slower to access at search time.

[~goankur] I wonder if encoding the scores more efficiently would be an option, e.g. using a bfloat16?

+1, that seems like a great idea!  Such an encoding should easily stay under Lucene's limits (as long as you have fewer than 64K such tokens, if all tokens had the max uint16 value of a bfloat16), and, that could represent the full range of numbers (clearly with some loss of precision).  Plus modern CPUs seem to maybe optimize for this representation (not sure if Hotspot can tap into that tho).  I like this path.  Though, I suspect compression would be worse for cases where the scores are all smallish positive integers, maybe.

It isn't viable to have an option that says "allow me to corrupt my index".

+1, I don't think we can safely relax this limit, even under a boolean option/flag.  This really is an abusive use-case for Lucene.

I can totally understand why we (Amazon product search) are using this feature, because we are trying to store a (sometimes massive) int score/ranking signal per query/atom X document.  But it is quite insane that we cannot contain the values to fit within int32 when summed across one doc X field.

[asfimport]

Tony Xu (@Tony-X) (migrated from JIRA)

Thanks Robert and Mike for the context. I also think we shouldn't have any option to  "allow me to corrupt my index". 

 

For this discussion I can think of two major kind of usecases for custom term frequency.

1. Users would like to override the frequency of certain token to boost scoring.

2. Users want to encode a generic map of term -> int, where the values could have nothing to do with frequency. Users are aware that scoring for this field won't make sense.

 

For case 1), we should enforce the checks to ensure scoring works, therefore not changing the field length check. In case 2), instead of dangerously bypassing, we can use the boolean flag to change field length accumulation from adding term's freq to simply increase by one (counting terms). I believe this gives the users flexibility without risking index corruption. 

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

no, you still dont get it. please read the PR that mike links to where i explain exactly why this dont work.

This issue is a "won't fix".

[asfimport]

Tony Xu (@Tony-X) (migrated from JIRA)

Okay I went thru the conversation in that PR about potential breaking points. 

I now believe the take away is even with custom term freq, you still have to play under the constraints and treat the values as frequency. I hope there is a mechanism for encoding the map of (term, score) that is as fast as posting access. On that note, maybe the right direction is to optimize payload?

 

Thanks for the helpful discussions.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

On that note, maybe the right direction is to optimize payload?

+1 to improve payloads performance if we can!  They have not received as much love lately as doc values :)