Skip to content

Update HNSW indexes to Lucene 10 #3097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lintool merged 22 commits into from
Jan 19, 2026
Merged

Update HNSW indexes to Lucene 10 #3097

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
1ba08c3
Testing HNSW for Lucene 10
lintool Jan 11, 2026
eb33a88
Changed code, configs remain same.
lintool Jan 11, 2026
d243426
Update params for large BEIR collections.
lintool Jan 14, 2026
732f151
Propagate params to hnsw cached and hnsw-int8
lintool Jan 14, 2026
6b4ce79
-efSearch 4000
lintool Jan 14, 2026
01c62b9
Remaining hnsw collections, threads=4.
lintool Jan 14, 2026
b515063
efC=500 for small collections.
lintool Jan 15, 2026
a6d6fa3
Changing to 2000 for all smaller collections.
lintool Jan 15, 2026
d267ca0
Int8 to SVQ, added BVQ
lintool Jan 15, 2026
9ac00f0
SVQ, after round 2, 22 smaller collections passed.
lintool Jan 15, 2026
78c7ad0
SQV, after tweaks to config for larger collections.
lintool Jan 15, 2026
9291a4b
Running all, tweaked trec-news.
lintool Jan 16, 2026
09c671d
aligned hnsw.onnx and hnsw.cached
lintool Jan 16, 2026
a3c0c76
aligned hnsw-bqv.onnx and hnsw-bqv.cached
lintool Jan 16, 2026
e9e633b
Merge branch 'lucene10' into lucene10-hnsw
lintool Jan 16, 2026
ffc6dcf
hnsw-sqv.onnx/cached reconcilation
lintool Jan 16, 2026
d77a4ba
hnsw-sqv.onnx/cached reconcilation
lintool Jan 16, 2026
2c379b4
hnsw.onnx/cached reconcilation
lintool Jan 16, 2026
c818df5
starting to tune parameters for MS MARCO - HNSW
lintool Jan 17, 2026
1d78f5e
BEIR fine tuning parameters.
lintool Jan 17, 2026
8315210
MS MARCO hnsw and hnsw-sqv parameters.
lintool Jan 17, 2026
a9c7328
BEIR tolerance tweaks.
lintool Jan 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
24 changes: 12 additions & 12 deletions ...-base-en-v1.5.parquet.hnsw-int8.cached.md → ...e-base-en-v1.5.parquet.hnsw-sqv.cached.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ This page describes regression experiments, integrated into Anserini's regressio

In these experiments, we are using cached queries (i.e., cached results of query encoding).

The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-int8.cached.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-int8.cached.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.
The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-sqv.cached.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-sqv.cached.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-int8.cached
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-sqv.cached
```

All the BEIR corpora, encoded by the BGE-base-en-v1.5 model and stored in Parquet format, are available for download:
Expand All @@ -33,12 +33,12 @@ Sample indexing command, building quantized HNSW indexes:

```
bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-threads 16 \
-threads 4 \
-collection ParquetDenseVectorCollection \
-input /path/to/beir-v1.0.0-arguana.bge-base-en-v1.5 \
-generator DenseVectorDocumentGenerator \
-index indexes/lucene-hnsw-int8.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-M 16 -efC 100 -quantize.int8 \
-index indexes/lucene-hnsw-sqv.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-M 16 -efC 500 -quantize.sqv \
>& logs/log.beir-v1.0.0-arguana.bge-base-en-v1.5 &
```

Expand All @@ -52,19 +52,19 @@ After indexing has completed, you should be able to perform retrieval as follows

```
bin/run.sh io.anserini.search.SearchHnswDenseVectors \
-index indexes/lucene-hnsw-int8.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-index indexes/lucene-hnsw-sqv.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-topics tools/topics-and-qrels/topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.gz \
-topicReader JsonStringVector \
-output runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-int8-cached.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt \
-hits 1000 -efSearch 1000 -removeQuery -threads 16 &
-output runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-sqv-cached.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt \
-hits 1000 -efSearch 2000 -removeQuery -threads 16 &
```

Evaluation can be performed using `trec_eval`:

```
bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-int8-cached.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-int8-cached.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-int8-cached.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-sqv-cached.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-sqv-cached.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-sqv-cached.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt
```

## Effectiveness
Expand Down
24 changes: 12 additions & 12 deletions ...ge-base-en-v1.5.parquet.hnsw-int8.onnx.md → ...bge-base-en-v1.5.parquet.hnsw-sqv.onnx.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ This page describes regression experiments, integrated into Anserini's regressio

In these experiments, we are using ONNX to perform query encoding on the fly.

The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-int8.onnx.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-int8.onnx.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.
The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-sqv.onnx.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-sqv.onnx.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-int8.onnx
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw-sqv.onnx
```

All the BEIR corpora, encoded by the BGE-base-en-v1.5 model and stored in Parquet format, are available for download:
Expand All @@ -33,12 +33,12 @@ Sample indexing command, building quantized HNSW indexes:

```
bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-threads 16 \
-threads 4 \
-collection ParquetDenseVectorCollection \
-input /path/to/beir-v1.0.0-arguana.bge-base-en-v1.5 \
-generator DenseVectorDocumentGenerator \
-index indexes/lucene-hnsw-int8.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-M 16 -efC 100 -quantize.int8 \
-index indexes/lucene-hnsw-sqv.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-M 16 -efC 500 -quantize.sqv \
>& logs/log.beir-v1.0.0-arguana.bge-base-en-v1.5 &
```

Expand All @@ -52,19 +52,19 @@ After indexing has completed, you should be able to perform retrieval as follows

```
bin/run.sh io.anserini.search.SearchHnswDenseVectors \
-index indexes/lucene-hnsw-int8.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-index indexes/lucene-hnsw-sqv.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-topics tools/topics-and-qrels/topics.beir-v1.0.0-arguana.test.tsv.gz \
-topicReader TsvString \
-output runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-int8-onnx.topics.beir-v1.0.0-arguana.test.txt \
-encoder BgeBaseEn15 -hits 1000 -efSearch 1000 -removeQuery -threads 16 &
-output runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-sqv-onnx.topics.beir-v1.0.0-arguana.test.txt \
-encoder BgeBaseEn15 -hits 1000 -efSearch 2000 -removeQuery -threads 16 &
```

Evaluation can be performed using `trec_eval`:

```
bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-int8-onnx.topics.beir-v1.0.0-arguana.test.txt
bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-int8-onnx.topics.beir-v1.0.0-arguana.test.txt
bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-int8-onnx.topics.beir-v1.0.0-arguana.test.txt
bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-sqv-onnx.topics.beir-v1.0.0-arguana.test.txt
bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-sqv-onnx.topics.beir-v1.0.0-arguana.test.txt
bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-sqv-onnx.topics.beir-v1.0.0-arguana.test.txt
```

## Effectiveness
Expand Down
6 changes: 3 additions & 3 deletions ...essions/regressions-beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw.cached.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,12 @@ Sample indexing command, building HNSW indexes:

```
bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-threads 16 \
-threads 4 \
-collection ParquetDenseVectorCollection \
-input /path/to/beir-v1.0.0-arguana.bge-base-en-v1.5 \
-generator DenseVectorDocumentGenerator \
-index indexes/lucene-hnsw.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-M 16 -efC 100 \
-M 16 -efC 500 \
>& logs/log.beir-v1.0.0-arguana.bge-base-en-v1.5 &
```

Expand All @@ -56,7 +56,7 @@ bin/run.sh io.anserini.search.SearchHnswDenseVectors \
-topics tools/topics-and-qrels/topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.gz \
-topicReader JsonStringVector \
-output runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-cached.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt \
-hits 1000 -efSearch 1000 -removeQuery -threads 16 &
-hits 1000 -efSearch 2000 -removeQuery -threads 16 &
```

Evaluation can be performed using `trec_eval`:
Expand Down
6 changes: 3 additions & 3 deletions ...gressions/regressions-beir-v1.0.0-arguana.bge-base-en-v1.5.parquet.hnsw.onnx.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,12 @@ Sample indexing command, building HNSW indexes:

```
bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-threads 16 \
-threads 4 \
-collection ParquetDenseVectorCollection \
-input /path/to/beir-v1.0.0-arguana.bge-base-en-v1.5 \
-generator DenseVectorDocumentGenerator \
-index indexes/lucene-hnsw.beir-v1.0.0-arguana.bge-base-en-v1.5/ \
-M 16 -efC 100 \
-M 16 -efC 500 \
>& logs/log.beir-v1.0.0-arguana.bge-base-en-v1.5 &
```

Expand All @@ -56,7 +56,7 @@ bin/run.sh io.anserini.search.SearchHnswDenseVectors \
-topics tools/topics-and-qrels/topics.beir-v1.0.0-arguana.test.tsv.gz \
-topicReader TsvString \
-output runs/run.beir-v1.0.0-arguana.bge-base-en-v1.5.bge-hnsw-onnx.topics.beir-v1.0.0-arguana.test.txt \
-encoder BgeBaseEn15 -hits 1000 -efSearch 1000 -removeQuery -threads 16 &
-encoder BgeBaseEn15 -hits 1000 -efSearch 2000 -removeQuery -threads 16 &
```

Evaluation can be performed using `trec_eval`:
Expand Down
24 changes: 12 additions & 12 deletions ...-base-en-v1.5.parquet.hnsw-int8.cached.md → ...e-base-en-v1.5.parquet.hnsw-sqv.cached.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ This page describes regression experiments, integrated into Anserini's regressio

In these experiments, we are using cached queries (i.e., cached results of query encoding).

The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-int8.cached.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-int8.cached.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.
The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-sqv.cached.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-sqv.cached.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-int8.cached
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-sqv.cached
```

All the BEIR corpora, encoded by the BGE-base-en-v1.5 model and stored in Parquet format, are available for download:
Expand All @@ -33,12 +33,12 @@ Sample indexing command, building quantized HNSW indexes:

```
bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-threads 16 \
-threads 4 \
-collection ParquetDenseVectorCollection \
-input /path/to/beir-v1.0.0-bioasq.bge-base-en-v1.5 \
-generator DenseVectorDocumentGenerator \
-index indexes/lucene-hnsw-int8.beir-v1.0.0-bioasq.bge-base-en-v1.5/ \
-M 16 -efC 100 -quantize.int8 \
-index indexes/lucene-hnsw-sqv.beir-v1.0.0-bioasq.bge-base-en-v1.5/ \
-M 32 -efC 1000 -quantize.sqv \
>& logs/log.beir-v1.0.0-bioasq.bge-base-en-v1.5 &
```

Expand All @@ -52,19 +52,19 @@ After indexing has completed, you should be able to perform retrieval as follows

```
bin/run.sh io.anserini.search.SearchHnswDenseVectors \
-index indexes/lucene-hnsw-int8.beir-v1.0.0-bioasq.bge-base-en-v1.5/ \
-index indexes/lucene-hnsw-sqv.beir-v1.0.0-bioasq.bge-base-en-v1.5/ \
-topics tools/topics-and-qrels/topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.gz \
-topicReader JsonStringVector \
-output runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-int8-cached.topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.txt \
-hits 1000 -efSearch 2000 -removeQuery -threads 16 &
-output runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-sqv-cached.topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.txt \
-hits 1000 -efSearch 16000 -removeQuery -threads 16 &
```

Evaluation can be performed using `trec_eval`:

```
bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-int8-cached.topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-int8-cached.topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-int8-cached.topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-sqv-cached.topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-sqv-cached.topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.txt
bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-sqv-cached.topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.txt
```

## Effectiveness
Expand Down
24 changes: 12 additions & 12 deletions ...ge-base-en-v1.5.parquet.hnsw-int8.onnx.md → ...bge-base-en-v1.5.parquet.hnsw-sqv.onnx.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ This page describes regression experiments, integrated into Anserini's regressio

In these experiments, we are using ONNX to perform query encoding on the fly.

The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-int8.onnx.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-int8.onnx.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.
The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-sqv.onnx.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-sqv.onnx.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-int8.onnx
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-bioasq.bge-base-en-v1.5.parquet.hnsw-sqv.onnx
```

All the BEIR corpora, encoded by the BGE-base-en-v1.5 model and stored in Parquet format, are available for download:
Expand All @@ -33,12 +33,12 @@ Sample indexing command, building quantized HNSW indexes:

```
bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-threads 16 \
-threads 4 \
-collection ParquetDenseVectorCollection \
-input /path/to/beir-v1.0.0-bioasq.bge-base-en-v1.5 \
-generator DenseVectorDocumentGenerator \
-index indexes/lucene-hnsw-int8.beir-v1.0.0-bioasq.bge-base-en-v1.5/ \
-M 16 -efC 500 -quantize.int8 \
-index indexes/lucene-hnsw-sqv.beir-v1.0.0-bioasq.bge-base-en-v1.5/ \
-M 32 -efC 1000 -quantize.sqv \
>& logs/log.beir-v1.0.0-bioasq.bge-base-en-v1.5 &
```

Expand All @@ -52,19 +52,19 @@ After indexing has completed, you should be able to perform retrieval as follows

```
bin/run.sh io.anserini.search.SearchHnswDenseVectors \
-index indexes/lucene-hnsw-int8.beir-v1.0.0-bioasq.bge-base-en-v1.5/ \
-index indexes/lucene-hnsw-sqv.beir-v1.0.0-bioasq.bge-base-en-v1.5/ \
-topics tools/topics-and-qrels/topics.beir-v1.0.0-bioasq.test.tsv.gz \
-topicReader TsvString \
-output runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-int8-onnx.topics.beir-v1.0.0-bioasq.test.txt \
-encoder BgeBaseEn15 -hits 1000 -efSearch 2000 -removeQuery -threads 16 &
-output runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-sqv-onnx.topics.beir-v1.0.0-bioasq.test.txt \
-encoder BgeBaseEn15 -hits 1000 -efSearch 16000 -removeQuery -threads 16 &
```

Evaluation can be performed using `trec_eval`:

```
bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-int8-onnx.topics.beir-v1.0.0-bioasq.test.txt
bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-int8-onnx.topics.beir-v1.0.0-bioasq.test.txt
bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-int8-onnx.topics.beir-v1.0.0-bioasq.test.txt
bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-sqv-onnx.topics.beir-v1.0.0-bioasq.test.txt
bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-sqv-onnx.topics.beir-v1.0.0-bioasq.test.txt
bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-bioasq.test.txt runs/run.beir-v1.0.0-bioasq.bge-base-en-v1.5.bge-hnsw-sqv-onnx.topics.beir-v1.0.0-bioasq.test.txt
```

## Effectiveness
Expand Down
Loading