Created a faster ingestion mode - pipeline

[dbzoo]

Created a faster ingestion mode - pipeline

Configuration

embedding:
  mode: ollama
  embed_dim: 768
  ingest_mode: pipeline
  count_workers: 2

Comparison (mm:ss) Ingesting 434 documents 144Mb all stores are in postgres using ollama for embeddings

Mode	2 Workers	4 Workers
pipeline	3:42	3:32
parallel	4:18	3:45
batch	6:43
simple	9:45

Using the local profile

Mode	2 Workers	4 Workers
pipeline	3:47	2:42
parallel	5:11	4:26

In the parallel ingest design, the blocking mutex for the index write causes a bottleneck stalling the embedding computations until the write operation completes. This is particularly problematic because the index is updated per file, exacerbating the slowdown. In contrast, the pipeline design adopts a non-blocking approach. Here, all worker data is fed into a single queue, where it accumulates before being written less frequently. This design choice allows for smoother and more efficient processing, as it minimizes the impact of filesystem operations on the overall workflow.

Add an ETA logger so you can get an idea how far its gone and when it going to finish

11:34:09.279 [INFO    ]     private_gpt.utils.eta - 237/434 - ETA 1m 39s @ 120/min

• Processed files / total files ETA - time remaining @ files processed / minute

The first log will appear after 30s of ingestion, and then every 60s thereafter.

[github-actions]

Published docs preview URL: https://privategpt-preview-a9ccd9ef-0949-4327-a8f2-1b1f816f1bc5.docs.buildwithfern.com

[github-actions]

Published docs preview URL: https://privategpt-preview-a10c7df0-2c41-46cb-958c-151f09400f61.docs.buildwithfern.com

[github-actions]

Published docs preview URL: https://privategpt-preview-2f8403c1-7839-4be4-a483-04a72a181e7c.docs.buildwithfern.com

[imartinez]

All I can say is 🙌
Impressive contribution and execution
It is great to have you as a contributor of the project 👏