Benchmark Haystack Pipeline

[srini047]

Currently there is no way to benchmark pipelines and its components. It's all either manual (intuition) or making use of logging and deriving results from traces/spans. Though this is fine but this still adds latency and misses the actual few ms due to network or function call. A native way to benchmark the pipelines is a good way to compare results and statistically derive valuable metrics out from your pipeline.

Describe the solution you'd like

• Benchmark as part of Pipeline() itself
• No external dependencies (using default Python methods)
• Results should be both Pipeline level and per component level
• Using percentiles instead of average as it provides more accurate and user-centric view of how the pipeline actually performs in the real world. So p50, p90, p99 are must including avg and total displayed.
• Display the benchmark result in a user-friendly way.

Describe alternatives you've considered

• Tracing the pipeline
• Retrieving timestamp from tracer span.
• Then computing per component and pipeline level metrics and then deduce the results.

This isn't a good DX and metrics won't match the actual world pipeline runs as well.

Additional context
Raising PR for review since I feel this is a good value addition to the haystack pipelines.

[sjrl]

Hey @srini047 thanks for opening!

We also have a request for tracking other metrics for a pipeline #10889 and it's possible we can achieve this using an in-memory tracer we have which I mention here #10889 (comment).

Still too early to say what the final approach will be, but we will let you know once we have more information.

P.S. I already commented this on your PR but I thought it would be good to have this info here as well.

[srini047]

Hey @srini047 thanks for opening!

We also have a request for tracking other metrics for a pipeline #10889 and it's possible we can achieve this using an in-memory tracer we have which I mention here #10889 (comment).

Still too early to say what the final approach will be, but we will let you know once we have more information.

P.S. I already commented this on your PR but I thought it would be good to have this info here as well.

Thanks for the acknowledgment and response @sjrl
If my understanding is not wrong, you would like to use the native tracer as part of pipeline benchmarking/metrics collection. I have tried that approach as well for something else and you can take a look at this if this is what it would look at the high level.

If yes then there's a catch with latency while using the Pipeline() over tracer:

 haystack pipeline benchmark pipelines/test.yaml \ 
  --input '{"double": {"value": 3}}' \
  --runs 10

  Benchmarking pipelines/test.yaml — 10 runs (1 warmup)

                                                                                  
  Component            Type                      p50      p95      avg   % total  
 ──────────────────────────────────────────────────────────────────────────────── 
  double               DoubleIt               0.01ms   0.02ms   0.01ms     12.1%  
  add_one              AddOne                 0.01ms   0.01ms   0.01ms     11.2%  
                                                                                  
  total                                       0.06ms   0.09ms   0.06ms      100%  
                                                                                  
  Slowest: double (12.1% of total)
  Fastest run: 0.05ms  |  Slowest run: 0.08ms

This is the output I have from the code above. We can see this has much better results as compared to the native benchmark (#10895 (comment)). I'm still skeptical how this is faster compared to using benchmarking logic in Pipeline() component directly. Hopefully this data point has some value in the decision making if applicable.

Thanks.

[sjrl]

Hey @srini047 I had a chance to talk about this with @julian-risch. We do see and understand the value of being able to benchmark the pipeline for the reasons you outlined.

However, we do think it would be best if we could avoid changing the code within Pipeline as much as possible and keep all benchmarking methods external and in a separate file (e.g. benchmark.py). This way we keep the Pipeline code smaller and we will want to support both Pipeline and AsyncPipeline in the benchmarking functions.