v1.5.0-rc.1

What's Changed

• chore(deps): bump the kubernetes group in /conformance with 4 updates by @dependabot[bot] in #2465
• chore(deps): bump github.com/prometheus/prometheus from 0.309.1 to 0.310.0 by @dependabot[bot] in #2464
• latencypredictor: training server mean objective + sliding window by @kaushikmitr in #2432
• [Feat]: Mirror request plugins execution to the response path by @abdallahsamabd in #2369
• remove redundant field body and isStreaming in requestcontrol.Response by @zetxqx in #2474
• Read metrics Datasource configuration from config file by @Mohamedma96 in #2441
• Update release promotion checklist for standalone chart by @danehans in #2479
• Adds missing license header by @davidbreitgand in #2484
• feat: Introduce Pluggable Parser Framework for EPP payload processing by @zetxqx in #2359
• Training server ensemble by @kaushikmitr in #2473
• Fix vllm gpu deployment yaml by @rahulgurnani in #2487
• feat(bbr): integrate request body into RequestContext for plugins by @noalimoy in #2442
• [epp/datalayer]: Move and rename FakeDataSource into pkg/epp/framework/plugins/datalayer/source/mocks package by @Mohamedma96 in #2429
• Updated go dependencies by @ahg-g in #2496
• release: include missing staging digests in release process by @danehans in #2497
• refactor(observability): extract common logging options to shared pac… by @yehuditkerido in #2395
• [pluggable bbr] remove empty placeholder plugin by @nirrozenbaum in #2451
• docs: switch quickstarts to agentgateway by @danehans in #2505
• latencypredictor: improve TPOT training accuracy by @kaushikmitr in #2509
• fix: log correct error from Prepare Data Plugins by @varad-ahirwadkar in #2512
• Second part of refactoring of the Ext Proc code between the EPP and the BBR by @shmuelk in #2446
• remove unused error code by @nirrozenbaum in #2516
• Add optional inference objective by @Gregory-Pereira in #1995
• chore(conformance): bump gateway-api to v1.5.0 by @danehans in #2519
• small refactor of pluggable bbr interfaces by @nirrozenbaum in #2513
• docs: add flow control user guide and concepts by @LukeAVanDrie in #2438
• feat: add inference request and received time in FlowControlRequest by @loicmarchal in #2475
• chore(deps): bump google.golang.org/grpc from 1.79.1 to 1.79.2 by @dependabot[bot] in #2533
• chore(deps): bump sigs.k8s.io/controller-runtime from 0.23.1 to 0.23.3 by @dependabot[bot] in #2535
• chore(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace from 1.41.0 to 1.42.0 by @dependabot[bot] in #2534
• chore(deps): bump golang.org/x/sync from 0.19.0 to 0.20.0 in /conformance by @dependabot[bot] in #2537
• chore(deps): bump sigs.k8s.io/controller-runtime from 0.23.1 to 0.23.3 in /conformance by @dependabot[bot] in #2538
• feat: pool-wide saturation computation + expose as gauge by @evacchi in #2343
• Add support for gRPC response trailers in EPP by @zetxqx in #2510
• latencypredictor: prediction server perf + mean + ensemble by @kaushikmitr in #2488
• Support arm64 builds by @ed-pai in #2350
• refactor: move chunking utilities to pkg/common/envoy/request by @noalimoy in #2549
• refactor: consolidate envoy request utilities under pkg/common/envoy by @asaadbalum in #2550
• feat(tracing): add tracing support and configuration options for BBR by @AtharvaPakade in #2518
• flowcontrol: add benchmark suite by @LukeAVanDrie in #2539
• Revert "Support arm64 builds (#2350)" by @kfswain in #2554
• [bbr] Apply request body mutations to ext_proc response by @asaadbalum in #2551
• [bbr] Apply response plugin body/header mutations to ext_proc response by @abdallahsamabd in #2477
• docs: add flow control guide to nav and cross-link by @LukeAVanDrie in #2546
• fix(bbr): enable response header/body processing in Istio EnvoyFilter by @abdallahsamabd in #2557
• test(envoy): add missing unit tests for envoy util functions by @yehuditkerido in #2529
• fix(bbr): remove body content from debug logs to prevent data leak by @abdallahsamabd in #2559
• fix(flaky) increase the timeout for setting up the ext_proc conn in integration test by @zetxqx in #2547
• [bbr] Skip body re-serialization when unchanged by @asaadbalum in #2556
• fix: Remove duplicate loop by @gyliu513 in #2564
• fix(bbr): add read lock to GetBaseModel to prevent concurrent map crash by @gyliu513 in #2570
• fix CI timeout issue with multi arch builds by @farazmd in #2572
• Token in flight features by @kaushikmitr in #2563
• feature: add slo-based deadline ordering policy by @loicmarchal in #2531
• test(bbr): add unit tests for body-field-to-header plugin by @asaadbalum in #2569
• revert multi arch build by @farazmd in #2580
• agentgateway: update docs and add conformance report by @danehans in #2565
• fix(conformance): use staging EPP image on main by @danehans in #2577
• feat: add additional labels to the existing scheduler_attempts_total metric by @lionelvillard in #2545
• fix(epp): avoid nested RLock in PoolGet to prevent potential deadlock by @gyliu513 in #2571
• chore(deps): bump golang.org/x/net from 0.48.0 to 0.51.0 in /conformance by @dependabot[bot] in #2469
• fix(prefix-cache): race conditions in indexer lock management by @hexfusion in #2501
• datastore minor cleanup by @nirrozenbaum in #2448
• feat(Flow Control)/Expand Flow Control capacity limits schema(resource.Quantity) by @BizerNotNull in #2492
• Add TokenizedPrompt to LLMRequest for external tokenization support by @acardace in #2542
• fix: race condition in PodUpdateOrAddIfNotExist reading pool without lock by @gyliu513 in #2578
• Expose latency prediction results to the endpoint by @RishabhSaini in #2478
• [pluggable bbr] Model to header extractor plugin by @davidbreitgand in #2392
• Fix request headers not captured in streaming mode by @abdallahsamabd in #2587
• [pluggable bbr] Initialize response plugins in runner based on configuration flags by @noalimoy in #2589
• fix(bbr): scope Istio EnvoyFilter to the selected inference gateway by @asaadbalum in #2568
• fix: log warning when pod ports don't match InferencePool targetPorts by @hexfusion in #2582
• fix(release): stop publishing a lora-syncer tag by @danehans in #2595
• chore(deps): bump sigs.k8s.io/gateway-api from 1.5.0 to 1.5.1 in /conformance by @dependabot[bot] in #2599
• feat: implement filtering with burst headroom by @LukeAVanDrie in #2594
• chore: improve EPP startup logging for config by @LukeAVanDrie in #2553
• Introduce datalayer plugins-runtime separation by @elevran in #2480
• (refactor)remove ResponseComplete interface, replace it with ResponseStream using endOfstream by @zetxqx in #2507
• feat: Support inflight token based saturation detection by @vishbhat in #2271
• Cleanup and small changes to tests related to BaseModelToHeaderPlugin by @davidbreitgand in #2607
• [Refactor] use the framework types directly instead of the aliased name by @Mohamedma96 in #2603
• [pluggable bbr] add CycleState for per-request shared plugin state by @noalimoy in #2612
• fix: Improved test coverage for scheduler_profile.go by @gyliu513 in #2610
• [pluggable bbr] wire CycleState into plugin interfaces by @noalimoy in #2614
• feat(bbr): introduce Handle abstraction for BBR plugin framework by @yehuditkerido in #2613
• [pluggable bbr] simplify CycleState to use string keys and any values by @noalimoy in #2615
• feat(bbr): add explicit plugin configuration to Helm chart values.yaml by @szedan-rh in #2604
• refactor: move fairness and ordering plugins to epp/framework path by @loicmarchal in #2608
• test(conformance): cover InferencePool appProtocol semantics by @danehans in #2574
• Add max TTFT to stickiness criteria by @kaushikmitr in #2601
• [fix] properly template extraContainerPorts by @Gregory-Pereira in #2621
• refactor(bbr): move test helpers into plugins/test by @yehuditkerido in #2627
• Bug: DynamicMetadata dropped from Envoy responses by @adelsam in #2593
• chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.41.0 to 1.42.0 by @dependabot[bot] in #2598
• chore(deps): bump sigs.k8s.io/gateway-api/conformance from 1.5.0 to 1.5.1 in /conformance by @dependabot[bot] in #2600
• Adding README's to the scorer implementations by @kfswain in #2609
• Update latency predictor helm defaults to production-tuned values by @BenjaminBraunDev in #2619
• fix: extract usage from ResponsesAPI streaming events by @varad-ahirwadkar in #2573
• Docs: BBR guide fix by @oshritf in #2541
• agentgateway: update v1.4.0 conformance report and guides by @danehans in #2642
• fixed a bug in istio EnvoyFilter in bbr chart by @nirrozenbaum in #2643
• Adds missing unit test for BaseModelToHeaderPlugin by @davidbreitgand in #2617
• [Fix] typo in pkg/epp/framework/plugins/scheduling/scorer/prefix/plugin.go by @roytman in #2661
• Add more tracing spans from orchestrator layer by @waterloong in #2239
• Adding missing information about BBR plugins to the README.md by @davidbreitgand in #2655
• fix(test): Add unit test for random picker by @gyliu513 in #2659
• chore: Fixed parameter error by @gyliu513 in #2668
• fix(epp): correct composite-least strategy in negative headroom selection by @gyliu513 in #2583
• chore(deps): bump google.golang.org/grpc from 1.79.2 to 1.79.3 by @dependabot[bot] in #2671
• fix duplicate responseBody call with Eos true for streaming response by @zetxqx in #2652
• refactor(bbr): remove GetReconciler from BaseModelToHeaderPlugin by @yehuditkerido in #2663
• Improve E2E test setup: CRD validation, idempotent creation, cluster logging by @Mohamedma96 in #2641
• refactor(bbr): move ConfigMap label filter into the plugin by @yehuditkerido in #2679
• gRPC gRPC-in-gRPC-out support for Generate non-streaming by @zetxqx in #2211
• fix typo in metrics guide by @nirrozenbaum in #2687
• [pluggable bbr] Cleanup by @davidbreitgand in #2683
• Latency predictor load tests by @kaushikmitr in #2620
• fix(release): select newest staging digest for tag by @danehans in #2631
• feat(bbr): return ImmediateResponse on plugin errors by @noalimoy in #2637
• Add support for /v1/embeddings endpoint by @sbekkerm in #2419
• grpc add gRPC support for streaming response by @zetxqx in #2686
• refactor: saturation plugins config loading by @LukeAVanDrie in #2648
• [grpc] add vllm-grpc deployment example for gpu and sim by @zetxqx in #2696
• upgrade go mod version + client go regeneration by @nirrozenbaum in #2697
• refactor: move SaturationDetector to framework pkg and improve godocs by @LukeAVanDrie in #2650
• Clean up NGINX docs by @sjberman in #2700
• Add predictor sub-plugin for latency prediction pipeline by @kaushikmitr in #2693
• feat: promote Saturation Detector to a pluggable extension point by @LukeAVanDrie in #2704
• cleanup: remove PriorityName field from PriorityBandConfig by @LukeAVanDrie in #2709
• fix(logging): map negative Zap levels to custom verbosity constants by @LukeAVanDrie in #2712
• cleanup: removes redundant parameters from the deferred anonymous function in pkg/epp/handlers/server.go by @zetxqx in #2714
• feat(flowcontrol): Add pluggable eviction queue for in-flight request tracking by @RishabhSaini in #2597
• small refactor to bbr handlers package by @nirrozenbaum in #2688
• rename latencypredictor predictedlatency by @kaushikmitr in #2725
• cleanup: remove stale dead code (>1 mo old) by @LukeAVanDrie in #2707
• feat: configurable EnvoyFilter op in BBR chart by @noyitz in #2727
• Add latency admission sub-plugin for SLO-based request shedding by @kaushikmitr in #2695
• test: refactor integration test harness to support data layer testing by @Mohamedma96 in #2664
• fix: accumulate ResponseSize in HandleResponseBody so response_sizes metric is recorded by @wenhug in #2726
• docs: improve Getting Started guide clarity for new contributors by @Neha-dot-Yadav in #2723
• fix: explain InferencePool ownership by @gyliu513 in #2701
• [flake] ignore metrics update after endpoint is released by @elevran in #2732
• fix: add subFilter anchor for INSERT_BEFORE by @noyitz in #2735
• [pluggable parser] refactor: replace ParsedBody 'any' with strongly-typed RequestPayload interface by @zetxqx in #2734
• Add rahulgurnani to set of people who can use benchmarking workflow by @rahulgurnani in #2743
• feat: support configurable HTTPRoute request timeout in inferencepool chart by @nicole-lihui in #2729
• [bbr] add integration tests for response plugin execution by @abdallahsamabd in #2450
• fix: expose pool-level average running requests metric and fix integer truncation by @wenhug in #2719
• Update benchmarking pipeline to use qwen deployments by @rlakhtakia in #2748
• [Fix] Datalayer error handling by @elevran in #2660
• Add extra time to initialize resources by @rlakhtakia in #2751
• Bump llm-d-inference-sim to v0.8.2 by @cyclinder in #2756
• fix bbr: skip gracefully when body field is missing by @noyitz in #2757
• chore(epp): migrate saturation detector plugins by @LukeAVanDrie in #2718
• feat: add flow control grafana dashboard by @LukeAVanDrie in #2752
• [grpc] add support for vLLM gRPC Embed endpoint in vllmgrpc-parser by @zetxqx in #2710
• feat: Add unit test for single_profile_handler.go by @gyliu513 in #2656
• fix: correct typos in code comments by @Mohamedma96 in #2374
• Unit test fixes - Add missing test cases for new datalayer metrics. by @Mohamedma96 in #2640
• chore(deps): bump golang.org/x/time from 0.14.0 to 0.15.0 by @dependabot[bot] in #2745
• test: refactor Kustomize utils to support delete operations by @AvineshTripathi in #2033
• logging: standardize EPP log levels with DEFAULT constants by @LukeAVanDrie in #2715
• fix(test): Add unit test for prediction by @gyliu513 in #2662
• Add NGINX Gateway Fabric conformance report by @sjberman in #2736
• Conformance: Create Dedicated Shared Location for Feature Names by @ericdbishop in #2434
• add pluggable parser document by @zetxqx in #2733
• [LatencyPredictor Decomposition] Add SLO headroom tier filter for latency-based routing by @kaushikmitr in #2754
• [LatencyPredictor Decomposition] Add prefix cache affinity filter for sticky endpoint routing by @kaushikmitr in #2753
• [LatencyPredictor Decomposition] Add latency scorer sub-plugin for headroom-based endpoint scoring by @kaushikmitr in #2694
• [pluggable parser] add passthrough parser for use case parsing the request body is not wanted by @zetxqx in #2738
• optimize httproute rendering by @learner0810 in #2728
• [Fix] Support prompt array for /completions request by @elevran in #2724
• fix: add read lock to AllOrderedPriorityLevels to prevent data race by @gyliu513 in #2674
• feat: usage limit policy plugin by @evacchi in #2530
• feat: extend with multimodal features for MM-aware prefix-cache scoring by @sagearc in #2720
• remove lora sidecar in favor of native resolver by @LukeAVanDrie in #2744
• docs: update release process for sub-modules by @LukeAVanDrie in #2711
• cleanup: modernize codebase using Go 1.26 go fix suite by @LukeAVanDrie in #2708
• Update vLLM manifest image tags by @danehans in #2768
• release: align conformance module tagging by @danehans in #2767
• Add trtllm-serve as a built-in engine type for metric scraping by @BenjaminBraunDev in #2713
• Adjust the images field in charts by @learner0810 in #2412
• feat(epp): implement token load-aware scorer plugin by @brent-srg in #2490
• refactor: rename PodLocator to EndpointCandidates by @cyclinder in #2548
• [feat] Endpoint datasource to notify on adding and removing endpoints by @elevran in #2749
• chore: align default model references to Qwen by @karimzakzouk in #2771
• [grpc support] update helm chart for gRPC and update getting-started doc by @zetxqx in #2774
• fix: rewrite model name in response body to match client-facing name by @gyliu513 in #2769
• [pluggable parser] helm expose parser and update getting started doc by @zetxqx in #2780
• Decompose predictedlatency plugin into 3 composable sub-plugins by @kaushikmitr in #2667
• fix(helm): not set inferencePool.AppProtocol when modelServerProtocol is not set by @zetxqx in #2781
• Enable DataLayer by default, Automatically add Data section if missing, when datalayer is enabled by @Mohamedma96 in #2639
• BBR pluggable framework proposal by @davidbreitgand in #1964
• simplify bbr plugins and update tests accordingly by @nirrozenbaum in #2783
• Refactor Prefix Cache Scorer into a separate PrepareData plugin and restructure related code by @rahulgurnani in #2453
• chore(deps): bump google.golang.org/grpc from 1.79.3 to 1.80.0 by @dependabot[bot] in #2792
• chore(deps): bump go.opentelemetry.io/otel from 1.42.0 to 1.43.0 by @dependabot[bot] in #2791
• Add prefix cache metric label overrides and gauge-style extraction for SGLang and Triton respectively by @BenjaminBraunDev in #2741
• Fix approx prefix cache dataproducer auto-initialization by @rahulgurnani in #2798
• chore(deps): bump go.opentelemetry.io/otel/sdk from 1.42.0 to 1.43.0 by @dependabot[bot] in #2795
• deprecate-v1alpha2-InferencePool-API by @learner0810 in #2796
• feat(Flow Control)/Expand Flow Control capacity limits schema(maxRequests) by @BizerNotNull in #2495
• flow control: refactor and group dependencies in FlowController constructor by @evacchi in #2775
• chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.42.0 to 1.43.0 by @dependabot[bot] in #2794
• token_estimator: handles Responses and Conversations API by @RishabhSaini in #2647
• Remove GW API CRD dependency in conformance tests by @ericdbishop in #2433
• fix: Remove test-specific field from Runner by generalizing NewDefaultManager options by @gyliu513 in #2787
• fix: add mutex to protect concurrent access to registeredFeatureGates by @gyliu513 in #2800
• fix: concurrencyTracker.dec drift by @gyliu513 in #2802

New Contributors

• @abdallahsamabd made their first contribution in #2369
• @Mohamedma96 made their first contribution in #2441
• @loicmarchal made their first contribution in #2475
• @ed-pai made their first contribution in #2350
• @gyliu513 made their first contribution in #2564
• @farazmd made their first contribution in #2572
• @lionelvillard made their first contribution in #2545
• @hexfusion made their first contribution in #2501
• @BizerNotNull made their first contribution in #2492
• @acardace made their first contribution in #2542
• @vishbhat made their first contribution in #2271
• @oshritf made their first contribution in #2541
• @roytman made their first contribution in #2661
• @waterloong made their first contribution in #2239
• @sbekkerm made their first contribution in #2419
• @noyitz made their first contribution in #2727
• @wenhug made their first contribution in #2726
• @Neha-dot-Yadav made their first contribution in #2723
• @nicole-lihui made their first contribution in #2729
• @cyclinder made their first contribution in #2756
• @brent-srg made their first contribution in #2490
• @karimzakzouk made their first contribution in #2771

Full Changelog: v1.4.0...v1.5.0-rc.1