Weighted ECMP (Link-Bandwidth) Fails When Routes are Leaked Between VRFs

[f0o]

Description

When attempting to implement Weighted ECMP using the BGP Link-Bandwidth extended community, the weights are not correctly applied to the kernel forwarding table if the candidate paths have been leaked from other VRFs.

The setup involves terminating eBGP sessions (from IXs and Transits) in separate, dedicated VRFs. Routes from these VRFs are then leaked into a master BGP instance in a different VRF where best-path selection and multipath calculation occurs.

Although BGP in the master VRF correctly identifies the multipath candidates and even displays the Link-Bandwidth community in show ip ro vrf VRF_MAIN 1.2.3.0/24, the resulting route installed in the kernel by Zebra has weight 1 for all paths, indicating a fallback to standard ECMP.

---

The following is true only for FRR 10.1.3:

The core of the issue becomes apparent when using the bgp bestpath bandwidth command:

When bgp bestpath bandwidth skip-missing is configured, the leaked multipath routes disappear entirely from the kernel routing table.

When bgp bestpath bandwidth default-weight-for-missing is configured, the routes reappear, but all are assigned the default weight 1.

This behavior strongly suggests that the Link-Bandwidth extended community attribute is being lost or invalidated during the VRF route-leaking process. The BGP instance in the master VRF receives the leaked path but seemingly no longer associates it with the bandwidth data, treating it as a path with a "missing" bandwidth attribute. This prevents the correct weights from being calculated and passed to Zebra for kernel installation.

Version

FRRouting 10.1.3 (rt1.sto1.se.as203038.net) on Linux(5.15.0-131-generic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--sbindir=/usr/lib/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--enable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'

---

FRRouting 10.2.1 (rt2.sto1.se.as203038.net) on Linux(5.15.0-131-generic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--sbindir=/usr/lib/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--enable-pim6d' '--disable-grpc' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'

How to reproduce

1. Setup VRFs: Configure multiple VRFs. For example, IX_SONIX, IX_STHIX, and a master VRF, IBGP.
2. Establish eBGP: Establish eBGP sessions in IX_SONIX and IX_STHIX.
3. Set Bandwidth on Ingress: On inbound routes from these eBGP peers, use a route-map to set different extcommunity bandwidth values.

! In IX_STHIX
route-map SET_BW_10G permit 10
set extcommunity bandwidth 10
!
! In IX_SONIX
route-map SET_BW_40G permit 10
set extcommunity bandwidth 40

4. Leak Routes: Configure BGP to leak the routes from the IX VRFs into IBGP.

router bgp 65000 vrf IBGP
 address-family ipv4 unicast
  rd vpn export 65000:1
  rt vpn import 65000:10
 exit-address-family
!
router bgp 65000 vrf IX_SONIX
 address-family ipv4 unicast
  rd vpn export 65000:11
  rt vpn export 65000:10
 exit-address-family
!
router bgp 65000 vrf IX_STHIX
 address-family ipv4 unicast
  rd vpn export 65000:12
  rt vpn export 65000:10
 exit-address-family
!

5. Configure Multipath in Master VRF: In IBGP, configure BGP to allow multipathing between different ASNs and to use the link-bandwidth community.

router bgp 65000 vrf IBGP
 bgp bestpath as-path multipath-relax
 bgp bestpath bandwidth skip-missing # or default-weight-for-missing

6. Observe Results:

# do show bgp vrf IBGP ipv4 unicast 104.21.66.8 multipath
BGP routing table entry for 104.21.64.0/20, version 118446597
Paths: (5 available, best #1, vrf IBGP)
  Imported from 203038:305:104.21.64.0/20
  13335, (aggregated by 13335 10.34.48.128)
    192.121.80.23 from 0.0.0.0 (185.243.23.255) vrf IX_STHIX(14) announce-nh-self
      Origin IGP, metric 100, localpref 200, valid, sourced, local, multipath, best (Local Pref)
      Community: 0:300 0:305
      Extended Community: LB:23456:1250000 (10.000 Mbps) RT:203038:300
      Large Community: 203038:752115101:0
      Originator: 185.243.23.255
      AddPath ID: RX 0, TX-All 3081569 TX-Best-Per-AS 0 TX-Best-Selected 0
      Advertised to: 185.230.223.78 185.243.23.254 178.248.237.29
      Last update: Wed Jul  9 17:41:22 2025
  Imported from 203038:302:104.21.64.0/20
  13335, (aggregated by 13335 10.34.48.128)
    185.1.215.60 from 0.0.0.0 (185.243.23.255) vrf IX_SONIX(13) announce-nh-self
      Origin IGP, metric 100, localpref 200, valid, sourced, local, multipath
      Community: 0:300 0:302
      Extended Community: LB:23456:5000000 (40.000 Mbps) RT:203038:300
      Large Community: 203038:752115104:0
      Originator: 185.243.23.255
      AddPath ID: RX 0, TX-All 409149 TX-Best-Per-AS 0 TX-Best-Selected 0
      Advertised to: 185.230.223.78
      Last update: Wed Jul  9 17:41:25 2025

# do sh ip ro vrf IBGP 104.21.66.8
Routing entry for 104.21.64.0/20
  Known via "bgp", distance 20, metric 100, vrf IBGP, best
  Last update 08:45:04 ago
  * 185.1.215.60, via enp94s0f1np2(vrf IX_SONIX), weight 1
  * 192.121.80.23, via vlan203(vrf IX_STHIX), weight 1

Expected behavior

Weighted ECMP to take effect and have output similar to:

# do sh ip ro vrf IBGP 104.21.66.8
Routing entry for 104.21.64.0/20
  Known via "bgp", distance 20, metric 100, vrf IBGP, best
  Last update 08:45:04 ago
  * 185.1.215.60, via enp94s0f1np2(vrf IX_SONIX), weight 4
  * 192.121.80.23, via vlan203(vrf IX_STHIX), weight 1

Actual behavior

No Weighted ECMP, just standard ECMP happening.

On 10.1.3 however it only works if you specify default-weight-for-missing. Otherwise the routes are just missing altogether.

Additional context

No response

Checklist

• I have searched the open issues for this bug.
• I have not included sensitive information in this report.

[ton31337]

Please show the full configurations, because I'm confused now what is vrf IBGP, and vrf VRF_MAIN...

[f0o]

vrf IBGP is vrf VRF_MAIN in the repro steps. the full config is >3k lines with a lot of unrelated stuff.

I deliberately took output from the production routers to show the issue instead of a lab env.

[f0o]

@ton31337 I've updated the ticket to use IBGP, IX_SONIX, IX_STHIX

[ton31337]

Found the root cause, will fix it soon.