![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure1-1.png\")
The environment can be considered as separate into two regions, where the \u201cLeft Region\u201d consists of the SD-WAN Edge Left-Edge-T3, Partner Gateway (PGW) vcg-40-sfpg01 and vcg-40-sfpg02, and PE routers PE1 and PE2. The \u201cRight Region\u201d consists of the SD-WAN Edge Right-Edge-T3, Partner Gateway vcg-40-nypg01 and PE3.<\/p>\n\n\n\n
The focus of the test is on the Left-Edge-T3 which assigned two partner gateways, with primary PGW vcg-40-sfpg01 and secondary PGW vcg-40-sfpg02. Under normal situation, the traffic path (normal path) from Support-L1 (10.11.41.13) to Support-R2 (10.21.42.23) is: Left-Edge-T3 –> vcg-40-sfpg01 –> PE1 –> PE3 –> vcg-40-nypg01 –> Right-Edge-T3. In case vcg-40-sfpg01 failed, the secondary PGW vcg-40-sfpg02 will pick up the traffic, then the traffic path (backup path) from Support-L1 (10.11.41.13) to Support-R2 (10.21.42.23) will become: Left-Edge-T3 –> vcg-40-sfpg02 –> PE2 -> PE3 –> vcg-40-nypg01 –> Right-Edge-T3. The return path will be ensured to be symmetric, the related configuration will be documented in the configuration section.<\/p>\n\n\n\n
In this post, we will compare the failover time from the normal path to the backup path for with and without BFD between the PGW and PE router.<\/p>\n\n\n\n
Note: The PE router is Cisco IOL in this test environment.<\/p>\n\n\n\n
SD-WAN Edge, PGW and PE router configurations \u2013 without BFD<\/h3>\n\n\n\n
In this section, the configuration of the SD-WAN Edge, SD-WAN Gateway and PE router will be documented for the scenario without BFD.<\/p>\n\n\n\n
Configuration of SD-WAN Edge<\/h4>\n\n\n\nLeft-Edge-T3<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure2-1.png\")
Figure 2: Left-Edge-T3 Cloud VPN and Gateway assignment configuration<\/figcaption><\/figure>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure3.png\")
Figure 3: Left-Edge-T3 VLAN and Interface settings<\/figcaption><\/figure>\n\n\n\nFrom the above screen capture (Figure 2 and 3), Left-Edge-T3 gets assigned with two PGW with primary PGW vcg-40-sfpg01 and secondary PGW vcg-40-sfpg02. The LAN facing interface is GE1 with IP address 10.11.41.254<\/p>\n\n\n\n
Right-Edge-T3<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure4.png\")
Figure 4: Right-Edge-T3 Cloud VPN and Gateway assignment configuration<\/figcaption><\/figure>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure5.png\")
Figure 5: Right-Edge-T3 VLAN and Interface settings<\/figcaption><\/figure>\n\n\n\nFrom the above screen capture, Right-Edge-T3 gets assigned with one PGW vcg-40-nypg01. The LAN facing interface is GE1 with IP address 10.21.42.254. Since there is no PGW failover test will be performed on Right-Edge-T3, so no redundant PGW is assigned.<\/p>\n\n\n\n
Configuration of SD-WAN Gateway, Partner Gateway<\/h4>\n\n\n\nvcg-40-sfpg01<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure6.png\")
Figure 6: vcg-40-sfpg01 configuration<\/figcaption><\/figure>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure7.png\")
Figure 7: vcg-40-sfpg01 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nThe PGW vcg-40-sfpg01 is configured with IP address 10.253.247.2\/29, with AS number 65121. It is peering with PE router PE1 with AS number 9998 and IP address 10.253.247.1. The BGP timer is with default value where keep alive is 60s and hold timers is 180s. The BFD is disabled.<\/p>\n\n\n\n
vcg-40-sfpg02<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure8.png\")
Figure 8: vcg-40-sfpg02 configuration<\/figcaption><\/figure>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure9.png\")
Figure 9: vcg-40-sfpg02 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nThe PGW vcg-40-sfpg02 is configured with IP address 10.253.247.10\/29, with AS number 65122. It is peering with PE router PE2 with AS number 9998 and IP address 10.253.247.9. The BGP timer is with default value where keep alive is 60s and hold timers is 180s. The BFD is disabled.<\/p>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure10.png\")
Figure 10: vcg-40-nypg01 configuration
<\/figcaption><\/figure>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure11.png\")
Figure 11: vcg-40-nypg02 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nIncluding the configuration of the vcg-40-nypg02 for completeness.<\/p>\n\n\n\n
Configuration of PE router<\/h4>\n\n\n\n
Since the focus of this post is the SD-WAN components, not the PE router, only related configuration of the PE router will be show. The VRF being used in the test is called Customer40b.<\/p>\n\n\n\n
PE1<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg02\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nConfiguration to ensure the traffic path is symmetric<\/h5>\n\n\n\n
Although the Left-Edge-T3 is aware the vcg-40-sfpg01 is the primary PGW as it is at order position #1, the MPLS backbone also need to ensure the return traffic prefer landing on PE1 instead of landing on PE2. This is done by auto community mapping at the PGW and route-map to adjust local preference at the PE router.<\/p>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure12.png\")
Figure 12: Community Mapping<\/figcaption><\/figure>\n\n\n\nPGW is configured with community mapping with priority #1 PGW routes come with community 9998:111 (655229039) and priority #2 PGW routes come with community 9998:222 (655229150). That means, for the routes advertised by vcg-40-sfpg01 to the PE1, those routes will be having community 9998:111. For the routes advertised by vcg-40-sfpg02 to the PE2, those routes will be having community 9998:222.<\/p>\n\n\n\n
The following configurations are the route-map comm-set-localpref to make PE1 is the preferred return path as route from vcg-40-sfpg01 is set with local preference of 200 while route from vcg-40-sfpg02 is set with local preference of 101.<\/p>\n\n\n\n
PE1 route-map configuration:<\/p>\n\n\n\n
!\nip community-list standard comm-priority1 permit 655229039\nip community-list standard comm-priority2 permit 655229150\n!\n! Set local preference to 200 when route comes with community 9998:111, set local preference to 101 when route comes with community 9998:222\n!\nroute-map comm-set-localpref permit 10\n match community comm-priority1\n set local-preference 200\n!\nroute-map comm-set-localpref permit 20\n match community comm-priority2\n set local-preference 101\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe same route-map, hence same mechanism is applied to PE2 as well, the configuration of the PE2 route-map is like a mirror copy of PE1, so it is not repeated here.<\/p>\n\n\n\n
Verification on SD-WAN Edge, PGW and PE router<\/h3>\n\n\n\n
The section will perform some verification to ensure the SD-WAN Edge, PGW, PE router has learnt the route properly.<\/p>\n\n\n\n
Verification on SD-WAN Edge<\/h4>\n\n\n\nLeft-Edge-T3:<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure13-1.png\")
Figure 13: Path and 10.21.42.0\/24 route on Left-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.21.42.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Left-Edge-T3 is able to form overlay tunnel to vcg-40-sfpg01 and vcg-40-sfpg02<\/li>
- Left-Edge-T3 learnt the route 10.21.42.0\/24 from both vcg-40-sfpg01 and vcg-40-sfpg02 where vcg-40-sfpg01 is the preferred Next Hop.<\/li><\/ol>\n\n\n\n
Right-Edge-T3:<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure14.png\")
Figure 14: Path and 10.11.41.0\/24 route on Right-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.11.41.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Right-Edge-T3 is able to form overlay tunnels to vcg-40-nypg01.<\/li>
- Right-Edge-T3 learnt the route 10.11.41.0\/24 from vcg-40-nypg01.<\/li><\/ol>\n\n\n\n
Verification on PGW<\/h4>\n\n\n\nvcg-40-sfpg01:<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure15-1.png\")
Figure 15: vcg-40-sfpg01 debug command output of bgp_view_summary and bgp_view <\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 15) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg01 has successfully established BGP peer with PE1 (10.253.247.1)<\/li>
- PGW vcg-40-sfpg01 learnt some routes from PE1, including the route 10.21.42.0\/24<\/li><\/ol>\n\n\n\n
vcg-40-sfpg02:<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure16-1.png\")
Figure 16: vcg-40-sfpg02 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 16) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg02. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg02 has successfully established BGP peer with PE2 (10.253.247.9).<\/li>
- PGW vcg-40-sfpg02 learnt some routes from PE2, including the route 10.21.42.0\/24.<\/li><\/ol>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure17.png\")
Figure 17: cg-40-nypg01 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-nypg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-nypg01 has successfully established BGP peer with PE3 (10.253.246.1).<\/li>
- PGW vcg-40-nypg01 learnt some routes from PE3, including the route 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
Verification on PE router<\/h4>\n\n\n\nPE1<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure18.png\")
Figure 18: PE1, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 18) is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE1. The output confirmed:<\/p>\n\n\n\n
- PE1 successfully having BGP peer with vcg-40-sfpg01 (10.253.247.2).<\/li>
- PE1 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg01.<\/li>
- PE1 successfully learnt the route 10.11.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE2<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure19.png\")
Figure 19: PE2, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE2. The output confirmed:<\/p>\n\n\n\n
- PE2 successfully having BGP peer with vcg-40-sfpg02 (10.253.247.10).<\/li>
- PE2 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg02 (and also from PE1 as PE1 learnt this route from vcg-40-sfpg01).<\/li>
- PE2 successfully learnt the route 10.21.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE3<\/h5>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure20.png\")
Figure 20: PE3, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE3. The output confirmed:<\/p>\n\n\n\n
- PE3 successfully having BGP peer with vcg-40-nypg01 (10.253.246.2).<\/li>
- PE3 successfully learnt the route 10.21.42.0\/24 from PGW vcg-40-nypg01.<\/li>
- PE3 successfully learnt the route 10.11.41.0\/24 from MPLS backbone. Note that the Next Hop is 10.10.10.21 which is the PE1 loopback IP address, the local preference is 200. This means the route-map on PE1 and PE2 is working, such that packets prefer going to PE1 for destination 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
PGW Failover Test \u2013 without BFD<\/h3>\n\n\n\n
The test is to test how long does it take for the traffic to switch from vcg-40-sfpg01 to vcg-40-sfpg02. In order to perform this test, the Linux machine support-L1 (10.11.41.13) issue a continuous ping to support-R2 (10.21.42.23). In the middle of the ping, the vcg-40-sfpg01 will get power off, and the time required to fail over will get recorded.<\/p>\n\n\n\n
Before the ping failover test, a traceroute from support-L1 (10.11.41.13) to support-R2 (10.21.42.23) is capture as follow:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure21.png\")
Figure 21: Client machine Support-L1 (10.11.41.13) traceroute to 10.21.42.23<\/figcaption><\/figure>\n\n\n\nThis traceroute confirmed the traffic path is \u201cLeft-Edge-T3 –> vcg-40-sfpg01 –> PE1 –> PE3 –>vcg-40-nypg01 –> Right-Edge-T3\u201d<\/p>\n\n\n\n
The test method is having Support-L1 (10.11.41.13) issue a continuous ping to Support-R2 (10.21.42.23), in the middle of the ping, PGW vcg-40-sfpg01 gets powered off. The following show the test result:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure22.png\")
Figure 22: Ping test result for primary PGW failed, with default BGP timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 82. There are 129 ping missed during the failover, which is roughly around 130 seconds.<\/p>\n\n\n\n
The reader might wonder why the failover takes so long, since the SD-WAN Edge Left-Edge-T3 is having Overlay Tunnel to both vcg-40-sfpg01 and vcg-40-sfpg02, when vcg-40-sfpg01 gets powered off, the corresponding Overlay Tunnel gets terminated. As a result, Left-Edge-T3 should be able to switch to vcg-40-sfpg02 in sub-second. This understanding is correct. But let\u2019s look at the diagram again to refresh the memory:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure23.png\")
Figure 23: Test Topology<\/figcaption><\/figure>\n\n\n\nThe failover takes long time because of the return traffic. The PE1 is configured to have route map to attract the return traffic. In order for the return traffic to be able to switch from PE1 to PE2, PE1 must stop advertising the 10.11.41.0\/24 route. PE1 will stop advertising when the BGP peer between PE1 and vcg-40-sfpg01 is down, with the default timer (60s keepalive and 180s hold timer), it can take a maximum of 180 seconds for the BGP peer to go down.<\/p>\n\n\n\n
One of the options to speed up the failover is having a smaller value of keepalive and hold timer. The following is the screen capture of repeating the same ping test with 3s keepalive and 9s hold timer:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure24a-1.png\")
Figure 24: Ping test result for primary PGW failed, with 3s keepalive and 9s hold timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 35. There are 9 ping missed during the failover, which the failover is roughly around 9-10 seconds.<\/p>\n\n\n\n
PGW and PE Router Configurations \u2013 with BFD<\/h4>\n\n\n\n
In this section, BFD will be enabled in the SD-WAN Gateways and PE routers. Other devices like the SD-WAN Edge will maintain the existing configuration.<\/p>\n\n\n\n
SD-WAN Gateway (Partner Gateway) \u2013 with BFD<\/h5>\n\n\n\n
The follow two screen captures show the PGW (vcg-40-sfpg01 and vcg-40-sfpg02) BGP peer with BFD enabled. In this test, the receive interval and transmit interval are both 300ms, the multiplier is 3.<\/p>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure25.png\")
Figure 25: vcg-40-sfpg01 BGP peer with 10.253.247.1 BFD enabled<\/figcaption><\/figure>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure26.png\")
Figure 26: vcg-40-sfpg02 BGP peer with 10.253.247.9 BFD enabled<\/figcaption><\/figure>\n\n\n\nPE router \u2013 with BFD<\/h4>\n\n\n\nPE1 configuration by enabling BFD for BGP peer 10.253.247.2<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/em><\/span>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 fall-over bfd<\/em><\/span>\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2 configuration by enabling BFD for BGP peer 10.253.247.10<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/span><\/em>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 fall-over bfd<\/span><\/em>\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe newly added configuration to enable BFD in the PE router is marked with italic<\/span><\/em> in the above configuration snippets.<\/p>\n\n\n\nNOTE: In the PE router sub-interface using for the test vrf, that is vrf Customer40b, the \u201cno bfd echo\u201d command is entered to disable the BFD echo mode. As of the VMware SD-WAN version 4.0, BFD echo mode is not supported, only BFD asynchronous mode is supported.<\/p><\/blockquote>\n\n\n\n
Verification of the BFD status<\/h3>\n\n\n\nVerification of BFD status at the PGW<\/h4>\n\n\n\n
There is now a “Gateway BFD Sessions” tab under “Monitor –> Routing –> BFD”<\/p>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure27.png\")
Figure 27: Gateway BFD Sessions<\/figcaption><\/figure>\n\n\n\nFrom the “Gateway BFD Sessions” GUI, we can see both vcg-40-sfpg01 and vcg-40-sfpg02 are having “UP” state of the BFD with the corresponding BGP peer, which means the BFD is working. In case more detail information is needed, in the PGW shell, there are two commands can check the BFD status, they are \u201cdebug.py –bfd_info\u201d and \u201cdebug.py –bfdd_dump\u201d. The output below is from PGW vcg-40-sfpg01 which shows the BFD peer is with “up” status. The following snippet shows the BFD debug.py output of vcg-40-sfpg01: <\/p>\n\n\n\n
vcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfd_info<\/span>\nEnterpriseId SEG Peer Address Local Address Detect Multiplier Transmit Interval Receive Interval Status\neb032999-da64-4b09-ab57-3b5f9949ffdb 0 10.253.247.1 10.253.247.2 3 300 300 UP\nvcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfdd_dump<\/span>\nshow running-config\nBuilding configuration\u2026\nCurrent configuration:\n!\nfrr version 7.0\nfrr defaults traditional\nhostname vcg-40-sfpg01\nlog file \/var\/log\/bfdd.log\n!\npassword zebra\n!\nline vty\naccess-class vty\n!\nbfd\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nno shutdown\n!\n!\nend\nshow bfd peers\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nID: 3238898200\nRemote ID: 1\nStatus: up\nUptime: 34 minute(s), 48 second(s)\nDiagnostics: ok\nRemote diagnostics: ok\nLocal timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 50ms\nRemote timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 0ms\nshow bfd peers counters\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nControl packet input: 9021 packets\nControl packet output: 81639 packets\nEcho packet input: 0 packets\nEcho packet output: 0 packets\nSession up events: 239\nSession down events: 238\nZebra notifications: 479\nNS UDP connections dump for entr ID: [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nState Recv-Q Send-Q Local Address:Port Peer Address:Port\nUNCONN 0 0 0.0.0.0:3784 0.0.0.0:* users:((\"bfdd\",pid=851,fd=16))\nUNCONN 0 0 0.0.0.0:49142 0.0.0.0:* users:((\"bfdd\",pid=851,fd=17))\nvcadmin@vcg-40-sfpg01:~$<\/pre>\n\n\n\nThe output of the debug commands from vcg-40-sfpg02 is very similar so it won\u2019t be repeated here.<\/p>\n\n\n\n
PE router Verification of BFD status<\/h5>\n\n\n\n
In the PE router, command \u201cshow bfd neighbors details\u201d is used to check the BFD status. The output below from PE1 confirmed the BFD status with PGW vcg-40-sfpg01 is up:<\/p>\n\n\n\n
PE1#show bfd neighbors details\nIPv4 Sessions\nNeighAddr LD\/RD RH\/RS State Int\n10.253.247.2 1\/3238898200 Up Up Et0\/2.341\nSession state is UP and not using echo function.\nSession Host: Software\nOurAddr: 10.253.247.1\nHandle: 1\nLocal Diag: 0, Demand mode: 0, Poll bit: 0\nMinTxInt: 300000, MinRxInt: 300000, Multiplier: 3\nReceived MinRxInt: 300000, Received Multiplier: 3\nHolddown (hits): 829(0), Hello (hits): 300(9183)\nRx Count: 9330, Rx Interval (ms) min\/max\/avg: 1\/301\/263 last: 71 ms ago\nTx Count: 9186, Tx Interval (ms) min\/max\/avg: 1\/310\/267 last: 206 ms ago\nElapsed time watermarks: 0 0 (last: 0)\nRegistered protocols: BGP CEF\nUptime: 00:40:51\nLast packet: Version: 1 - Diagnostic: 0\nState bit: Up - Demand bit: 0\nPoll bit: 0 - Final bit: 0\nC bit: 0\nMultiplier: 3 - Length: 24\nMy Discr.: 3238898200 - Your Discr.: 1\nMin tx interval: 300000 - Min rx interval: 300000\nMin Echo interval: 50000\nPE1#<\/pre>\n\n\n\nAgain, the output from PE2 is very similar to PE1 so it won\u2019t be included here.<\/p>\n\n\n\n
PGW Failover Test \u2013 with BFD<\/h3>\n\n\n\n
With BFD enabled between the PGW and PE router, let\u2019s repeat the ping Test from client support-L1 (10.11.41.13) to support-R2 (10.21.42.23), with a power off of primary PGW vcg-40-sfpg01. The following screen capture is the ping test result:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure28.png\")
Figure 28: Ping test result from Support-L1 (10.11.41.13) to Support-R2 (10.21.42.23) with primary PGW failure, BFD is enabled in the PGW<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the power off of the primary PGW vcg-40-sfpg01 cause 1 ping loss (between icmp sequence 70 and 72). The failover time is around 1-2 seconds. Since the failover time is a lot faster. The ping test is being performed again with the additional parameter \u201c-i 0.1\u201d, this means the ping will be send every 0.1s instead of the default which is sent every 1s. The following is the screen capture of the ping test result with 0.1s interval:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.sdwan2.com\/wp-content\/uploads\/2020\/10\/figure29.png\")
Figure 29: Repeat of the ping test with primary PGW vcg-40-sfpg01, the ping interval is lower to 0.1s<\/figcaption><\/figure>\n\n\n\nThere are 8 ping loss (between sequence number 22 and 31), with each ping separate of 0.1s, this means 9 x 0.1s interval which result of 0.9s. As a result, it should be fine to conclude the failover time of the primary PGW failure is 1s with the BFD enabled BGP between PGW and PE router.<\/p>\n\n\n\n
Maximum BFD session supported per PGW<\/h3>\n\n\n\n
As of version 4.0, the maximum number of BFD sessions supported on PGW is 250. Although this post is not talking about the BFD on SD-WAN Edge, for documentation purpose, the maximum number of BFD sessions supported in the SD-WAN Edge is 50.<\/p>\n\n\n\n
This post “BGP with BFD enabled in SD-WAN Partner Gateway” ended here.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"Background In VMware SD-WAN (Velocloud) version 4.0, both SD-WAN Edge and SD-WAN Gateway added the support of Bidirectional Forwarding Detection (BFD). The objective of this post is to document the configuration of enabling BFD for the BGP peering between the SD-WAN Gateway (working as Partner Gateway) and PE router. There will be ping test result […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"templates\/template-fullwidth.php","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[9,8,5],"tags":[],"class_list":["post-168","post","type-post","status-publish","format-standard","hentry","category-partner-gateway","category-sd-wan-routing","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":36,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":241,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions\/241"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
From the above screen capture (Figure 2 and 3), Left-Edge-T3 gets assigned with two PGW with primary PGW vcg-40-sfpg01 and secondary PGW vcg-40-sfpg02. The LAN facing interface is GE1 with IP address 10.11.41.254<\/p>\n\n\n\n
Right-Edge-T3<\/h5>\n\n\n\nFigure 4: Right-Edge-T3 Cloud VPN and Gateway assignment configuration<\/figcaption><\/figure>\n\n\n\nFigure 5: Right-Edge-T3 VLAN and Interface settings<\/figcaption><\/figure>\n\n\n\nFrom the above screen capture, Right-Edge-T3 gets assigned with one PGW vcg-40-nypg01. The LAN facing interface is GE1 with IP address 10.21.42.254. Since there is no PGW failover test will be performed on Right-Edge-T3, so no redundant PGW is assigned.<\/p>\n\n\n\n
Configuration of SD-WAN Gateway, Partner Gateway<\/h4>\n\n\n\nvcg-40-sfpg01<\/h5>\n\n\n\nFigure 6: vcg-40-sfpg01 configuration<\/figcaption><\/figure>\n\n\n\nFigure 7: vcg-40-sfpg01 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nThe PGW vcg-40-sfpg01 is configured with IP address 10.253.247.2\/29, with AS number 65121. It is peering with PE router PE1 with AS number 9998 and IP address 10.253.247.1. The BGP timer is with default value where keep alive is 60s and hold timers is 180s. The BFD is disabled.<\/p>\n\n\n\n
vcg-40-sfpg02<\/h5>\n\n\n\nFigure 8: vcg-40-sfpg02 configuration<\/figcaption><\/figure>\n\n\n\nFigure 9: vcg-40-sfpg02 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nThe PGW vcg-40-sfpg02 is configured with IP address 10.253.247.10\/29, with AS number 65122. It is peering with PE router PE2 with AS number 9998 and IP address 10.253.247.9. The BGP timer is with default value where keep alive is 60s and hold timers is 180s. The BFD is disabled.<\/p>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 10: vcg-40-nypg01 configuration
<\/figcaption><\/figure>\n\n\n\nFigure 11: vcg-40-nypg02 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nIncluding the configuration of the vcg-40-nypg02 for completeness.<\/p>\n\n\n\n
Configuration of PE router<\/h4>\n\n\n\n
Since the focus of this post is the SD-WAN components, not the PE router, only related configuration of the PE router will be show. The VRF being used in the test is called Customer40b.<\/p>\n\n\n\n
PE1<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg02\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nConfiguration to ensure the traffic path is symmetric<\/h5>\n\n\n\n
Although the Left-Edge-T3 is aware the vcg-40-sfpg01 is the primary PGW as it is at order position #1, the MPLS backbone also need to ensure the return traffic prefer landing on PE1 instead of landing on PE2. This is done by auto community mapping at the PGW and route-map to adjust local preference at the PE router.<\/p>\n\n\n\nFigure 12: Community Mapping<\/figcaption><\/figure>\n\n\n\nPGW is configured with community mapping with priority #1 PGW routes come with community 9998:111 (655229039) and priority #2 PGW routes come with community 9998:222 (655229150). That means, for the routes advertised by vcg-40-sfpg01 to the PE1, those routes will be having community 9998:111. For the routes advertised by vcg-40-sfpg02 to the PE2, those routes will be having community 9998:222.<\/p>\n\n\n\n
The following configurations are the route-map comm-set-localpref to make PE1 is the preferred return path as route from vcg-40-sfpg01 is set with local preference of 200 while route from vcg-40-sfpg02 is set with local preference of 101.<\/p>\n\n\n\n
PE1 route-map configuration:<\/p>\n\n\n\n
!\nip community-list standard comm-priority1 permit 655229039\nip community-list standard comm-priority2 permit 655229150\n!\n! Set local preference to 200 when route comes with community 9998:111, set local preference to 101 when route comes with community 9998:222\n!\nroute-map comm-set-localpref permit 10\n match community comm-priority1\n set local-preference 200\n!\nroute-map comm-set-localpref permit 20\n match community comm-priority2\n set local-preference 101\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe same route-map, hence same mechanism is applied to PE2 as well, the configuration of the PE2 route-map is like a mirror copy of PE1, so it is not repeated here.<\/p>\n\n\n\n
Verification on SD-WAN Edge, PGW and PE router<\/h3>\n\n\n\n
The section will perform some verification to ensure the SD-WAN Edge, PGW, PE router has learnt the route properly.<\/p>\n\n\n\n
Verification on SD-WAN Edge<\/h4>\n\n\n\nLeft-Edge-T3:<\/h5>\n\n\n\nFigure 13: Path and 10.21.42.0\/24 route on Left-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.21.42.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Left-Edge-T3 is able to form overlay tunnel to vcg-40-sfpg01 and vcg-40-sfpg02<\/li>
- Left-Edge-T3 learnt the route 10.21.42.0\/24 from both vcg-40-sfpg01 and vcg-40-sfpg02 where vcg-40-sfpg01 is the preferred Next Hop.<\/li><\/ol>\n\n\n\n
Right-Edge-T3:<\/h5>\n\n\n\nFigure 14: Path and 10.11.41.0\/24 route on Right-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.11.41.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Right-Edge-T3 is able to form overlay tunnels to vcg-40-nypg01.<\/li>
- Right-Edge-T3 learnt the route 10.11.41.0\/24 from vcg-40-nypg01.<\/li><\/ol>\n\n\n\n
Verification on PGW<\/h4>\n\n\n\nvcg-40-sfpg01:<\/h5>\n\n\n\nFigure 15: vcg-40-sfpg01 debug command output of bgp_view_summary and bgp_view <\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 15) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg01 has successfully established BGP peer with PE1 (10.253.247.1)<\/li>
- PGW vcg-40-sfpg01 learnt some routes from PE1, including the route 10.21.42.0\/24<\/li><\/ol>\n\n\n\n
vcg-40-sfpg02:<\/h5>\n\n\n\nFigure 16: vcg-40-sfpg02 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 16) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg02. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg02 has successfully established BGP peer with PE2 (10.253.247.9).<\/li>
- PGW vcg-40-sfpg02 learnt some routes from PE2, including the route 10.21.42.0\/24.<\/li><\/ol>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 17: cg-40-nypg01 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-nypg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-nypg01 has successfully established BGP peer with PE3 (10.253.246.1).<\/li>
- PGW vcg-40-nypg01 learnt some routes from PE3, including the route 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
Verification on PE router<\/h4>\n\n\n\nPE1<\/h5>\n\n\n\nFigure 18: PE1, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 18) is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE1. The output confirmed:<\/p>\n\n\n\n
- PE1 successfully having BGP peer with vcg-40-sfpg01 (10.253.247.2).<\/li>
- PE1 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg01.<\/li>
- PE1 successfully learnt the route 10.11.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE2<\/h5>\n\n\n\nFigure 19: PE2, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE2. The output confirmed:<\/p>\n\n\n\n
- PE2 successfully having BGP peer with vcg-40-sfpg02 (10.253.247.10).<\/li>
- PE2 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg02 (and also from PE1 as PE1 learnt this route from vcg-40-sfpg01).<\/li>
- PE2 successfully learnt the route 10.21.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE3<\/h5>\n\n\n\nFigure 20: PE3, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE3. The output confirmed:<\/p>\n\n\n\n
- PE3 successfully having BGP peer with vcg-40-nypg01 (10.253.246.2).<\/li>
- PE3 successfully learnt the route 10.21.42.0\/24 from PGW vcg-40-nypg01.<\/li>
- PE3 successfully learnt the route 10.11.41.0\/24 from MPLS backbone. Note that the Next Hop is 10.10.10.21 which is the PE1 loopback IP address, the local preference is 200. This means the route-map on PE1 and PE2 is working, such that packets prefer going to PE1 for destination 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
PGW Failover Test \u2013 without BFD<\/h3>\n\n\n\n
The test is to test how long does it take for the traffic to switch from vcg-40-sfpg01 to vcg-40-sfpg02. In order to perform this test, the Linux machine support-L1 (10.11.41.13) issue a continuous ping to support-R2 (10.21.42.23). In the middle of the ping, the vcg-40-sfpg01 will get power off, and the time required to fail over will get recorded.<\/p>\n\n\n\n
Before the ping failover test, a traceroute from support-L1 (10.11.41.13) to support-R2 (10.21.42.23) is capture as follow:<\/p>\n\n\n\nFigure 21: Client machine Support-L1 (10.11.41.13) traceroute to 10.21.42.23<\/figcaption><\/figure>\n\n\n\nThis traceroute confirmed the traffic path is \u201cLeft-Edge-T3 –> vcg-40-sfpg01 –> PE1 –> PE3 –>vcg-40-nypg01 –> Right-Edge-T3\u201d<\/p>\n\n\n\n
The test method is having Support-L1 (10.11.41.13) issue a continuous ping to Support-R2 (10.21.42.23), in the middle of the ping, PGW vcg-40-sfpg01 gets powered off. The following show the test result:<\/p>\n\n\n\nFigure 22: Ping test result for primary PGW failed, with default BGP timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 82. There are 129 ping missed during the failover, which is roughly around 130 seconds.<\/p>\n\n\n\n
The reader might wonder why the failover takes so long, since the SD-WAN Edge Left-Edge-T3 is having Overlay Tunnel to both vcg-40-sfpg01 and vcg-40-sfpg02, when vcg-40-sfpg01 gets powered off, the corresponding Overlay Tunnel gets terminated. As a result, Left-Edge-T3 should be able to switch to vcg-40-sfpg02 in sub-second. This understanding is correct. But let\u2019s look at the diagram again to refresh the memory:<\/p>\n\n\n\nFigure 23: Test Topology<\/figcaption><\/figure>\n\n\n\nThe failover takes long time because of the return traffic. The PE1 is configured to have route map to attract the return traffic. In order for the return traffic to be able to switch from PE1 to PE2, PE1 must stop advertising the 10.11.41.0\/24 route. PE1 will stop advertising when the BGP peer between PE1 and vcg-40-sfpg01 is down, with the default timer (60s keepalive and 180s hold timer), it can take a maximum of 180 seconds for the BGP peer to go down.<\/p>\n\n\n\n
One of the options to speed up the failover is having a smaller value of keepalive and hold timer. The following is the screen capture of repeating the same ping test with 3s keepalive and 9s hold timer:<\/p>\n\n\n\nFigure 24: Ping test result for primary PGW failed, with 3s keepalive and 9s hold timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 35. There are 9 ping missed during the failover, which the failover is roughly around 9-10 seconds.<\/p>\n\n\n\n
PGW and PE Router Configurations \u2013 with BFD<\/h4>\n\n\n\n
In this section, BFD will be enabled in the SD-WAN Gateways and PE routers. Other devices like the SD-WAN Edge will maintain the existing configuration.<\/p>\n\n\n\n
SD-WAN Gateway (Partner Gateway) \u2013 with BFD<\/h5>\n\n\n\n
The follow two screen captures show the PGW (vcg-40-sfpg01 and vcg-40-sfpg02) BGP peer with BFD enabled. In this test, the receive interval and transmit interval are both 300ms, the multiplier is 3.<\/p>\n\n\n\nFigure 25: vcg-40-sfpg01 BGP peer with 10.253.247.1 BFD enabled<\/figcaption><\/figure>\n\n\n\nFigure 26: vcg-40-sfpg02 BGP peer with 10.253.247.9 BFD enabled<\/figcaption><\/figure>\n\n\n\nPE router \u2013 with BFD<\/h4>\n\n\n\nPE1 configuration by enabling BFD for BGP peer 10.253.247.2<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/em><\/span>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 fall-over bfd<\/em><\/span>\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2 configuration by enabling BFD for BGP peer 10.253.247.10<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/span><\/em>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 fall-over bfd<\/span><\/em>\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe newly added configuration to enable BFD in the PE router is marked with italic<\/span><\/em> in the above configuration snippets.<\/p>\n\n\n\nNOTE: In the PE router sub-interface using for the test vrf, that is vrf Customer40b, the \u201cno bfd echo\u201d command is entered to disable the BFD echo mode. As of the VMware SD-WAN version 4.0, BFD echo mode is not supported, only BFD asynchronous mode is supported.<\/p><\/blockquote>\n\n\n\n
Verification of the BFD status<\/h3>\n\n\n\nVerification of BFD status at the PGW<\/h4>\n\n\n\n
There is now a “Gateway BFD Sessions” tab under “Monitor –> Routing –> BFD”<\/p>\n\n\n\nFigure 27: Gateway BFD Sessions<\/figcaption><\/figure>\n\n\n\nFrom the “Gateway BFD Sessions” GUI, we can see both vcg-40-sfpg01 and vcg-40-sfpg02 are having “UP” state of the BFD with the corresponding BGP peer, which means the BFD is working. In case more detail information is needed, in the PGW shell, there are two commands can check the BFD status, they are \u201cdebug.py –bfd_info\u201d and \u201cdebug.py –bfdd_dump\u201d. The output below is from PGW vcg-40-sfpg01 which shows the BFD peer is with “up” status. The following snippet shows the BFD debug.py output of vcg-40-sfpg01: <\/p>\n\n\n\n
vcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfd_info<\/span>\nEnterpriseId SEG Peer Address Local Address Detect Multiplier Transmit Interval Receive Interval Status\neb032999-da64-4b09-ab57-3b5f9949ffdb 0 10.253.247.1 10.253.247.2 3 300 300 UP\nvcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfdd_dump<\/span>\nshow running-config\nBuilding configuration\u2026\nCurrent configuration:\n!\nfrr version 7.0\nfrr defaults traditional\nhostname vcg-40-sfpg01\nlog file \/var\/log\/bfdd.log\n!\npassword zebra\n!\nline vty\naccess-class vty\n!\nbfd\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nno shutdown\n!\n!\nend\nshow bfd peers\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nID: 3238898200\nRemote ID: 1\nStatus: up\nUptime: 34 minute(s), 48 second(s)\nDiagnostics: ok\nRemote diagnostics: ok\nLocal timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 50ms\nRemote timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 0ms\nshow bfd peers counters\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nControl packet input: 9021 packets\nControl packet output: 81639 packets\nEcho packet input: 0 packets\nEcho packet output: 0 packets\nSession up events: 239\nSession down events: 238\nZebra notifications: 479\nNS UDP connections dump for entr ID: [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nState Recv-Q Send-Q Local Address:Port Peer Address:Port\nUNCONN 0 0 0.0.0.0:3784 0.0.0.0:* users:((\"bfdd\",pid=851,fd=16))\nUNCONN 0 0 0.0.0.0:49142 0.0.0.0:* users:((\"bfdd\",pid=851,fd=17))\nvcadmin@vcg-40-sfpg01:~$<\/pre>\n\n\n\nThe output of the debug commands from vcg-40-sfpg02 is very similar so it won\u2019t be repeated here.<\/p>\n\n\n\n
PE router Verification of BFD status<\/h5>\n\n\n\n
In the PE router, command \u201cshow bfd neighbors details\u201d is used to check the BFD status. The output below from PE1 confirmed the BFD status with PGW vcg-40-sfpg01 is up:<\/p>\n\n\n\n
PE1#show bfd neighbors details\nIPv4 Sessions\nNeighAddr LD\/RD RH\/RS State Int\n10.253.247.2 1\/3238898200 Up Up Et0\/2.341\nSession state is UP and not using echo function.\nSession Host: Software\nOurAddr: 10.253.247.1\nHandle: 1\nLocal Diag: 0, Demand mode: 0, Poll bit: 0\nMinTxInt: 300000, MinRxInt: 300000, Multiplier: 3\nReceived MinRxInt: 300000, Received Multiplier: 3\nHolddown (hits): 829(0), Hello (hits): 300(9183)\nRx Count: 9330, Rx Interval (ms) min\/max\/avg: 1\/301\/263 last: 71 ms ago\nTx Count: 9186, Tx Interval (ms) min\/max\/avg: 1\/310\/267 last: 206 ms ago\nElapsed time watermarks: 0 0 (last: 0)\nRegistered protocols: BGP CEF\nUptime: 00:40:51\nLast packet: Version: 1 - Diagnostic: 0\nState bit: Up - Demand bit: 0\nPoll bit: 0 - Final bit: 0\nC bit: 0\nMultiplier: 3 - Length: 24\nMy Discr.: 3238898200 - Your Discr.: 1\nMin tx interval: 300000 - Min rx interval: 300000\nMin Echo interval: 50000\nPE1#<\/pre>\n\n\n\nAgain, the output from PE2 is very similar to PE1 so it won\u2019t be included here.<\/p>\n\n\n\n
PGW Failover Test \u2013 with BFD<\/h3>\n\n\n\n
With BFD enabled between the PGW and PE router, let\u2019s repeat the ping Test from client support-L1 (10.11.41.13) to support-R2 (10.21.42.23), with a power off of primary PGW vcg-40-sfpg01. The following screen capture is the ping test result:<\/p>\n\n\n\nFigure 28: Ping test result from Support-L1 (10.11.41.13) to Support-R2 (10.21.42.23) with primary PGW failure, BFD is enabled in the PGW<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the power off of the primary PGW vcg-40-sfpg01 cause 1 ping loss (between icmp sequence 70 and 72). The failover time is around 1-2 seconds. Since the failover time is a lot faster. The ping test is being performed again with the additional parameter \u201c-i 0.1\u201d, this means the ping will be send every 0.1s instead of the default which is sent every 1s. The following is the screen capture of the ping test result with 0.1s interval:<\/p>\n\n\n\nFigure 29: Repeat of the ping test with primary PGW vcg-40-sfpg01, the ping interval is lower to 0.1s<\/figcaption><\/figure>\n\n\n\nThere are 8 ping loss (between sequence number 22 and 31), with each ping separate of 0.1s, this means 9 x 0.1s interval which result of 0.9s. As a result, it should be fine to conclude the failover time of the primary PGW failure is 1s with the BFD enabled BGP between PGW and PE router.<\/p>\n\n\n\n
Maximum BFD session supported per PGW<\/h3>\n\n\n\n
As of version 4.0, the maximum number of BFD sessions supported on PGW is 250. Although this post is not talking about the BFD on SD-WAN Edge, for documentation purpose, the maximum number of BFD sessions supported in the SD-WAN Edge is 50.<\/p>\n\n\n\n
This post “BGP with BFD enabled in SD-WAN Partner Gateway” ended here.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"Background In VMware SD-WAN (Velocloud) version 4.0, both SD-WAN Edge and SD-WAN Gateway added the support of Bidirectional Forwarding Detection (BFD). The objective of this post is to document the configuration of enabling BFD for the BGP peering between the SD-WAN Gateway (working as Partner Gateway) and PE router. There will be ping test result […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"templates\/template-fullwidth.php","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[9,8,5],"tags":[],"class_list":["post-168","post","type-post","status-publish","format-standard","hentry","category-partner-gateway","category-sd-wan-routing","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":36,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":241,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions\/241"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
From the above screen capture, Right-Edge-T3 gets assigned with one PGW vcg-40-nypg01. The LAN facing interface is GE1 with IP address 10.21.42.254. Since there is no PGW failover test will be performed on Right-Edge-T3, so no redundant PGW is assigned.<\/p>\n\n\n\n
Configuration of SD-WAN Gateway, Partner Gateway<\/h4>\n\n\n\nvcg-40-sfpg01<\/h5>\n\n\n\nFigure 6: vcg-40-sfpg01 configuration<\/figcaption><\/figure>\n\n\n\nFigure 7: vcg-40-sfpg01 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nThe PGW vcg-40-sfpg01 is configured with IP address 10.253.247.2\/29, with AS number 65121. It is peering with PE router PE1 with AS number 9998 and IP address 10.253.247.1. The BGP timer is with default value where keep alive is 60s and hold timers is 180s. The BFD is disabled.<\/p>\n\n\n\n
vcg-40-sfpg02<\/h5>\n\n\n\nFigure 8: vcg-40-sfpg02 configuration<\/figcaption><\/figure>\n\n\n\nFigure 9: vcg-40-sfpg02 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nThe PGW vcg-40-sfpg02 is configured with IP address 10.253.247.10\/29, with AS number 65122. It is peering with PE router PE2 with AS number 9998 and IP address 10.253.247.9. The BGP timer is with default value where keep alive is 60s and hold timers is 180s. The BFD is disabled.<\/p>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 10: vcg-40-nypg01 configuration
<\/figcaption><\/figure>\n\n\n\nFigure 11: vcg-40-nypg02 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nIncluding the configuration of the vcg-40-nypg02 for completeness.<\/p>\n\n\n\n
Configuration of PE router<\/h4>\n\n\n\n
Since the focus of this post is the SD-WAN components, not the PE router, only related configuration of the PE router will be show. The VRF being used in the test is called Customer40b.<\/p>\n\n\n\n
PE1<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg02\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nConfiguration to ensure the traffic path is symmetric<\/h5>\n\n\n\n
Although the Left-Edge-T3 is aware the vcg-40-sfpg01 is the primary PGW as it is at order position #1, the MPLS backbone also need to ensure the return traffic prefer landing on PE1 instead of landing on PE2. This is done by auto community mapping at the PGW and route-map to adjust local preference at the PE router.<\/p>\n\n\n\nFigure 12: Community Mapping<\/figcaption><\/figure>\n\n\n\nPGW is configured with community mapping with priority #1 PGW routes come with community 9998:111 (655229039) and priority #2 PGW routes come with community 9998:222 (655229150). That means, for the routes advertised by vcg-40-sfpg01 to the PE1, those routes will be having community 9998:111. For the routes advertised by vcg-40-sfpg02 to the PE2, those routes will be having community 9998:222.<\/p>\n\n\n\n
The following configurations are the route-map comm-set-localpref to make PE1 is the preferred return path as route from vcg-40-sfpg01 is set with local preference of 200 while route from vcg-40-sfpg02 is set with local preference of 101.<\/p>\n\n\n\n
PE1 route-map configuration:<\/p>\n\n\n\n
!\nip community-list standard comm-priority1 permit 655229039\nip community-list standard comm-priority2 permit 655229150\n!\n! Set local preference to 200 when route comes with community 9998:111, set local preference to 101 when route comes with community 9998:222\n!\nroute-map comm-set-localpref permit 10\n match community comm-priority1\n set local-preference 200\n!\nroute-map comm-set-localpref permit 20\n match community comm-priority2\n set local-preference 101\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe same route-map, hence same mechanism is applied to PE2 as well, the configuration of the PE2 route-map is like a mirror copy of PE1, so it is not repeated here.<\/p>\n\n\n\n
Verification on SD-WAN Edge, PGW and PE router<\/h3>\n\n\n\n
The section will perform some verification to ensure the SD-WAN Edge, PGW, PE router has learnt the route properly.<\/p>\n\n\n\n
Verification on SD-WAN Edge<\/h4>\n\n\n\nLeft-Edge-T3:<\/h5>\n\n\n\nFigure 13: Path and 10.21.42.0\/24 route on Left-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.21.42.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Left-Edge-T3 is able to form overlay tunnel to vcg-40-sfpg01 and vcg-40-sfpg02<\/li>
- Left-Edge-T3 learnt the route 10.21.42.0\/24 from both vcg-40-sfpg01 and vcg-40-sfpg02 where vcg-40-sfpg01 is the preferred Next Hop.<\/li><\/ol>\n\n\n\n
Right-Edge-T3:<\/h5>\n\n\n\nFigure 14: Path and 10.11.41.0\/24 route on Right-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.11.41.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Right-Edge-T3 is able to form overlay tunnels to vcg-40-nypg01.<\/li>
- Right-Edge-T3 learnt the route 10.11.41.0\/24 from vcg-40-nypg01.<\/li><\/ol>\n\n\n\n
Verification on PGW<\/h4>\n\n\n\nvcg-40-sfpg01:<\/h5>\n\n\n\nFigure 15: vcg-40-sfpg01 debug command output of bgp_view_summary and bgp_view <\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 15) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg01 has successfully established BGP peer with PE1 (10.253.247.1)<\/li>
- PGW vcg-40-sfpg01 learnt some routes from PE1, including the route 10.21.42.0\/24<\/li><\/ol>\n\n\n\n
vcg-40-sfpg02:<\/h5>\n\n\n\nFigure 16: vcg-40-sfpg02 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 16) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg02. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg02 has successfully established BGP peer with PE2 (10.253.247.9).<\/li>
- PGW vcg-40-sfpg02 learnt some routes from PE2, including the route 10.21.42.0\/24.<\/li><\/ol>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 17: cg-40-nypg01 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-nypg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-nypg01 has successfully established BGP peer with PE3 (10.253.246.1).<\/li>
- PGW vcg-40-nypg01 learnt some routes from PE3, including the route 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
Verification on PE router<\/h4>\n\n\n\nPE1<\/h5>\n\n\n\nFigure 18: PE1, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 18) is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE1. The output confirmed:<\/p>\n\n\n\n
- PE1 successfully having BGP peer with vcg-40-sfpg01 (10.253.247.2).<\/li>
- PE1 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg01.<\/li>
- PE1 successfully learnt the route 10.11.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE2<\/h5>\n\n\n\nFigure 19: PE2, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE2. The output confirmed:<\/p>\n\n\n\n
- PE2 successfully having BGP peer with vcg-40-sfpg02 (10.253.247.10).<\/li>
- PE2 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg02 (and also from PE1 as PE1 learnt this route from vcg-40-sfpg01).<\/li>
- PE2 successfully learnt the route 10.21.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE3<\/h5>\n\n\n\nFigure 20: PE3, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE3. The output confirmed:<\/p>\n\n\n\n
- PE3 successfully having BGP peer with vcg-40-nypg01 (10.253.246.2).<\/li>
- PE3 successfully learnt the route 10.21.42.0\/24 from PGW vcg-40-nypg01.<\/li>
- PE3 successfully learnt the route 10.11.41.0\/24 from MPLS backbone. Note that the Next Hop is 10.10.10.21 which is the PE1 loopback IP address, the local preference is 200. This means the route-map on PE1 and PE2 is working, such that packets prefer going to PE1 for destination 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
PGW Failover Test \u2013 without BFD<\/h3>\n\n\n\n
The test is to test how long does it take for the traffic to switch from vcg-40-sfpg01 to vcg-40-sfpg02. In order to perform this test, the Linux machine support-L1 (10.11.41.13) issue a continuous ping to support-R2 (10.21.42.23). In the middle of the ping, the vcg-40-sfpg01 will get power off, and the time required to fail over will get recorded.<\/p>\n\n\n\n
Before the ping failover test, a traceroute from support-L1 (10.11.41.13) to support-R2 (10.21.42.23) is capture as follow:<\/p>\n\n\n\nFigure 21: Client machine Support-L1 (10.11.41.13) traceroute to 10.21.42.23<\/figcaption><\/figure>\n\n\n\nThis traceroute confirmed the traffic path is \u201cLeft-Edge-T3 –> vcg-40-sfpg01 –> PE1 –> PE3 –>vcg-40-nypg01 –> Right-Edge-T3\u201d<\/p>\n\n\n\n
The test method is having Support-L1 (10.11.41.13) issue a continuous ping to Support-R2 (10.21.42.23), in the middle of the ping, PGW vcg-40-sfpg01 gets powered off. The following show the test result:<\/p>\n\n\n\nFigure 22: Ping test result for primary PGW failed, with default BGP timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 82. There are 129 ping missed during the failover, which is roughly around 130 seconds.<\/p>\n\n\n\n
The reader might wonder why the failover takes so long, since the SD-WAN Edge Left-Edge-T3 is having Overlay Tunnel to both vcg-40-sfpg01 and vcg-40-sfpg02, when vcg-40-sfpg01 gets powered off, the corresponding Overlay Tunnel gets terminated. As a result, Left-Edge-T3 should be able to switch to vcg-40-sfpg02 in sub-second. This understanding is correct. But let\u2019s look at the diagram again to refresh the memory:<\/p>\n\n\n\nFigure 23: Test Topology<\/figcaption><\/figure>\n\n\n\nThe failover takes long time because of the return traffic. The PE1 is configured to have route map to attract the return traffic. In order for the return traffic to be able to switch from PE1 to PE2, PE1 must stop advertising the 10.11.41.0\/24 route. PE1 will stop advertising when the BGP peer between PE1 and vcg-40-sfpg01 is down, with the default timer (60s keepalive and 180s hold timer), it can take a maximum of 180 seconds for the BGP peer to go down.<\/p>\n\n\n\n
One of the options to speed up the failover is having a smaller value of keepalive and hold timer. The following is the screen capture of repeating the same ping test with 3s keepalive and 9s hold timer:<\/p>\n\n\n\nFigure 24: Ping test result for primary PGW failed, with 3s keepalive and 9s hold timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 35. There are 9 ping missed during the failover, which the failover is roughly around 9-10 seconds.<\/p>\n\n\n\n
PGW and PE Router Configurations \u2013 with BFD<\/h4>\n\n\n\n
In this section, BFD will be enabled in the SD-WAN Gateways and PE routers. Other devices like the SD-WAN Edge will maintain the existing configuration.<\/p>\n\n\n\n
SD-WAN Gateway (Partner Gateway) \u2013 with BFD<\/h5>\n\n\n\n
The follow two screen captures show the PGW (vcg-40-sfpg01 and vcg-40-sfpg02) BGP peer with BFD enabled. In this test, the receive interval and transmit interval are both 300ms, the multiplier is 3.<\/p>\n\n\n\nFigure 25: vcg-40-sfpg01 BGP peer with 10.253.247.1 BFD enabled<\/figcaption><\/figure>\n\n\n\nFigure 26: vcg-40-sfpg02 BGP peer with 10.253.247.9 BFD enabled<\/figcaption><\/figure>\n\n\n\nPE router \u2013 with BFD<\/h4>\n\n\n\nPE1 configuration by enabling BFD for BGP peer 10.253.247.2<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/em><\/span>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 fall-over bfd<\/em><\/span>\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2 configuration by enabling BFD for BGP peer 10.253.247.10<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/span><\/em>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 fall-over bfd<\/span><\/em>\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe newly added configuration to enable BFD in the PE router is marked with italic<\/span><\/em> in the above configuration snippets.<\/p>\n\n\n\nNOTE: In the PE router sub-interface using for the test vrf, that is vrf Customer40b, the \u201cno bfd echo\u201d command is entered to disable the BFD echo mode. As of the VMware SD-WAN version 4.0, BFD echo mode is not supported, only BFD asynchronous mode is supported.<\/p><\/blockquote>\n\n\n\n
Verification of the BFD status<\/h3>\n\n\n\nVerification of BFD status at the PGW<\/h4>\n\n\n\n
There is now a “Gateway BFD Sessions” tab under “Monitor –> Routing –> BFD”<\/p>\n\n\n\nFigure 27: Gateway BFD Sessions<\/figcaption><\/figure>\n\n\n\nFrom the “Gateway BFD Sessions” GUI, we can see both vcg-40-sfpg01 and vcg-40-sfpg02 are having “UP” state of the BFD with the corresponding BGP peer, which means the BFD is working. In case more detail information is needed, in the PGW shell, there are two commands can check the BFD status, they are \u201cdebug.py –bfd_info\u201d and \u201cdebug.py –bfdd_dump\u201d. The output below is from PGW vcg-40-sfpg01 which shows the BFD peer is with “up” status. The following snippet shows the BFD debug.py output of vcg-40-sfpg01: <\/p>\n\n\n\n
vcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfd_info<\/span>\nEnterpriseId SEG Peer Address Local Address Detect Multiplier Transmit Interval Receive Interval Status\neb032999-da64-4b09-ab57-3b5f9949ffdb 0 10.253.247.1 10.253.247.2 3 300 300 UP\nvcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfdd_dump<\/span>\nshow running-config\nBuilding configuration\u2026\nCurrent configuration:\n!\nfrr version 7.0\nfrr defaults traditional\nhostname vcg-40-sfpg01\nlog file \/var\/log\/bfdd.log\n!\npassword zebra\n!\nline vty\naccess-class vty\n!\nbfd\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nno shutdown\n!\n!\nend\nshow bfd peers\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nID: 3238898200\nRemote ID: 1\nStatus: up\nUptime: 34 minute(s), 48 second(s)\nDiagnostics: ok\nRemote diagnostics: ok\nLocal timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 50ms\nRemote timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 0ms\nshow bfd peers counters\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nControl packet input: 9021 packets\nControl packet output: 81639 packets\nEcho packet input: 0 packets\nEcho packet output: 0 packets\nSession up events: 239\nSession down events: 238\nZebra notifications: 479\nNS UDP connections dump for entr ID: [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nState Recv-Q Send-Q Local Address:Port Peer Address:Port\nUNCONN 0 0 0.0.0.0:3784 0.0.0.0:* users:((\"bfdd\",pid=851,fd=16))\nUNCONN 0 0 0.0.0.0:49142 0.0.0.0:* users:((\"bfdd\",pid=851,fd=17))\nvcadmin@vcg-40-sfpg01:~$<\/pre>\n\n\n\nThe output of the debug commands from vcg-40-sfpg02 is very similar so it won\u2019t be repeated here.<\/p>\n\n\n\n
PE router Verification of BFD status<\/h5>\n\n\n\n
In the PE router, command \u201cshow bfd neighbors details\u201d is used to check the BFD status. The output below from PE1 confirmed the BFD status with PGW vcg-40-sfpg01 is up:<\/p>\n\n\n\n
PE1#show bfd neighbors details\nIPv4 Sessions\nNeighAddr LD\/RD RH\/RS State Int\n10.253.247.2 1\/3238898200 Up Up Et0\/2.341\nSession state is UP and not using echo function.\nSession Host: Software\nOurAddr: 10.253.247.1\nHandle: 1\nLocal Diag: 0, Demand mode: 0, Poll bit: 0\nMinTxInt: 300000, MinRxInt: 300000, Multiplier: 3\nReceived MinRxInt: 300000, Received Multiplier: 3\nHolddown (hits): 829(0), Hello (hits): 300(9183)\nRx Count: 9330, Rx Interval (ms) min\/max\/avg: 1\/301\/263 last: 71 ms ago\nTx Count: 9186, Tx Interval (ms) min\/max\/avg: 1\/310\/267 last: 206 ms ago\nElapsed time watermarks: 0 0 (last: 0)\nRegistered protocols: BGP CEF\nUptime: 00:40:51\nLast packet: Version: 1 - Diagnostic: 0\nState bit: Up - Demand bit: 0\nPoll bit: 0 - Final bit: 0\nC bit: 0\nMultiplier: 3 - Length: 24\nMy Discr.: 3238898200 - Your Discr.: 1\nMin tx interval: 300000 - Min rx interval: 300000\nMin Echo interval: 50000\nPE1#<\/pre>\n\n\n\nAgain, the output from PE2 is very similar to PE1 so it won\u2019t be included here.<\/p>\n\n\n\n
PGW Failover Test \u2013 with BFD<\/h3>\n\n\n\n
With BFD enabled between the PGW and PE router, let\u2019s repeat the ping Test from client support-L1 (10.11.41.13) to support-R2 (10.21.42.23), with a power off of primary PGW vcg-40-sfpg01. The following screen capture is the ping test result:<\/p>\n\n\n\nFigure 28: Ping test result from Support-L1 (10.11.41.13) to Support-R2 (10.21.42.23) with primary PGW failure, BFD is enabled in the PGW<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the power off of the primary PGW vcg-40-sfpg01 cause 1 ping loss (between icmp sequence 70 and 72). The failover time is around 1-2 seconds. Since the failover time is a lot faster. The ping test is being performed again with the additional parameter \u201c-i 0.1\u201d, this means the ping will be send every 0.1s instead of the default which is sent every 1s. The following is the screen capture of the ping test result with 0.1s interval:<\/p>\n\n\n\nFigure 29: Repeat of the ping test with primary PGW vcg-40-sfpg01, the ping interval is lower to 0.1s<\/figcaption><\/figure>\n\n\n\nThere are 8 ping loss (between sequence number 22 and 31), with each ping separate of 0.1s, this means 9 x 0.1s interval which result of 0.9s. As a result, it should be fine to conclude the failover time of the primary PGW failure is 1s with the BFD enabled BGP between PGW and PE router.<\/p>\n\n\n\n
Maximum BFD session supported per PGW<\/h3>\n\n\n\n
As of version 4.0, the maximum number of BFD sessions supported on PGW is 250. Although this post is not talking about the BFD on SD-WAN Edge, for documentation purpose, the maximum number of BFD sessions supported in the SD-WAN Edge is 50.<\/p>\n\n\n\n
This post “BGP with BFD enabled in SD-WAN Partner Gateway” ended here.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"Background In VMware SD-WAN (Velocloud) version 4.0, both SD-WAN Edge and SD-WAN Gateway added the support of Bidirectional Forwarding Detection (BFD). The objective of this post is to document the configuration of enabling BFD for the BGP peering between the SD-WAN Gateway (working as Partner Gateway) and PE router. There will be ping test result […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"templates\/template-fullwidth.php","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[9,8,5],"tags":[],"class_list":["post-168","post","type-post","status-publish","format-standard","hentry","category-partner-gateway","category-sd-wan-routing","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":36,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":241,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions\/241"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
The PGW vcg-40-sfpg01 is configured with IP address 10.253.247.2\/29, with AS number 65121. It is peering with PE router PE1 with AS number 9998 and IP address 10.253.247.1. The BGP timer is with default value where keep alive is 60s and hold timers is 180s. The BFD is disabled.<\/p>\n\n\n\n
vcg-40-sfpg02<\/h5>\n\n\n\nFigure 8: vcg-40-sfpg02 configuration<\/figcaption><\/figure>\n\n\n\nFigure 9: vcg-40-sfpg02 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nThe PGW vcg-40-sfpg02 is configured with IP address 10.253.247.10\/29, with AS number 65122. It is peering with PE router PE2 with AS number 9998 and IP address 10.253.247.9. The BGP timer is with default value where keep alive is 60s and hold timers is 180s. The BFD is disabled.<\/p>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 10: vcg-40-nypg01 configuration
<\/figcaption><\/figure>\n\n\n\nFigure 11: vcg-40-nypg02 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nIncluding the configuration of the vcg-40-nypg02 for completeness.<\/p>\n\n\n\n
Configuration of PE router<\/h4>\n\n\n\n
Since the focus of this post is the SD-WAN components, not the PE router, only related configuration of the PE router will be show. The VRF being used in the test is called Customer40b.<\/p>\n\n\n\n
PE1<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg02\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nConfiguration to ensure the traffic path is symmetric<\/h5>\n\n\n\n
Although the Left-Edge-T3 is aware the vcg-40-sfpg01 is the primary PGW as it is at order position #1, the MPLS backbone also need to ensure the return traffic prefer landing on PE1 instead of landing on PE2. This is done by auto community mapping at the PGW and route-map to adjust local preference at the PE router.<\/p>\n\n\n\nFigure 12: Community Mapping<\/figcaption><\/figure>\n\n\n\nPGW is configured with community mapping with priority #1 PGW routes come with community 9998:111 (655229039) and priority #2 PGW routes come with community 9998:222 (655229150). That means, for the routes advertised by vcg-40-sfpg01 to the PE1, those routes will be having community 9998:111. For the routes advertised by vcg-40-sfpg02 to the PE2, those routes will be having community 9998:222.<\/p>\n\n\n\n
The following configurations are the route-map comm-set-localpref to make PE1 is the preferred return path as route from vcg-40-sfpg01 is set with local preference of 200 while route from vcg-40-sfpg02 is set with local preference of 101.<\/p>\n\n\n\n
PE1 route-map configuration:<\/p>\n\n\n\n
!\nip community-list standard comm-priority1 permit 655229039\nip community-list standard comm-priority2 permit 655229150\n!\n! Set local preference to 200 when route comes with community 9998:111, set local preference to 101 when route comes with community 9998:222\n!\nroute-map comm-set-localpref permit 10\n match community comm-priority1\n set local-preference 200\n!\nroute-map comm-set-localpref permit 20\n match community comm-priority2\n set local-preference 101\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe same route-map, hence same mechanism is applied to PE2 as well, the configuration of the PE2 route-map is like a mirror copy of PE1, so it is not repeated here.<\/p>\n\n\n\n
Verification on SD-WAN Edge, PGW and PE router<\/h3>\n\n\n\n
The section will perform some verification to ensure the SD-WAN Edge, PGW, PE router has learnt the route properly.<\/p>\n\n\n\n
Verification on SD-WAN Edge<\/h4>\n\n\n\nLeft-Edge-T3:<\/h5>\n\n\n\nFigure 13: Path and 10.21.42.0\/24 route on Left-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.21.42.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Left-Edge-T3 is able to form overlay tunnel to vcg-40-sfpg01 and vcg-40-sfpg02<\/li>
- Left-Edge-T3 learnt the route 10.21.42.0\/24 from both vcg-40-sfpg01 and vcg-40-sfpg02 where vcg-40-sfpg01 is the preferred Next Hop.<\/li><\/ol>\n\n\n\n
Right-Edge-T3:<\/h5>\n\n\n\nFigure 14: Path and 10.11.41.0\/24 route on Right-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.11.41.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Right-Edge-T3 is able to form overlay tunnels to vcg-40-nypg01.<\/li>
- Right-Edge-T3 learnt the route 10.11.41.0\/24 from vcg-40-nypg01.<\/li><\/ol>\n\n\n\n
Verification on PGW<\/h4>\n\n\n\nvcg-40-sfpg01:<\/h5>\n\n\n\nFigure 15: vcg-40-sfpg01 debug command output of bgp_view_summary and bgp_view <\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 15) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg01 has successfully established BGP peer with PE1 (10.253.247.1)<\/li>
- PGW vcg-40-sfpg01 learnt some routes from PE1, including the route 10.21.42.0\/24<\/li><\/ol>\n\n\n\n
vcg-40-sfpg02:<\/h5>\n\n\n\nFigure 16: vcg-40-sfpg02 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 16) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg02. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg02 has successfully established BGP peer with PE2 (10.253.247.9).<\/li>
- PGW vcg-40-sfpg02 learnt some routes from PE2, including the route 10.21.42.0\/24.<\/li><\/ol>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 17: cg-40-nypg01 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-nypg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-nypg01 has successfully established BGP peer with PE3 (10.253.246.1).<\/li>
- PGW vcg-40-nypg01 learnt some routes from PE3, including the route 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
Verification on PE router<\/h4>\n\n\n\nPE1<\/h5>\n\n\n\nFigure 18: PE1, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 18) is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE1. The output confirmed:<\/p>\n\n\n\n
- PE1 successfully having BGP peer with vcg-40-sfpg01 (10.253.247.2).<\/li>
- PE1 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg01.<\/li>
- PE1 successfully learnt the route 10.11.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE2<\/h5>\n\n\n\nFigure 19: PE2, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE2. The output confirmed:<\/p>\n\n\n\n
- PE2 successfully having BGP peer with vcg-40-sfpg02 (10.253.247.10).<\/li>
- PE2 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg02 (and also from PE1 as PE1 learnt this route from vcg-40-sfpg01).<\/li>
- PE2 successfully learnt the route 10.21.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE3<\/h5>\n\n\n\nFigure 20: PE3, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE3. The output confirmed:<\/p>\n\n\n\n
- PE3 successfully having BGP peer with vcg-40-nypg01 (10.253.246.2).<\/li>
- PE3 successfully learnt the route 10.21.42.0\/24 from PGW vcg-40-nypg01.<\/li>
- PE3 successfully learnt the route 10.11.41.0\/24 from MPLS backbone. Note that the Next Hop is 10.10.10.21 which is the PE1 loopback IP address, the local preference is 200. This means the route-map on PE1 and PE2 is working, such that packets prefer going to PE1 for destination 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
PGW Failover Test \u2013 without BFD<\/h3>\n\n\n\n
The test is to test how long does it take for the traffic to switch from vcg-40-sfpg01 to vcg-40-sfpg02. In order to perform this test, the Linux machine support-L1 (10.11.41.13) issue a continuous ping to support-R2 (10.21.42.23). In the middle of the ping, the vcg-40-sfpg01 will get power off, and the time required to fail over will get recorded.<\/p>\n\n\n\n
Before the ping failover test, a traceroute from support-L1 (10.11.41.13) to support-R2 (10.21.42.23) is capture as follow:<\/p>\n\n\n\nFigure 21: Client machine Support-L1 (10.11.41.13) traceroute to 10.21.42.23<\/figcaption><\/figure>\n\n\n\nThis traceroute confirmed the traffic path is \u201cLeft-Edge-T3 –> vcg-40-sfpg01 –> PE1 –> PE3 –>vcg-40-nypg01 –> Right-Edge-T3\u201d<\/p>\n\n\n\n
The test method is having Support-L1 (10.11.41.13) issue a continuous ping to Support-R2 (10.21.42.23), in the middle of the ping, PGW vcg-40-sfpg01 gets powered off. The following show the test result:<\/p>\n\n\n\nFigure 22: Ping test result for primary PGW failed, with default BGP timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 82. There are 129 ping missed during the failover, which is roughly around 130 seconds.<\/p>\n\n\n\n
The reader might wonder why the failover takes so long, since the SD-WAN Edge Left-Edge-T3 is having Overlay Tunnel to both vcg-40-sfpg01 and vcg-40-sfpg02, when vcg-40-sfpg01 gets powered off, the corresponding Overlay Tunnel gets terminated. As a result, Left-Edge-T3 should be able to switch to vcg-40-sfpg02 in sub-second. This understanding is correct. But let\u2019s look at the diagram again to refresh the memory:<\/p>\n\n\n\nFigure 23: Test Topology<\/figcaption><\/figure>\n\n\n\nThe failover takes long time because of the return traffic. The PE1 is configured to have route map to attract the return traffic. In order for the return traffic to be able to switch from PE1 to PE2, PE1 must stop advertising the 10.11.41.0\/24 route. PE1 will stop advertising when the BGP peer between PE1 and vcg-40-sfpg01 is down, with the default timer (60s keepalive and 180s hold timer), it can take a maximum of 180 seconds for the BGP peer to go down.<\/p>\n\n\n\n
One of the options to speed up the failover is having a smaller value of keepalive and hold timer. The following is the screen capture of repeating the same ping test with 3s keepalive and 9s hold timer:<\/p>\n\n\n\nFigure 24: Ping test result for primary PGW failed, with 3s keepalive and 9s hold timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 35. There are 9 ping missed during the failover, which the failover is roughly around 9-10 seconds.<\/p>\n\n\n\n
PGW and PE Router Configurations \u2013 with BFD<\/h4>\n\n\n\n
In this section, BFD will be enabled in the SD-WAN Gateways and PE routers. Other devices like the SD-WAN Edge will maintain the existing configuration.<\/p>\n\n\n\n
SD-WAN Gateway (Partner Gateway) \u2013 with BFD<\/h5>\n\n\n\n
The follow two screen captures show the PGW (vcg-40-sfpg01 and vcg-40-sfpg02) BGP peer with BFD enabled. In this test, the receive interval and transmit interval are both 300ms, the multiplier is 3.<\/p>\n\n\n\nFigure 25: vcg-40-sfpg01 BGP peer with 10.253.247.1 BFD enabled<\/figcaption><\/figure>\n\n\n\nFigure 26: vcg-40-sfpg02 BGP peer with 10.253.247.9 BFD enabled<\/figcaption><\/figure>\n\n\n\nPE router \u2013 with BFD<\/h4>\n\n\n\nPE1 configuration by enabling BFD for BGP peer 10.253.247.2<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/em><\/span>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 fall-over bfd<\/em><\/span>\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2 configuration by enabling BFD for BGP peer 10.253.247.10<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/span><\/em>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 fall-over bfd<\/span><\/em>\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe newly added configuration to enable BFD in the PE router is marked with italic<\/span><\/em> in the above configuration snippets.<\/p>\n\n\n\nNOTE: In the PE router sub-interface using for the test vrf, that is vrf Customer40b, the \u201cno bfd echo\u201d command is entered to disable the BFD echo mode. As of the VMware SD-WAN version 4.0, BFD echo mode is not supported, only BFD asynchronous mode is supported.<\/p><\/blockquote>\n\n\n\n
Verification of the BFD status<\/h3>\n\n\n\nVerification of BFD status at the PGW<\/h4>\n\n\n\n
There is now a “Gateway BFD Sessions” tab under “Monitor –> Routing –> BFD”<\/p>\n\n\n\nFigure 27: Gateway BFD Sessions<\/figcaption><\/figure>\n\n\n\nFrom the “Gateway BFD Sessions” GUI, we can see both vcg-40-sfpg01 and vcg-40-sfpg02 are having “UP” state of the BFD with the corresponding BGP peer, which means the BFD is working. In case more detail information is needed, in the PGW shell, there are two commands can check the BFD status, they are \u201cdebug.py –bfd_info\u201d and \u201cdebug.py –bfdd_dump\u201d. The output below is from PGW vcg-40-sfpg01 which shows the BFD peer is with “up” status. The following snippet shows the BFD debug.py output of vcg-40-sfpg01: <\/p>\n\n\n\n
vcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfd_info<\/span>\nEnterpriseId SEG Peer Address Local Address Detect Multiplier Transmit Interval Receive Interval Status\neb032999-da64-4b09-ab57-3b5f9949ffdb 0 10.253.247.1 10.253.247.2 3 300 300 UP\nvcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfdd_dump<\/span>\nshow running-config\nBuilding configuration\u2026\nCurrent configuration:\n!\nfrr version 7.0\nfrr defaults traditional\nhostname vcg-40-sfpg01\nlog file \/var\/log\/bfdd.log\n!\npassword zebra\n!\nline vty\naccess-class vty\n!\nbfd\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nno shutdown\n!\n!\nend\nshow bfd peers\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nID: 3238898200\nRemote ID: 1\nStatus: up\nUptime: 34 minute(s), 48 second(s)\nDiagnostics: ok\nRemote diagnostics: ok\nLocal timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 50ms\nRemote timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 0ms\nshow bfd peers counters\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nControl packet input: 9021 packets\nControl packet output: 81639 packets\nEcho packet input: 0 packets\nEcho packet output: 0 packets\nSession up events: 239\nSession down events: 238\nZebra notifications: 479\nNS UDP connections dump for entr ID: [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nState Recv-Q Send-Q Local Address:Port Peer Address:Port\nUNCONN 0 0 0.0.0.0:3784 0.0.0.0:* users:((\"bfdd\",pid=851,fd=16))\nUNCONN 0 0 0.0.0.0:49142 0.0.0.0:* users:((\"bfdd\",pid=851,fd=17))\nvcadmin@vcg-40-sfpg01:~$<\/pre>\n\n\n\nThe output of the debug commands from vcg-40-sfpg02 is very similar so it won\u2019t be repeated here.<\/p>\n\n\n\n
PE router Verification of BFD status<\/h5>\n\n\n\n
In the PE router, command \u201cshow bfd neighbors details\u201d is used to check the BFD status. The output below from PE1 confirmed the BFD status with PGW vcg-40-sfpg01 is up:<\/p>\n\n\n\n
PE1#show bfd neighbors details\nIPv4 Sessions\nNeighAddr LD\/RD RH\/RS State Int\n10.253.247.2 1\/3238898200 Up Up Et0\/2.341\nSession state is UP and not using echo function.\nSession Host: Software\nOurAddr: 10.253.247.1\nHandle: 1\nLocal Diag: 0, Demand mode: 0, Poll bit: 0\nMinTxInt: 300000, MinRxInt: 300000, Multiplier: 3\nReceived MinRxInt: 300000, Received Multiplier: 3\nHolddown (hits): 829(0), Hello (hits): 300(9183)\nRx Count: 9330, Rx Interval (ms) min\/max\/avg: 1\/301\/263 last: 71 ms ago\nTx Count: 9186, Tx Interval (ms) min\/max\/avg: 1\/310\/267 last: 206 ms ago\nElapsed time watermarks: 0 0 (last: 0)\nRegistered protocols: BGP CEF\nUptime: 00:40:51\nLast packet: Version: 1 - Diagnostic: 0\nState bit: Up - Demand bit: 0\nPoll bit: 0 - Final bit: 0\nC bit: 0\nMultiplier: 3 - Length: 24\nMy Discr.: 3238898200 - Your Discr.: 1\nMin tx interval: 300000 - Min rx interval: 300000\nMin Echo interval: 50000\nPE1#<\/pre>\n\n\n\nAgain, the output from PE2 is very similar to PE1 so it won\u2019t be included here.<\/p>\n\n\n\n
PGW Failover Test \u2013 with BFD<\/h3>\n\n\n\n
With BFD enabled between the PGW and PE router, let\u2019s repeat the ping Test from client support-L1 (10.11.41.13) to support-R2 (10.21.42.23), with a power off of primary PGW vcg-40-sfpg01. The following screen capture is the ping test result:<\/p>\n\n\n\nFigure 28: Ping test result from Support-L1 (10.11.41.13) to Support-R2 (10.21.42.23) with primary PGW failure, BFD is enabled in the PGW<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the power off of the primary PGW vcg-40-sfpg01 cause 1 ping loss (between icmp sequence 70 and 72). The failover time is around 1-2 seconds. Since the failover time is a lot faster. The ping test is being performed again with the additional parameter \u201c-i 0.1\u201d, this means the ping will be send every 0.1s instead of the default which is sent every 1s. The following is the screen capture of the ping test result with 0.1s interval:<\/p>\n\n\n\nFigure 29: Repeat of the ping test with primary PGW vcg-40-sfpg01, the ping interval is lower to 0.1s<\/figcaption><\/figure>\n\n\n\nThere are 8 ping loss (between sequence number 22 and 31), with each ping separate of 0.1s, this means 9 x 0.1s interval which result of 0.9s. As a result, it should be fine to conclude the failover time of the primary PGW failure is 1s with the BFD enabled BGP between PGW and PE router.<\/p>\n\n\n\n
Maximum BFD session supported per PGW<\/h3>\n\n\n\n
As of version 4.0, the maximum number of BFD sessions supported on PGW is 250. Although this post is not talking about the BFD on SD-WAN Edge, for documentation purpose, the maximum number of BFD sessions supported in the SD-WAN Edge is 50.<\/p>\n\n\n\n
This post “BGP with BFD enabled in SD-WAN Partner Gateway” ended here.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"Background In VMware SD-WAN (Velocloud) version 4.0, both SD-WAN Edge and SD-WAN Gateway added the support of Bidirectional Forwarding Detection (BFD). The objective of this post is to document the configuration of enabling BFD for the BGP peering between the SD-WAN Gateway (working as Partner Gateway) and PE router. There will be ping test result […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"templates\/template-fullwidth.php","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[9,8,5],"tags":[],"class_list":["post-168","post","type-post","status-publish","format-standard","hentry","category-partner-gateway","category-sd-wan-routing","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":36,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":241,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions\/241"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
The PGW vcg-40-sfpg02 is configured with IP address 10.253.247.10\/29, with AS number 65122. It is peering with PE router PE2 with AS number 9998 and IP address 10.253.247.9. The BGP timer is with default value where keep alive is 60s and hold timers is 180s. The BFD is disabled.<\/p>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 10: vcg-40-nypg01 configuration
<\/figcaption><\/figure>\n\n\n\nFigure 11: vcg-40-nypg02 configuration, BGP timer values are using default<\/figcaption><\/figure>\n\n\n\nIncluding the configuration of the vcg-40-nypg02 for completeness.<\/p>\n\n\n\n
Configuration of PE router<\/h4>\n\n\n\n
Since the focus of this post is the SD-WAN components, not the PE router, only related configuration of the PE router will be show. The VRF being used in the test is called Customer40b.<\/p>\n\n\n\n
PE1<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg02\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nConfiguration to ensure the traffic path is symmetric<\/h5>\n\n\n\n
Although the Left-Edge-T3 is aware the vcg-40-sfpg01 is the primary PGW as it is at order position #1, the MPLS backbone also need to ensure the return traffic prefer landing on PE1 instead of landing on PE2. This is done by auto community mapping at the PGW and route-map to adjust local preference at the PE router.<\/p>\n\n\n\nFigure 12: Community Mapping<\/figcaption><\/figure>\n\n\n\nPGW is configured with community mapping with priority #1 PGW routes come with community 9998:111 (655229039) and priority #2 PGW routes come with community 9998:222 (655229150). That means, for the routes advertised by vcg-40-sfpg01 to the PE1, those routes will be having community 9998:111. For the routes advertised by vcg-40-sfpg02 to the PE2, those routes will be having community 9998:222.<\/p>\n\n\n\n
The following configurations are the route-map comm-set-localpref to make PE1 is the preferred return path as route from vcg-40-sfpg01 is set with local preference of 200 while route from vcg-40-sfpg02 is set with local preference of 101.<\/p>\n\n\n\n
PE1 route-map configuration:<\/p>\n\n\n\n
!\nip community-list standard comm-priority1 permit 655229039\nip community-list standard comm-priority2 permit 655229150\n!\n! Set local preference to 200 when route comes with community 9998:111, set local preference to 101 when route comes with community 9998:222\n!\nroute-map comm-set-localpref permit 10\n match community comm-priority1\n set local-preference 200\n!\nroute-map comm-set-localpref permit 20\n match community comm-priority2\n set local-preference 101\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe same route-map, hence same mechanism is applied to PE2 as well, the configuration of the PE2 route-map is like a mirror copy of PE1, so it is not repeated here.<\/p>\n\n\n\n
Verification on SD-WAN Edge, PGW and PE router<\/h3>\n\n\n\n
The section will perform some verification to ensure the SD-WAN Edge, PGW, PE router has learnt the route properly.<\/p>\n\n\n\n
Verification on SD-WAN Edge<\/h4>\n\n\n\nLeft-Edge-T3:<\/h5>\n\n\n\nFigure 13: Path and 10.21.42.0\/24 route on Left-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.21.42.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Left-Edge-T3 is able to form overlay tunnel to vcg-40-sfpg01 and vcg-40-sfpg02<\/li>
- Left-Edge-T3 learnt the route 10.21.42.0\/24 from both vcg-40-sfpg01 and vcg-40-sfpg02 where vcg-40-sfpg01 is the preferred Next Hop.<\/li><\/ol>\n\n\n\n
Right-Edge-T3:<\/h5>\n\n\n\nFigure 14: Path and 10.11.41.0\/24 route on Right-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.11.41.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Right-Edge-T3 is able to form overlay tunnels to vcg-40-nypg01.<\/li>
- Right-Edge-T3 learnt the route 10.11.41.0\/24 from vcg-40-nypg01.<\/li><\/ol>\n\n\n\n
Verification on PGW<\/h4>\n\n\n\nvcg-40-sfpg01:<\/h5>\n\n\n\nFigure 15: vcg-40-sfpg01 debug command output of bgp_view_summary and bgp_view <\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 15) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg01 has successfully established BGP peer with PE1 (10.253.247.1)<\/li>
- PGW vcg-40-sfpg01 learnt some routes from PE1, including the route 10.21.42.0\/24<\/li><\/ol>\n\n\n\n
vcg-40-sfpg02:<\/h5>\n\n\n\nFigure 16: vcg-40-sfpg02 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 16) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg02. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg02 has successfully established BGP peer with PE2 (10.253.247.9).<\/li>
- PGW vcg-40-sfpg02 learnt some routes from PE2, including the route 10.21.42.0\/24.<\/li><\/ol>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 17: cg-40-nypg01 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-nypg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-nypg01 has successfully established BGP peer with PE3 (10.253.246.1).<\/li>
- PGW vcg-40-nypg01 learnt some routes from PE3, including the route 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
Verification on PE router<\/h4>\n\n\n\nPE1<\/h5>\n\n\n\nFigure 18: PE1, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 18) is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE1. The output confirmed:<\/p>\n\n\n\n
- PE1 successfully having BGP peer with vcg-40-sfpg01 (10.253.247.2).<\/li>
- PE1 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg01.<\/li>
- PE1 successfully learnt the route 10.11.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE2<\/h5>\n\n\n\nFigure 19: PE2, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE2. The output confirmed:<\/p>\n\n\n\n
- PE2 successfully having BGP peer with vcg-40-sfpg02 (10.253.247.10).<\/li>
- PE2 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg02 (and also from PE1 as PE1 learnt this route from vcg-40-sfpg01).<\/li>
- PE2 successfully learnt the route 10.21.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE3<\/h5>\n\n\n\nFigure 20: PE3, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE3. The output confirmed:<\/p>\n\n\n\n
- PE3 successfully having BGP peer with vcg-40-nypg01 (10.253.246.2).<\/li>
- PE3 successfully learnt the route 10.21.42.0\/24 from PGW vcg-40-nypg01.<\/li>
- PE3 successfully learnt the route 10.11.41.0\/24 from MPLS backbone. Note that the Next Hop is 10.10.10.21 which is the PE1 loopback IP address, the local preference is 200. This means the route-map on PE1 and PE2 is working, such that packets prefer going to PE1 for destination 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
PGW Failover Test \u2013 without BFD<\/h3>\n\n\n\n
The test is to test how long does it take for the traffic to switch from vcg-40-sfpg01 to vcg-40-sfpg02. In order to perform this test, the Linux machine support-L1 (10.11.41.13) issue a continuous ping to support-R2 (10.21.42.23). In the middle of the ping, the vcg-40-sfpg01 will get power off, and the time required to fail over will get recorded.<\/p>\n\n\n\n
Before the ping failover test, a traceroute from support-L1 (10.11.41.13) to support-R2 (10.21.42.23) is capture as follow:<\/p>\n\n\n\nFigure 21: Client machine Support-L1 (10.11.41.13) traceroute to 10.21.42.23<\/figcaption><\/figure>\n\n\n\nThis traceroute confirmed the traffic path is \u201cLeft-Edge-T3 –> vcg-40-sfpg01 –> PE1 –> PE3 –>vcg-40-nypg01 –> Right-Edge-T3\u201d<\/p>\n\n\n\n
The test method is having Support-L1 (10.11.41.13) issue a continuous ping to Support-R2 (10.21.42.23), in the middle of the ping, PGW vcg-40-sfpg01 gets powered off. The following show the test result:<\/p>\n\n\n\nFigure 22: Ping test result for primary PGW failed, with default BGP timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 82. There are 129 ping missed during the failover, which is roughly around 130 seconds.<\/p>\n\n\n\n
The reader might wonder why the failover takes so long, since the SD-WAN Edge Left-Edge-T3 is having Overlay Tunnel to both vcg-40-sfpg01 and vcg-40-sfpg02, when vcg-40-sfpg01 gets powered off, the corresponding Overlay Tunnel gets terminated. As a result, Left-Edge-T3 should be able to switch to vcg-40-sfpg02 in sub-second. This understanding is correct. But let\u2019s look at the diagram again to refresh the memory:<\/p>\n\n\n\nFigure 23: Test Topology<\/figcaption><\/figure>\n\n\n\nThe failover takes long time because of the return traffic. The PE1 is configured to have route map to attract the return traffic. In order for the return traffic to be able to switch from PE1 to PE2, PE1 must stop advertising the 10.11.41.0\/24 route. PE1 will stop advertising when the BGP peer between PE1 and vcg-40-sfpg01 is down, with the default timer (60s keepalive and 180s hold timer), it can take a maximum of 180 seconds for the BGP peer to go down.<\/p>\n\n\n\n
One of the options to speed up the failover is having a smaller value of keepalive and hold timer. The following is the screen capture of repeating the same ping test with 3s keepalive and 9s hold timer:<\/p>\n\n\n\nFigure 24: Ping test result for primary PGW failed, with 3s keepalive and 9s hold timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 35. There are 9 ping missed during the failover, which the failover is roughly around 9-10 seconds.<\/p>\n\n\n\n
PGW and PE Router Configurations \u2013 with BFD<\/h4>\n\n\n\n
In this section, BFD will be enabled in the SD-WAN Gateways and PE routers. Other devices like the SD-WAN Edge will maintain the existing configuration.<\/p>\n\n\n\n
SD-WAN Gateway (Partner Gateway) \u2013 with BFD<\/h5>\n\n\n\n
The follow two screen captures show the PGW (vcg-40-sfpg01 and vcg-40-sfpg02) BGP peer with BFD enabled. In this test, the receive interval and transmit interval are both 300ms, the multiplier is 3.<\/p>\n\n\n\nFigure 25: vcg-40-sfpg01 BGP peer with 10.253.247.1 BFD enabled<\/figcaption><\/figure>\n\n\n\nFigure 26: vcg-40-sfpg02 BGP peer with 10.253.247.9 BFD enabled<\/figcaption><\/figure>\n\n\n\nPE router \u2013 with BFD<\/h4>\n\n\n\nPE1 configuration by enabling BFD for BGP peer 10.253.247.2<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/em><\/span>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 fall-over bfd<\/em><\/span>\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2 configuration by enabling BFD for BGP peer 10.253.247.10<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/span><\/em>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 fall-over bfd<\/span><\/em>\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe newly added configuration to enable BFD in the PE router is marked with italic<\/span><\/em> in the above configuration snippets.<\/p>\n\n\n\nNOTE: In the PE router sub-interface using for the test vrf, that is vrf Customer40b, the \u201cno bfd echo\u201d command is entered to disable the BFD echo mode. As of the VMware SD-WAN version 4.0, BFD echo mode is not supported, only BFD asynchronous mode is supported.<\/p><\/blockquote>\n\n\n\n
Verification of the BFD status<\/h3>\n\n\n\nVerification of BFD status at the PGW<\/h4>\n\n\n\n
There is now a “Gateway BFD Sessions” tab under “Monitor –> Routing –> BFD”<\/p>\n\n\n\nFigure 27: Gateway BFD Sessions<\/figcaption><\/figure>\n\n\n\nFrom the “Gateway BFD Sessions” GUI, we can see both vcg-40-sfpg01 and vcg-40-sfpg02 are having “UP” state of the BFD with the corresponding BGP peer, which means the BFD is working. In case more detail information is needed, in the PGW shell, there are two commands can check the BFD status, they are \u201cdebug.py –bfd_info\u201d and \u201cdebug.py –bfdd_dump\u201d. The output below is from PGW vcg-40-sfpg01 which shows the BFD peer is with “up” status. The following snippet shows the BFD debug.py output of vcg-40-sfpg01: <\/p>\n\n\n\n
vcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfd_info<\/span>\nEnterpriseId SEG Peer Address Local Address Detect Multiplier Transmit Interval Receive Interval Status\neb032999-da64-4b09-ab57-3b5f9949ffdb 0 10.253.247.1 10.253.247.2 3 300 300 UP\nvcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfdd_dump<\/span>\nshow running-config\nBuilding configuration\u2026\nCurrent configuration:\n!\nfrr version 7.0\nfrr defaults traditional\nhostname vcg-40-sfpg01\nlog file \/var\/log\/bfdd.log\n!\npassword zebra\n!\nline vty\naccess-class vty\n!\nbfd\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nno shutdown\n!\n!\nend\nshow bfd peers\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nID: 3238898200\nRemote ID: 1\nStatus: up\nUptime: 34 minute(s), 48 second(s)\nDiagnostics: ok\nRemote diagnostics: ok\nLocal timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 50ms\nRemote timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 0ms\nshow bfd peers counters\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nControl packet input: 9021 packets\nControl packet output: 81639 packets\nEcho packet input: 0 packets\nEcho packet output: 0 packets\nSession up events: 239\nSession down events: 238\nZebra notifications: 479\nNS UDP connections dump for entr ID: [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nState Recv-Q Send-Q Local Address:Port Peer Address:Port\nUNCONN 0 0 0.0.0.0:3784 0.0.0.0:* users:((\"bfdd\",pid=851,fd=16))\nUNCONN 0 0 0.0.0.0:49142 0.0.0.0:* users:((\"bfdd\",pid=851,fd=17))\nvcadmin@vcg-40-sfpg01:~$<\/pre>\n\n\n\nThe output of the debug commands from vcg-40-sfpg02 is very similar so it won\u2019t be repeated here.<\/p>\n\n\n\n
PE router Verification of BFD status<\/h5>\n\n\n\n
In the PE router, command \u201cshow bfd neighbors details\u201d is used to check the BFD status. The output below from PE1 confirmed the BFD status with PGW vcg-40-sfpg01 is up:<\/p>\n\n\n\n
PE1#show bfd neighbors details\nIPv4 Sessions\nNeighAddr LD\/RD RH\/RS State Int\n10.253.247.2 1\/3238898200 Up Up Et0\/2.341\nSession state is UP and not using echo function.\nSession Host: Software\nOurAddr: 10.253.247.1\nHandle: 1\nLocal Diag: 0, Demand mode: 0, Poll bit: 0\nMinTxInt: 300000, MinRxInt: 300000, Multiplier: 3\nReceived MinRxInt: 300000, Received Multiplier: 3\nHolddown (hits): 829(0), Hello (hits): 300(9183)\nRx Count: 9330, Rx Interval (ms) min\/max\/avg: 1\/301\/263 last: 71 ms ago\nTx Count: 9186, Tx Interval (ms) min\/max\/avg: 1\/310\/267 last: 206 ms ago\nElapsed time watermarks: 0 0 (last: 0)\nRegistered protocols: BGP CEF\nUptime: 00:40:51\nLast packet: Version: 1 - Diagnostic: 0\nState bit: Up - Demand bit: 0\nPoll bit: 0 - Final bit: 0\nC bit: 0\nMultiplier: 3 - Length: 24\nMy Discr.: 3238898200 - Your Discr.: 1\nMin tx interval: 300000 - Min rx interval: 300000\nMin Echo interval: 50000\nPE1#<\/pre>\n\n\n\nAgain, the output from PE2 is very similar to PE1 so it won\u2019t be included here.<\/p>\n\n\n\n
PGW Failover Test \u2013 with BFD<\/h3>\n\n\n\n
With BFD enabled between the PGW and PE router, let\u2019s repeat the ping Test from client support-L1 (10.11.41.13) to support-R2 (10.21.42.23), with a power off of primary PGW vcg-40-sfpg01. The following screen capture is the ping test result:<\/p>\n\n\n\nFigure 28: Ping test result from Support-L1 (10.11.41.13) to Support-R2 (10.21.42.23) with primary PGW failure, BFD is enabled in the PGW<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the power off of the primary PGW vcg-40-sfpg01 cause 1 ping loss (between icmp sequence 70 and 72). The failover time is around 1-2 seconds. Since the failover time is a lot faster. The ping test is being performed again with the additional parameter \u201c-i 0.1\u201d, this means the ping will be send every 0.1s instead of the default which is sent every 1s. The following is the screen capture of the ping test result with 0.1s interval:<\/p>\n\n\n\nFigure 29: Repeat of the ping test with primary PGW vcg-40-sfpg01, the ping interval is lower to 0.1s<\/figcaption><\/figure>\n\n\n\nThere are 8 ping loss (between sequence number 22 and 31), with each ping separate of 0.1s, this means 9 x 0.1s interval which result of 0.9s. As a result, it should be fine to conclude the failover time of the primary PGW failure is 1s with the BFD enabled BGP between PGW and PE router.<\/p>\n\n\n\n
Maximum BFD session supported per PGW<\/h3>\n\n\n\n
As of version 4.0, the maximum number of BFD sessions supported on PGW is 250. Although this post is not talking about the BFD on SD-WAN Edge, for documentation purpose, the maximum number of BFD sessions supported in the SD-WAN Edge is 50.<\/p>\n\n\n\n
This post “BGP with BFD enabled in SD-WAN Partner Gateway” ended here.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"Background In VMware SD-WAN (Velocloud) version 4.0, both SD-WAN Edge and SD-WAN Gateway added the support of Bidirectional Forwarding Detection (BFD). The objective of this post is to document the configuration of enabling BFD for the BGP peering between the SD-WAN Gateway (working as Partner Gateway) and PE router. There will be ping test result […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"templates\/template-fullwidth.php","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[9,8,5],"tags":[],"class_list":["post-168","post","type-post","status-publish","format-standard","hentry","category-partner-gateway","category-sd-wan-routing","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":36,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":241,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions\/241"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
<\/figcaption><\/figure>\n\n\n\n
Including the configuration of the vcg-40-nypg02 for completeness.<\/p>\n\n\n\n
Configuration of PE router<\/h4>\n\n\n\n
Since the focus of this post is the SD-WAN components, not the PE router, only related configuration of the PE router will be show. The VRF being used in the test is called Customer40b.<\/p>\n\n\n\n
PE1<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg02\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nConfiguration to ensure the traffic path is symmetric<\/h5>\n\n\n\n
Although the Left-Edge-T3 is aware the vcg-40-sfpg01 is the primary PGW as it is at order position #1, the MPLS backbone also need to ensure the return traffic prefer landing on PE1 instead of landing on PE2. This is done by auto community mapping at the PGW and route-map to adjust local preference at the PE router.<\/p>\n\n\n\nFigure 12: Community Mapping<\/figcaption><\/figure>\n\n\n\nPGW is configured with community mapping with priority #1 PGW routes come with community 9998:111 (655229039) and priority #2 PGW routes come with community 9998:222 (655229150). That means, for the routes advertised by vcg-40-sfpg01 to the PE1, those routes will be having community 9998:111. For the routes advertised by vcg-40-sfpg02 to the PE2, those routes will be having community 9998:222.<\/p>\n\n\n\n
The following configurations are the route-map comm-set-localpref to make PE1 is the preferred return path as route from vcg-40-sfpg01 is set with local preference of 200 while route from vcg-40-sfpg02 is set with local preference of 101.<\/p>\n\n\n\n
PE1 route-map configuration:<\/p>\n\n\n\n
!\nip community-list standard comm-priority1 permit 655229039\nip community-list standard comm-priority2 permit 655229150\n!\n! Set local preference to 200 when route comes with community 9998:111, set local preference to 101 when route comes with community 9998:222\n!\nroute-map comm-set-localpref permit 10\n match community comm-priority1\n set local-preference 200\n!\nroute-map comm-set-localpref permit 20\n match community comm-priority2\n set local-preference 101\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe same route-map, hence same mechanism is applied to PE2 as well, the configuration of the PE2 route-map is like a mirror copy of PE1, so it is not repeated here.<\/p>\n\n\n\n
Verification on SD-WAN Edge, PGW and PE router<\/h3>\n\n\n\n
The section will perform some verification to ensure the SD-WAN Edge, PGW, PE router has learnt the route properly.<\/p>\n\n\n\n
Verification on SD-WAN Edge<\/h4>\n\n\n\nLeft-Edge-T3:<\/h5>\n\n\n\nFigure 13: Path and 10.21.42.0\/24 route on Left-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.21.42.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Left-Edge-T3 is able to form overlay tunnel to vcg-40-sfpg01 and vcg-40-sfpg02<\/li>
- Left-Edge-T3 learnt the route 10.21.42.0\/24 from both vcg-40-sfpg01 and vcg-40-sfpg02 where vcg-40-sfpg01 is the preferred Next Hop.<\/li><\/ol>\n\n\n\n
Right-Edge-T3:<\/h5>\n\n\n\nFigure 14: Path and 10.11.41.0\/24 route on Right-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.11.41.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Right-Edge-T3 is able to form overlay tunnels to vcg-40-nypg01.<\/li>
- Right-Edge-T3 learnt the route 10.11.41.0\/24 from vcg-40-nypg01.<\/li><\/ol>\n\n\n\n
Verification on PGW<\/h4>\n\n\n\nvcg-40-sfpg01:<\/h5>\n\n\n\nFigure 15: vcg-40-sfpg01 debug command output of bgp_view_summary and bgp_view <\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 15) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg01 has successfully established BGP peer with PE1 (10.253.247.1)<\/li>
- PGW vcg-40-sfpg01 learnt some routes from PE1, including the route 10.21.42.0\/24<\/li><\/ol>\n\n\n\n
vcg-40-sfpg02:<\/h5>\n\n\n\nFigure 16: vcg-40-sfpg02 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 16) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg02. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg02 has successfully established BGP peer with PE2 (10.253.247.9).<\/li>
- PGW vcg-40-sfpg02 learnt some routes from PE2, including the route 10.21.42.0\/24.<\/li><\/ol>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 17: cg-40-nypg01 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-nypg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-nypg01 has successfully established BGP peer with PE3 (10.253.246.1).<\/li>
- PGW vcg-40-nypg01 learnt some routes from PE3, including the route 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
Verification on PE router<\/h4>\n\n\n\nPE1<\/h5>\n\n\n\nFigure 18: PE1, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 18) is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE1. The output confirmed:<\/p>\n\n\n\n
- PE1 successfully having BGP peer with vcg-40-sfpg01 (10.253.247.2).<\/li>
- PE1 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg01.<\/li>
- PE1 successfully learnt the route 10.11.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE2<\/h5>\n\n\n\nFigure 19: PE2, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE2. The output confirmed:<\/p>\n\n\n\n
- PE2 successfully having BGP peer with vcg-40-sfpg02 (10.253.247.10).<\/li>
- PE2 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg02 (and also from PE1 as PE1 learnt this route from vcg-40-sfpg01).<\/li>
- PE2 successfully learnt the route 10.21.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE3<\/h5>\n\n\n\nFigure 20: PE3, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE3. The output confirmed:<\/p>\n\n\n\n
- PE3 successfully having BGP peer with vcg-40-nypg01 (10.253.246.2).<\/li>
- PE3 successfully learnt the route 10.21.42.0\/24 from PGW vcg-40-nypg01.<\/li>
- PE3 successfully learnt the route 10.11.41.0\/24 from MPLS backbone. Note that the Next Hop is 10.10.10.21 which is the PE1 loopback IP address, the local preference is 200. This means the route-map on PE1 and PE2 is working, such that packets prefer going to PE1 for destination 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
PGW Failover Test \u2013 without BFD<\/h3>\n\n\n\n
The test is to test how long does it take for the traffic to switch from vcg-40-sfpg01 to vcg-40-sfpg02. In order to perform this test, the Linux machine support-L1 (10.11.41.13) issue a continuous ping to support-R2 (10.21.42.23). In the middle of the ping, the vcg-40-sfpg01 will get power off, and the time required to fail over will get recorded.<\/p>\n\n\n\n
Before the ping failover test, a traceroute from support-L1 (10.11.41.13) to support-R2 (10.21.42.23) is capture as follow:<\/p>\n\n\n\nFigure 21: Client machine Support-L1 (10.11.41.13) traceroute to 10.21.42.23<\/figcaption><\/figure>\n\n\n\nThis traceroute confirmed the traffic path is \u201cLeft-Edge-T3 –> vcg-40-sfpg01 –> PE1 –> PE3 –>vcg-40-nypg01 –> Right-Edge-T3\u201d<\/p>\n\n\n\n
The test method is having Support-L1 (10.11.41.13) issue a continuous ping to Support-R2 (10.21.42.23), in the middle of the ping, PGW vcg-40-sfpg01 gets powered off. The following show the test result:<\/p>\n\n\n\nFigure 22: Ping test result for primary PGW failed, with default BGP timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 82. There are 129 ping missed during the failover, which is roughly around 130 seconds.<\/p>\n\n\n\n
The reader might wonder why the failover takes so long, since the SD-WAN Edge Left-Edge-T3 is having Overlay Tunnel to both vcg-40-sfpg01 and vcg-40-sfpg02, when vcg-40-sfpg01 gets powered off, the corresponding Overlay Tunnel gets terminated. As a result, Left-Edge-T3 should be able to switch to vcg-40-sfpg02 in sub-second. This understanding is correct. But let\u2019s look at the diagram again to refresh the memory:<\/p>\n\n\n\nFigure 23: Test Topology<\/figcaption><\/figure>\n\n\n\nThe failover takes long time because of the return traffic. The PE1 is configured to have route map to attract the return traffic. In order for the return traffic to be able to switch from PE1 to PE2, PE1 must stop advertising the 10.11.41.0\/24 route. PE1 will stop advertising when the BGP peer between PE1 and vcg-40-sfpg01 is down, with the default timer (60s keepalive and 180s hold timer), it can take a maximum of 180 seconds for the BGP peer to go down.<\/p>\n\n\n\n
One of the options to speed up the failover is having a smaller value of keepalive and hold timer. The following is the screen capture of repeating the same ping test with 3s keepalive and 9s hold timer:<\/p>\n\n\n\nFigure 24: Ping test result for primary PGW failed, with 3s keepalive and 9s hold timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 35. There are 9 ping missed during the failover, which the failover is roughly around 9-10 seconds.<\/p>\n\n\n\n
PGW and PE Router Configurations \u2013 with BFD<\/h4>\n\n\n\n
In this section, BFD will be enabled in the SD-WAN Gateways and PE routers. Other devices like the SD-WAN Edge will maintain the existing configuration.<\/p>\n\n\n\n
SD-WAN Gateway (Partner Gateway) \u2013 with BFD<\/h5>\n\n\n\n
The follow two screen captures show the PGW (vcg-40-sfpg01 and vcg-40-sfpg02) BGP peer with BFD enabled. In this test, the receive interval and transmit interval are both 300ms, the multiplier is 3.<\/p>\n\n\n\nFigure 25: vcg-40-sfpg01 BGP peer with 10.253.247.1 BFD enabled<\/figcaption><\/figure>\n\n\n\nFigure 26: vcg-40-sfpg02 BGP peer with 10.253.247.9 BFD enabled<\/figcaption><\/figure>\n\n\n\nPE router \u2013 with BFD<\/h4>\n\n\n\nPE1 configuration by enabling BFD for BGP peer 10.253.247.2<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/em><\/span>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 fall-over bfd<\/em><\/span>\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2 configuration by enabling BFD for BGP peer 10.253.247.10<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/span><\/em>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 fall-over bfd<\/span><\/em>\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe newly added configuration to enable BFD in the PE router is marked with italic<\/span><\/em> in the above configuration snippets.<\/p>\n\n\n\nNOTE: In the PE router sub-interface using for the test vrf, that is vrf Customer40b, the \u201cno bfd echo\u201d command is entered to disable the BFD echo mode. As of the VMware SD-WAN version 4.0, BFD echo mode is not supported, only BFD asynchronous mode is supported.<\/p><\/blockquote>\n\n\n\n
Verification of the BFD status<\/h3>\n\n\n\nVerification of BFD status at the PGW<\/h4>\n\n\n\n
There is now a “Gateway BFD Sessions” tab under “Monitor –> Routing –> BFD”<\/p>\n\n\n\nFigure 27: Gateway BFD Sessions<\/figcaption><\/figure>\n\n\n\nFrom the “Gateway BFD Sessions” GUI, we can see both vcg-40-sfpg01 and vcg-40-sfpg02 are having “UP” state of the BFD with the corresponding BGP peer, which means the BFD is working. In case more detail information is needed, in the PGW shell, there are two commands can check the BFD status, they are \u201cdebug.py –bfd_info\u201d and \u201cdebug.py –bfdd_dump\u201d. The output below is from PGW vcg-40-sfpg01 which shows the BFD peer is with “up” status. The following snippet shows the BFD debug.py output of vcg-40-sfpg01: <\/p>\n\n\n\n
vcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfd_info<\/span>\nEnterpriseId SEG Peer Address Local Address Detect Multiplier Transmit Interval Receive Interval Status\neb032999-da64-4b09-ab57-3b5f9949ffdb 0 10.253.247.1 10.253.247.2 3 300 300 UP\nvcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfdd_dump<\/span>\nshow running-config\nBuilding configuration\u2026\nCurrent configuration:\n!\nfrr version 7.0\nfrr defaults traditional\nhostname vcg-40-sfpg01\nlog file \/var\/log\/bfdd.log\n!\npassword zebra\n!\nline vty\naccess-class vty\n!\nbfd\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nno shutdown\n!\n!\nend\nshow bfd peers\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nID: 3238898200\nRemote ID: 1\nStatus: up\nUptime: 34 minute(s), 48 second(s)\nDiagnostics: ok\nRemote diagnostics: ok\nLocal timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 50ms\nRemote timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 0ms\nshow bfd peers counters\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nControl packet input: 9021 packets\nControl packet output: 81639 packets\nEcho packet input: 0 packets\nEcho packet output: 0 packets\nSession up events: 239\nSession down events: 238\nZebra notifications: 479\nNS UDP connections dump for entr ID: [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nState Recv-Q Send-Q Local Address:Port Peer Address:Port\nUNCONN 0 0 0.0.0.0:3784 0.0.0.0:* users:((\"bfdd\",pid=851,fd=16))\nUNCONN 0 0 0.0.0.0:49142 0.0.0.0:* users:((\"bfdd\",pid=851,fd=17))\nvcadmin@vcg-40-sfpg01:~$<\/pre>\n\n\n\nThe output of the debug commands from vcg-40-sfpg02 is very similar so it won\u2019t be repeated here.<\/p>\n\n\n\n
PE router Verification of BFD status<\/h5>\n\n\n\n
In the PE router, command \u201cshow bfd neighbors details\u201d is used to check the BFD status. The output below from PE1 confirmed the BFD status with PGW vcg-40-sfpg01 is up:<\/p>\n\n\n\n
PE1#show bfd neighbors details\nIPv4 Sessions\nNeighAddr LD\/RD RH\/RS State Int\n10.253.247.2 1\/3238898200 Up Up Et0\/2.341\nSession state is UP and not using echo function.\nSession Host: Software\nOurAddr: 10.253.247.1\nHandle: 1\nLocal Diag: 0, Demand mode: 0, Poll bit: 0\nMinTxInt: 300000, MinRxInt: 300000, Multiplier: 3\nReceived MinRxInt: 300000, Received Multiplier: 3\nHolddown (hits): 829(0), Hello (hits): 300(9183)\nRx Count: 9330, Rx Interval (ms) min\/max\/avg: 1\/301\/263 last: 71 ms ago\nTx Count: 9186, Tx Interval (ms) min\/max\/avg: 1\/310\/267 last: 206 ms ago\nElapsed time watermarks: 0 0 (last: 0)\nRegistered protocols: BGP CEF\nUptime: 00:40:51\nLast packet: Version: 1 - Diagnostic: 0\nState bit: Up - Demand bit: 0\nPoll bit: 0 - Final bit: 0\nC bit: 0\nMultiplier: 3 - Length: 24\nMy Discr.: 3238898200 - Your Discr.: 1\nMin tx interval: 300000 - Min rx interval: 300000\nMin Echo interval: 50000\nPE1#<\/pre>\n\n\n\nAgain, the output from PE2 is very similar to PE1 so it won\u2019t be included here.<\/p>\n\n\n\n
PGW Failover Test \u2013 with BFD<\/h3>\n\n\n\n
With BFD enabled between the PGW and PE router, let\u2019s repeat the ping Test from client support-L1 (10.11.41.13) to support-R2 (10.21.42.23), with a power off of primary PGW vcg-40-sfpg01. The following screen capture is the ping test result:<\/p>\n\n\n\nFigure 28: Ping test result from Support-L1 (10.11.41.13) to Support-R2 (10.21.42.23) with primary PGW failure, BFD is enabled in the PGW<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the power off of the primary PGW vcg-40-sfpg01 cause 1 ping loss (between icmp sequence 70 and 72). The failover time is around 1-2 seconds. Since the failover time is a lot faster. The ping test is being performed again with the additional parameter \u201c-i 0.1\u201d, this means the ping will be send every 0.1s instead of the default which is sent every 1s. The following is the screen capture of the ping test result with 0.1s interval:<\/p>\n\n\n\nFigure 29: Repeat of the ping test with primary PGW vcg-40-sfpg01, the ping interval is lower to 0.1s<\/figcaption><\/figure>\n\n\n\nThere are 8 ping loss (between sequence number 22 and 31), with each ping separate of 0.1s, this means 9 x 0.1s interval which result of 0.9s. As a result, it should be fine to conclude the failover time of the primary PGW failure is 1s with the BFD enabled BGP between PGW and PE router.<\/p>\n\n\n\n
Maximum BFD session supported per PGW<\/h3>\n\n\n\n
As of version 4.0, the maximum number of BFD sessions supported on PGW is 250. Although this post is not talking about the BFD on SD-WAN Edge, for documentation purpose, the maximum number of BFD sessions supported in the SD-WAN Edge is 50.<\/p>\n\n\n\n
This post “BGP with BFD enabled in SD-WAN Partner Gateway” ended here.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"Background In VMware SD-WAN (Velocloud) version 4.0, both SD-WAN Edge and SD-WAN Gateway added the support of Bidirectional Forwarding Detection (BFD). The objective of this post is to document the configuration of enabling BFD for the BGP peering between the SD-WAN Gateway (working as Partner Gateway) and PE router. There will be ping test result […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"templates\/template-fullwidth.php","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[9,8,5],"tags":[],"class_list":["post-168","post","type-post","status-publish","format-standard","hentry","category-partner-gateway","category-sd-wan-routing","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":36,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":241,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions\/241"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
PE2<\/h5>\n\n\n\n!\n! sub-interface for this SD-WAN Customer\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n!\n! BGP configuration to peer with PGW vcg-40-sfpg02\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nConfiguration to ensure the traffic path is symmetric<\/h5>\n\n\n\n
Although the Left-Edge-T3 is aware the vcg-40-sfpg01 is the primary PGW as it is at order position #1, the MPLS backbone also need to ensure the return traffic prefer landing on PE1 instead of landing on PE2. This is done by auto community mapping at the PGW and route-map to adjust local preference at the PE router.<\/p>\n\n\n\nFigure 12: Community Mapping<\/figcaption><\/figure>\n\n\n\nPGW is configured with community mapping with priority #1 PGW routes come with community 9998:111 (655229039) and priority #2 PGW routes come with community 9998:222 (655229150). That means, for the routes advertised by vcg-40-sfpg01 to the PE1, those routes will be having community 9998:111. For the routes advertised by vcg-40-sfpg02 to the PE2, those routes will be having community 9998:222.<\/p>\n\n\n\n
The following configurations are the route-map comm-set-localpref to make PE1 is the preferred return path as route from vcg-40-sfpg01 is set with local preference of 200 while route from vcg-40-sfpg02 is set with local preference of 101.<\/p>\n\n\n\n
PE1 route-map configuration:<\/p>\n\n\n\n
!\nip community-list standard comm-priority1 permit 655229039\nip community-list standard comm-priority2 permit 655229150\n!\n! Set local preference to 200 when route comes with community 9998:111, set local preference to 101 when route comes with community 9998:222\n!\nroute-map comm-set-localpref permit 10\n match community comm-priority1\n set local-preference 200\n!\nroute-map comm-set-localpref permit 20\n match community comm-priority2\n set local-preference 101\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe same route-map, hence same mechanism is applied to PE2 as well, the configuration of the PE2 route-map is like a mirror copy of PE1, so it is not repeated here.<\/p>\n\n\n\n
Verification on SD-WAN Edge, PGW and PE router<\/h3>\n\n\n\n
The section will perform some verification to ensure the SD-WAN Edge, PGW, PE router has learnt the route properly.<\/p>\n\n\n\n
Verification on SD-WAN Edge<\/h4>\n\n\n\nLeft-Edge-T3:<\/h5>\n\n\n\nFigure 13: Path and 10.21.42.0\/24 route on Left-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.21.42.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Left-Edge-T3 is able to form overlay tunnel to vcg-40-sfpg01 and vcg-40-sfpg02<\/li>
- Left-Edge-T3 learnt the route 10.21.42.0\/24 from both vcg-40-sfpg01 and vcg-40-sfpg02 where vcg-40-sfpg01 is the preferred Next Hop.<\/li><\/ol>\n\n\n\n
Right-Edge-T3:<\/h5>\n\n\n\nFigure 14: Path and 10.11.41.0\/24 route on Right-Edge-T3<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.11.41.0\u201d. The output confirmed:<\/p>\n\n\n\n
- Right-Edge-T3 is able to form overlay tunnels to vcg-40-nypg01.<\/li>
- Right-Edge-T3 learnt the route 10.11.41.0\/24 from vcg-40-nypg01.<\/li><\/ol>\n\n\n\n
Verification on PGW<\/h4>\n\n\n\nvcg-40-sfpg01:<\/h5>\n\n\n\nFigure 15: vcg-40-sfpg01 debug command output of bgp_view_summary and bgp_view <\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 15) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg01 has successfully established BGP peer with PE1 (10.253.247.1)<\/li>
- PGW vcg-40-sfpg01 learnt some routes from PE1, including the route 10.21.42.0\/24<\/li><\/ol>\n\n\n\n
vcg-40-sfpg02:<\/h5>\n\n\n\nFigure 16: vcg-40-sfpg02 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 16) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg02. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-sfpg02 has successfully established BGP peer with PE2 (10.253.247.9).<\/li>
- PGW vcg-40-sfpg02 learnt some routes from PE2, including the route 10.21.42.0\/24.<\/li><\/ol>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\nFigure 17: cg-40-nypg01 debug command output of bgp_view_summary and bgp_view<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-nypg01. The output confirmed:<\/p>\n\n\n\n
- PGW vcg-40-nypg01 has successfully established BGP peer with PE3 (10.253.246.1).<\/li>
- PGW vcg-40-nypg01 learnt some routes from PE3, including the route 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
Verification on PE router<\/h4>\n\n\n\nPE1<\/h5>\n\n\n\nFigure 18: PE1, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure (Figure 18) is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE1. The output confirmed:<\/p>\n\n\n\n
- PE1 successfully having BGP peer with vcg-40-sfpg01 (10.253.247.2).<\/li>
- PE1 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg01.<\/li>
- PE1 successfully learnt the route 10.11.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE2<\/h5>\n\n\n\nFigure 19: PE2, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE2. The output confirmed:<\/p>\n\n\n\n
- PE2 successfully having BGP peer with vcg-40-sfpg02 (10.253.247.10).<\/li>
- PE2 successfully learnt the route 10.11.41.0\/24 from PGW vcg-40-sfpg02 (and also from PE1 as PE1 learnt this route from vcg-40-sfpg01).<\/li>
- PE2 successfully learnt the route 10.21.42.0\/24 from MPLS backbone.<\/li><\/ol>\n\n\n\n
PE3<\/h5>\n\n\n\nFigure 20: PE3, show ip bgp summary and show ip bgp output<\/figcaption><\/figure>\n\n\n\nThe above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE3. The output confirmed:<\/p>\n\n\n\n
- PE3 successfully having BGP peer with vcg-40-nypg01 (10.253.246.2).<\/li>
- PE3 successfully learnt the route 10.21.42.0\/24 from PGW vcg-40-nypg01.<\/li>
- PE3 successfully learnt the route 10.11.41.0\/24 from MPLS backbone. Note that the Next Hop is 10.10.10.21 which is the PE1 loopback IP address, the local preference is 200. This means the route-map on PE1 and PE2 is working, such that packets prefer going to PE1 for destination 10.11.41.0\/24.<\/li><\/ol>\n\n\n\n
PGW Failover Test \u2013 without BFD<\/h3>\n\n\n\n
The test is to test how long does it take for the traffic to switch from vcg-40-sfpg01 to vcg-40-sfpg02. In order to perform this test, the Linux machine support-L1 (10.11.41.13) issue a continuous ping to support-R2 (10.21.42.23). In the middle of the ping, the vcg-40-sfpg01 will get power off, and the time required to fail over will get recorded.<\/p>\n\n\n\n
Before the ping failover test, a traceroute from support-L1 (10.11.41.13) to support-R2 (10.21.42.23) is capture as follow:<\/p>\n\n\n\nFigure 21: Client machine Support-L1 (10.11.41.13) traceroute to 10.21.42.23<\/figcaption><\/figure>\n\n\n\nThis traceroute confirmed the traffic path is \u201cLeft-Edge-T3 –> vcg-40-sfpg01 –> PE1 –> PE3 –>vcg-40-nypg01 –> Right-Edge-T3\u201d<\/p>\n\n\n\n
The test method is having Support-L1 (10.11.41.13) issue a continuous ping to Support-R2 (10.21.42.23), in the middle of the ping, PGW vcg-40-sfpg01 gets powered off. The following show the test result:<\/p>\n\n\n\nFigure 22: Ping test result for primary PGW failed, with default BGP timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 82. There are 129 ping missed during the failover, which is roughly around 130 seconds.<\/p>\n\n\n\n
The reader might wonder why the failover takes so long, since the SD-WAN Edge Left-Edge-T3 is having Overlay Tunnel to both vcg-40-sfpg01 and vcg-40-sfpg02, when vcg-40-sfpg01 gets powered off, the corresponding Overlay Tunnel gets terminated. As a result, Left-Edge-T3 should be able to switch to vcg-40-sfpg02 in sub-second. This understanding is correct. But let\u2019s look at the diagram again to refresh the memory:<\/p>\n\n\n\nFigure 23: Test Topology<\/figcaption><\/figure>\n\n\n\nThe failover takes long time because of the return traffic. The PE1 is configured to have route map to attract the return traffic. In order for the return traffic to be able to switch from PE1 to PE2, PE1 must stop advertising the 10.11.41.0\/24 route. PE1 will stop advertising when the BGP peer between PE1 and vcg-40-sfpg01 is down, with the default timer (60s keepalive and 180s hold timer), it can take a maximum of 180 seconds for the BGP peer to go down.<\/p>\n\n\n\n
One of the options to speed up the failover is having a smaller value of keepalive and hold timer. The following is the screen capture of repeating the same ping test with 3s keepalive and 9s hold timer:<\/p>\n\n\n\nFigure 24: Ping test result for primary PGW failed, with 3s keepalive and 9s hold timer<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 35. There are 9 ping missed during the failover, which the failover is roughly around 9-10 seconds.<\/p>\n\n\n\n
PGW and PE Router Configurations \u2013 with BFD<\/h4>\n\n\n\n
In this section, BFD will be enabled in the SD-WAN Gateways and PE routers. Other devices like the SD-WAN Edge will maintain the existing configuration.<\/p>\n\n\n\n
SD-WAN Gateway (Partner Gateway) \u2013 with BFD<\/h5>\n\n\n\n
The follow two screen captures show the PGW (vcg-40-sfpg01 and vcg-40-sfpg02) BGP peer with BFD enabled. In this test, the receive interval and transmit interval are both 300ms, the multiplier is 3.<\/p>\n\n\n\nFigure 25: vcg-40-sfpg01 BGP peer with 10.253.247.1 BFD enabled<\/figcaption><\/figure>\n\n\n\nFigure 26: vcg-40-sfpg02 BGP peer with 10.253.247.9 BFD enabled<\/figcaption><\/figure>\n\n\n\nPE router \u2013 with BFD<\/h4>\n\n\n\nPE1 configuration by enabling BFD for BGP peer 10.253.247.2<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/em><\/span>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 fall-over bfd<\/em><\/span>\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2 configuration by enabling BFD for BGP peer 10.253.247.10<\/h5>\n\n\n\n!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/span><\/em>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 fall-over bfd<\/span><\/em>\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nThe newly added configuration to enable BFD in the PE router is marked with italic<\/span><\/em> in the above configuration snippets.<\/p>\n\n\n\nNOTE: In the PE router sub-interface using for the test vrf, that is vrf Customer40b, the \u201cno bfd echo\u201d command is entered to disable the BFD echo mode. As of the VMware SD-WAN version 4.0, BFD echo mode is not supported, only BFD asynchronous mode is supported.<\/p><\/blockquote>\n\n\n\n
Verification of the BFD status<\/h3>\n\n\n\nVerification of BFD status at the PGW<\/h4>\n\n\n\n
There is now a “Gateway BFD Sessions” tab under “Monitor –> Routing –> BFD”<\/p>\n\n\n\nFigure 27: Gateway BFD Sessions<\/figcaption><\/figure>\n\n\n\nFrom the “Gateway BFD Sessions” GUI, we can see both vcg-40-sfpg01 and vcg-40-sfpg02 are having “UP” state of the BFD with the corresponding BGP peer, which means the BFD is working. In case more detail information is needed, in the PGW shell, there are two commands can check the BFD status, they are \u201cdebug.py –bfd_info\u201d and \u201cdebug.py –bfdd_dump\u201d. The output below is from PGW vcg-40-sfpg01 which shows the BFD peer is with “up” status. The following snippet shows the BFD debug.py output of vcg-40-sfpg01: <\/p>\n\n\n\n
vcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfd_info<\/span>\nEnterpriseId SEG Peer Address Local Address Detect Multiplier Transmit Interval Receive Interval Status\neb032999-da64-4b09-ab57-3b5f9949ffdb 0 10.253.247.1 10.253.247.2 3 300 300 UP\nvcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfdd_dump<\/span>\nshow running-config\nBuilding configuration\u2026\nCurrent configuration:\n!\nfrr version 7.0\nfrr defaults traditional\nhostname vcg-40-sfpg01\nlog file \/var\/log\/bfdd.log\n!\npassword zebra\n!\nline vty\naccess-class vty\n!\nbfd\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nno shutdown\n!\n!\nend\nshow bfd peers\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nID: 3238898200\nRemote ID: 1\nStatus: up\nUptime: 34 minute(s), 48 second(s)\nDiagnostics: ok\nRemote diagnostics: ok\nLocal timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 50ms\nRemote timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 0ms\nshow bfd peers counters\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nControl packet input: 9021 packets\nControl packet output: 81639 packets\nEcho packet input: 0 packets\nEcho packet output: 0 packets\nSession up events: 239\nSession down events: 238\nZebra notifications: 479\nNS UDP connections dump for entr ID: [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nState Recv-Q Send-Q Local Address:Port Peer Address:Port\nUNCONN 0 0 0.0.0.0:3784 0.0.0.0:* users:((\"bfdd\",pid=851,fd=16))\nUNCONN 0 0 0.0.0.0:49142 0.0.0.0:* users:((\"bfdd\",pid=851,fd=17))\nvcadmin@vcg-40-sfpg01:~$<\/pre>\n\n\n\nThe output of the debug commands from vcg-40-sfpg02 is very similar so it won\u2019t be repeated here.<\/p>\n\n\n\n
PE router Verification of BFD status<\/h5>\n\n\n\n
In the PE router, command \u201cshow bfd neighbors details\u201d is used to check the BFD status. The output below from PE1 confirmed the BFD status with PGW vcg-40-sfpg01 is up:<\/p>\n\n\n\n
PE1#show bfd neighbors details\nIPv4 Sessions\nNeighAddr LD\/RD RH\/RS State Int\n10.253.247.2 1\/3238898200 Up Up Et0\/2.341\nSession state is UP and not using echo function.\nSession Host: Software\nOurAddr: 10.253.247.1\nHandle: 1\nLocal Diag: 0, Demand mode: 0, Poll bit: 0\nMinTxInt: 300000, MinRxInt: 300000, Multiplier: 3\nReceived MinRxInt: 300000, Received Multiplier: 3\nHolddown (hits): 829(0), Hello (hits): 300(9183)\nRx Count: 9330, Rx Interval (ms) min\/max\/avg: 1\/301\/263 last: 71 ms ago\nTx Count: 9186, Tx Interval (ms) min\/max\/avg: 1\/310\/267 last: 206 ms ago\nElapsed time watermarks: 0 0 (last: 0)\nRegistered protocols: BGP CEF\nUptime: 00:40:51\nLast packet: Version: 1 - Diagnostic: 0\nState bit: Up - Demand bit: 0\nPoll bit: 0 - Final bit: 0\nC bit: 0\nMultiplier: 3 - Length: 24\nMy Discr.: 3238898200 - Your Discr.: 1\nMin tx interval: 300000 - Min rx interval: 300000\nMin Echo interval: 50000\nPE1#<\/pre>\n\n\n\nAgain, the output from PE2 is very similar to PE1 so it won\u2019t be included here.<\/p>\n\n\n\n
PGW Failover Test \u2013 with BFD<\/h3>\n\n\n\n
With BFD enabled between the PGW and PE router, let\u2019s repeat the ping Test from client support-L1 (10.11.41.13) to support-R2 (10.21.42.23), with a power off of primary PGW vcg-40-sfpg01. The following screen capture is the ping test result:<\/p>\n\n\n\nFigure 28: Ping test result from Support-L1 (10.11.41.13) to Support-R2 (10.21.42.23) with primary PGW failure, BFD is enabled in the PGW<\/figcaption><\/figure>\n\n\n\nFrom the ping screen capture, the power off of the primary PGW vcg-40-sfpg01 cause 1 ping loss (between icmp sequence 70 and 72). The failover time is around 1-2 seconds. Since the failover time is a lot faster. The ping test is being performed again with the additional parameter \u201c-i 0.1\u201d, this means the ping will be send every 0.1s instead of the default which is sent every 1s. The following is the screen capture of the ping test result with 0.1s interval:<\/p>\n\n\n\nFigure 29: Repeat of the ping test with primary PGW vcg-40-sfpg01, the ping interval is lower to 0.1s<\/figcaption><\/figure>\n\n\n\nThere are 8 ping loss (between sequence number 22 and 31), with each ping separate of 0.1s, this means 9 x 0.1s interval which result of 0.9s. As a result, it should be fine to conclude the failover time of the primary PGW failure is 1s with the BFD enabled BGP between PGW and PE router.<\/p>\n\n\n\n
Maximum BFD session supported per PGW<\/h3>\n\n\n\n
As of version 4.0, the maximum number of BFD sessions supported on PGW is 250. Although this post is not talking about the BFD on SD-WAN Edge, for documentation purpose, the maximum number of BFD sessions supported in the SD-WAN Edge is 50.<\/p>\n\n\n\n
This post “BGP with BFD enabled in SD-WAN Partner Gateway” ended here.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"Background In VMware SD-WAN (Velocloud) version 4.0, both SD-WAN Edge and SD-WAN Gateway added the support of Bidirectional Forwarding Detection (BFD). The objective of this post is to document the configuration of enabling BFD for the BGP peering between the SD-WAN Gateway (working as Partner Gateway) and PE router. There will be ping test result […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"templates\/template-fullwidth.php","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[9,8,5],"tags":[],"class_list":["post-168","post","type-post","status-publish","format-standard","hentry","category-partner-gateway","category-sd-wan-routing","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":36,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":241,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions\/241"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Configuration to ensure the traffic path is symmetric<\/h5>\n\n\n\n
Although the Left-Edge-T3 is aware the vcg-40-sfpg01 is the primary PGW as it is at order position #1, the MPLS backbone also need to ensure the return traffic prefer landing on PE1 instead of landing on PE2. This is done by auto community mapping at the PGW and route-map to adjust local preference at the PE router.<\/p>\n\n\n\n PGW is configured with community mapping with priority #1 PGW routes come with community 9998:111 (655229039) and priority #2 PGW routes come with community 9998:222 (655229150). That means, for the routes advertised by vcg-40-sfpg01 to the PE1, those routes will be having community 9998:111. For the routes advertised by vcg-40-sfpg02 to the PE2, those routes will be having community 9998:222.<\/p>\n\n\n\n The following configurations are the route-map comm-set-localpref to make PE1 is the preferred return path as route from vcg-40-sfpg01 is set with local preference of 200 while route from vcg-40-sfpg02 is set with local preference of 101.<\/p>\n\n\n\n PE1 route-map configuration:<\/p>\n\n\n\n The same route-map, hence same mechanism is applied to PE2 as well, the configuration of the PE2 route-map is like a mirror copy of PE1, so it is not repeated here.<\/p>\n\n\n\n The section will perform some verification to ensure the SD-WAN Edge, PGW, PE router has learnt the route properly.<\/p>\n\n\n\n The above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.21.42.0\u201d. The output confirmed:<\/p>\n\n\n\n The above figure is the screen capture of output of \u201cdebug.py –path\u201d and \u201cdebug.py –routes 10.11.41.0\u201d. The output confirmed:<\/p>\n\n\n\n The above figure (Figure 15) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg01. The output confirmed:<\/p>\n\n\n\n The above figure (Figure 16) is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-sfpg02. The output confirmed:<\/p>\n\n\n\n The above figure is the screen capture of output of \u201cdebug.py –bgp_view_summary\u201d and \u201cdebug.py –bgp_view\u201d of PGW vcg-40-nypg01. The output confirmed:<\/p>\n\n\n\n The above figure (Figure 18) is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE1. The output confirmed:<\/p>\n\n\n\n The above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE2. The output confirmed:<\/p>\n\n\n\n The above figure is the screen capture of output of \u201cshow ip bgp vpnv4 vrf <VRF name> summary\u201d and \u201cshow ip bgp vpnv4 vrf <VRF name>\u201d of PE3. The output confirmed:<\/p>\n\n\n\n The test is to test how long does it take for the traffic to switch from vcg-40-sfpg01 to vcg-40-sfpg02. In order to perform this test, the Linux machine support-L1 (10.11.41.13) issue a continuous ping to support-R2 (10.21.42.23). In the middle of the ping, the vcg-40-sfpg01 will get power off, and the time required to fail over will get recorded.<\/p>\n\n\n\n Before the ping failover test, a traceroute from support-L1 (10.11.41.13) to support-R2 (10.21.42.23) is capture as follow:<\/p>\n\n\n\n This traceroute confirmed the traffic path is \u201cLeft-Edge-T3 –> vcg-40-sfpg01 –> PE1 –> PE3 –>vcg-40-nypg01 –> Right-Edge-T3\u201d<\/p>\n\n\n\n The test method is having Support-L1 (10.11.41.13) issue a continuous ping to Support-R2 (10.21.42.23), in the middle of the ping, PGW vcg-40-sfpg01 gets powered off. The following show the test result:<\/p>\n\n\n\n From the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 82. There are 129 ping missed during the failover, which is roughly around 130 seconds.<\/p>\n\n\n\n The reader might wonder why the failover takes so long, since the SD-WAN Edge Left-Edge-T3 is having Overlay Tunnel to both vcg-40-sfpg01 and vcg-40-sfpg02, when vcg-40-sfpg01 gets powered off, the corresponding Overlay Tunnel gets terminated. As a result, Left-Edge-T3 should be able to switch to vcg-40-sfpg02 in sub-second. This understanding is correct. But let\u2019s look at the diagram again to refresh the memory:<\/p>\n\n\n\n The failover takes long time because of the return traffic. The PE1 is configured to have route map to attract the return traffic. In order for the return traffic to be able to switch from PE1 to PE2, PE1 must stop advertising the 10.11.41.0\/24 route. PE1 will stop advertising when the BGP peer between PE1 and vcg-40-sfpg01 is down, with the default timer (60s keepalive and 180s hold timer), it can take a maximum of 180 seconds for the BGP peer to go down.<\/p>\n\n\n\n One of the options to speed up the failover is having a smaller value of keepalive and hold timer. The following is the screen capture of repeating the same ping test with 3s keepalive and 9s hold timer:<\/p>\n\n\n\n From the ping screen capture, the PGW vcg-40-pg01 get powered off right after the icmp sequence number 35. There are 9 ping missed during the failover, which the failover is roughly around 9-10 seconds.<\/p>\n\n\n\n In this section, BFD will be enabled in the SD-WAN Gateways and PE routers. Other devices like the SD-WAN Edge will maintain the existing configuration.<\/p>\n\n\n\n The follow two screen captures show the PGW (vcg-40-sfpg01 and vcg-40-sfpg02) BGP peer with BFD enabled. In this test, the receive interval and transmit interval are both 300ms, the multiplier is 3.<\/p>\n\n\n\n The newly added configuration to enable BFD in the PE router is marked with italic<\/span><\/em> in the above configuration snippets.<\/p>\n\n\n\n NOTE: In the PE router sub-interface using for the test vrf, that is vrf Customer40b, the \u201cno bfd echo\u201d command is entered to disable the BFD echo mode. As of the VMware SD-WAN version 4.0, BFD echo mode is not supported, only BFD asynchronous mode is supported.<\/p><\/blockquote>\n\n\n\n There is now a “Gateway BFD Sessions” tab under “Monitor –> Routing –> BFD”<\/p>\n\n\n\n From the “Gateway BFD Sessions” GUI, we can see both vcg-40-sfpg01 and vcg-40-sfpg02 are having “UP” state of the BFD with the corresponding BGP peer, which means the BFD is working. In case more detail information is needed, in the PGW shell, there are two commands can check the BFD status, they are \u201cdebug.py –bfd_info\u201d and \u201cdebug.py –bfdd_dump\u201d. The output below is from PGW vcg-40-sfpg01 which shows the BFD peer is with “up” status. The following snippet shows the BFD debug.py output of vcg-40-sfpg01: <\/p>\n\n\n\n The output of the debug commands from vcg-40-sfpg02 is very similar so it won\u2019t be repeated here.<\/p>\n\n\n\n In the PE router, command \u201cshow bfd neighbors details\u201d is used to check the BFD status. The output below from PE1 confirmed the BFD status with PGW vcg-40-sfpg01 is up:<\/p>\n\n\n\n Again, the output from PE2 is very similar to PE1 so it won\u2019t be included here.<\/p>\n\n\n\n With BFD enabled between the PGW and PE router, let\u2019s repeat the ping Test from client support-L1 (10.11.41.13) to support-R2 (10.21.42.23), with a power off of primary PGW vcg-40-sfpg01. The following screen capture is the ping test result:<\/p>\n\n\n\n From the ping screen capture, the power off of the primary PGW vcg-40-sfpg01 cause 1 ping loss (between icmp sequence 70 and 72). The failover time is around 1-2 seconds. Since the failover time is a lot faster. The ping test is being performed again with the additional parameter \u201c-i 0.1\u201d, this means the ping will be send every 0.1s instead of the default which is sent every 1s. The following is the screen capture of the ping test result with 0.1s interval:<\/p>\n\n\n\n There are 8 ping loss (between sequence number 22 and 31), with each ping separate of 0.1s, this means 9 x 0.1s interval which result of 0.9s. As a result, it should be fine to conclude the failover time of the primary PGW failure is 1s with the BFD enabled BGP between PGW and PE router.<\/p>\n\n\n\n As of version 4.0, the maximum number of BFD sessions supported on PGW is 250. Although this post is not talking about the BFD on SD-WAN Edge, for documentation purpose, the maximum number of BFD sessions supported in the SD-WAN Edge is 50.<\/p>\n\n\n\n This post “BGP with BFD enabled in SD-WAN Partner Gateway” ended here.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":" Background In VMware SD-WAN (Velocloud) version 4.0, both SD-WAN Edge and SD-WAN Gateway added the support of Bidirectional Forwarding Detection (BFD). The objective of this post is to document the configuration of enabling BFD for the BGP peering between the SD-WAN Gateway (working as Partner Gateway) and PE router. There will be ping test result […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"templates\/template-fullwidth.php","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[9,8,5],"tags":[],"class_list":["post-168","post","type-post","status-publish","format-standard","hentry","category-partner-gateway","category-sd-wan-routing","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":36,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":241,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/168\/revisions\/241"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}!\nip community-list standard comm-priority1 permit 655229039\nip community-list standard comm-priority2 permit 655229150\n!\n! Set local preference to 200 when route comes with community 9998:111, set local preference to 101 when route comes with community 9998:222\n!\nroute-map comm-set-localpref permit 10\n match community comm-priority1\n set local-preference 200\n!\nroute-map comm-set-localpref permit 20\n match community comm-priority2\n set local-preference 101\n!\n! BGP configuration to peer with PGW vcg-40-sfpg01\n!\nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\n
Verification on SD-WAN Edge, PGW and PE router<\/h3>\n\n\n\n
Verification on SD-WAN Edge<\/h4>\n\n\n\n
Left-Edge-T3:<\/h5>\n\n\n\n
Right-Edge-T3:<\/h5>\n\n\n\n
Verification on PGW<\/h4>\n\n\n\n
vcg-40-sfpg01:<\/h5>\n\n\n\n
vcg-40-sfpg02:<\/h5>\n\n\n\n
vcg-40-nypg01<\/h5>\n\n\n\n
Verification on PE router<\/h4>\n\n\n\n
PE1<\/h5>\n\n\n\n
PE2<\/h5>\n\n\n\n
PE3<\/h5>\n\n\n\n
PGW Failover Test \u2013 without BFD<\/h3>\n\n\n\n
PGW and PE Router Configurations \u2013 with BFD<\/h4>\n\n\n\n
SD-WAN Gateway (Partner Gateway) \u2013 with BFD<\/h5>\n\n\n\n
PE router \u2013 with BFD<\/h4>\n\n\n\n
PE1 configuration by enabling BFD for BGP peer 10.253.247.2<\/h5>\n\n\n\n
!\ninterface Ethernet0\/2.341\n encapsulation dot1Q 341\n ip vrf forwarding Customer40b\n ip address 10.253.247.1 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/em><\/span>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.0 mask 255.255.255.248\n network 10.253.247.120 mask 255.255.255.248\n neighbor 10.253.247.2 remote-as 65121\n neighbor 10.253.247.2 fall-over bfd<\/em><\/span>\n neighbor 10.253.247.2 activate\n neighbor 10.253.247.2 send-community both\n neighbor 10.253.247.2 soft-reconfiguration inbound\n neighbor 10.253.247.2 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nPE2 configuration by enabling BFD for BGP peer 10.253.247.10<\/h5>\n\n\n\n
!\ninterface Ethernet0\/2.342\n encapsulation dot1Q 342\n ip vrf forwarding Customer40b\n ip address 10.253.247.9 255.255.255.248\n bfd interval 300 min_rx 300 multiplier 3\n no bfd echo<\/span><\/em>\n!\n \nrouter bgp 9998\n address-family ipv4 vrf Customer40b\n network 10.253.247.8 mask 255.255.255.248\n network 10.253.247.128 mask 255.255.255.248\n neighbor 10.253.247.10 remote-as 65122\n neighbor 10.253.247.10 fall-over bfd<\/span><\/em>\n neighbor 10.253.247.10 activate\n neighbor 10.253.247.10 send-community both\n neighbor 10.253.247.10 soft-reconfiguration inbound\n neighbor 10.253.247.10 route-map comm-set-localpref in\n exit-address-family<\/pre>\n\n\n\nVerification of the BFD status<\/h3>\n\n\n\n
Verification of BFD status at the PGW<\/h4>\n\n\n\n
vcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfd_info<\/span>\nEnterpriseId SEG Peer Address Local Address Detect Multiplier Transmit Interval Receive Interval Status\neb032999-da64-4b09-ab57-3b5f9949ffdb 0 10.253.247.1 10.253.247.2 3 300 300 UP\nvcadmin@vcg-40-sfpg01:~$ sudo \/opt\/vc\/bin\/debug.py --bfdd_dump<\/span>\nshow running-config\nBuilding configuration\u2026\nCurrent configuration:\n!\nfrr version 7.0\nfrr defaults traditional\nhostname vcg-40-sfpg01\nlog file \/var\/log\/bfdd.log\n!\npassword zebra\n!\nline vty\naccess-class vty\n!\nbfd\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nno shutdown\n!\n!\nend\nshow bfd peers\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nID: 3238898200\nRemote ID: 1\nStatus: up\nUptime: 34 minute(s), 48 second(s)\nDiagnostics: ok\nRemote diagnostics: ok\nLocal timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 50ms\nRemote timers:\nReceive interval: 300ms\nTransmission interval: 300ms\nEcho transmission interval: 0ms\nshow bfd peers counters\nBFD Peers:\npeer 10.253.247.1 local-address 10.253.247.2 vrf [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nControl packet input: 9021 packets\nControl packet output: 81639 packets\nEcho packet input: 0 packets\nEcho packet output: 0 packets\nSession up events: 239\nSession down events: 238\nZebra notifications: 479\nNS UDP connections dump for entr ID: [eb032999-da64-4b09-ab57-3b5f9949ffdb:0:1]\nState Recv-Q Send-Q Local Address:Port Peer Address:Port\nUNCONN 0 0 0.0.0.0:3784 0.0.0.0:* users:((\"bfdd\",pid=851,fd=16))\nUNCONN 0 0 0.0.0.0:49142 0.0.0.0:* users:((\"bfdd\",pid=851,fd=17))\nvcadmin@vcg-40-sfpg01:~$<\/pre>\n\n\n\nPE router Verification of BFD status<\/h5>\n\n\n\n
PE1#show bfd neighbors details\nIPv4 Sessions\nNeighAddr LD\/RD RH\/RS State Int\n10.253.247.2 1\/3238898200 Up Up Et0\/2.341\nSession state is UP and not using echo function.\nSession Host: Software\nOurAddr: 10.253.247.1\nHandle: 1\nLocal Diag: 0, Demand mode: 0, Poll bit: 0\nMinTxInt: 300000, MinRxInt: 300000, Multiplier: 3\nReceived MinRxInt: 300000, Received Multiplier: 3\nHolddown (hits): 829(0), Hello (hits): 300(9183)\nRx Count: 9330, Rx Interval (ms) min\/max\/avg: 1\/301\/263 last: 71 ms ago\nTx Count: 9186, Tx Interval (ms) min\/max\/avg: 1\/310\/267 last: 206 ms ago\nElapsed time watermarks: 0 0 (last: 0)\nRegistered protocols: BGP CEF\nUptime: 00:40:51\nLast packet: Version: 1 - Diagnostic: 0\nState bit: Up - Demand bit: 0\nPoll bit: 0 - Final bit: 0\nC bit: 0\nMultiplier: 3 - Length: 24\nMy Discr.: 3238898200 - Your Discr.: 1\nMin tx interval: 300000 - Min rx interval: 300000\nMin Echo interval: 50000\nPE1#<\/pre>\n\n\n\n
PGW Failover Test \u2013 with BFD<\/h3>\n\n\n\n
Maximum BFD session supported per PGW<\/h3>\n\n\n\n