{"id":248,"date":"2021-01-27T10:32:49","date_gmt":"2021-01-27T10:32:49","guid":{"rendered":"https:\/\/www.sdwan2.com\/?p=248"},"modified":"2021-01-27T10:32:49","modified_gmt":"2021-01-27T10:32:49","slug":"local-internet-breakout-with-two-internet-links-vmware-sd-wan-by-velocloud","status":"publish","type":"post","link":"https:\/\/www.sdwan2.com\/index.php\/2021\/01\/27\/local-internet-breakout-with-two-internet-links-vmware-sd-wan-by-velocloud\/","title":{"rendered":"Local Internet Breakout with two Internet Links – VMware SD-WAN by Velocloud"},"content":{"rendered":"\n

Background<\/h3>\n\n\n\n

For users behind the VMware SD-WAN Edge (that is user on the LAN side), one of the options for access Internet is local Internet breakout. When the SD-WAN Edge connected to more than one Internet link, questions like \u201cwhich link will be picked for local breakout?\u201d, \u201ccan the SD-WAN Edge perform traffic load balance?\u201d come up. In this post, will dig out the behavior of VMware SD-WAN local Internet breakout when the SD-WAN Edge is connected with two Internet links.<\/p>\n\n\n\n

Lab environment<\/h3>\n\n\n\n

In order to verify the content of this post is correct or not, tests will be conducted in a lab environment for verification. The lab is a closed environment (dark site), such that the Internet is a simulated Internet.<\/p>\n\n\n\n

The following diagram shows the topology for the lab:<\/p>\n\n\n\n

\"\"
Figure 1<\/figcaption><\/figure>\n\n\n\n

The topology consists of a VMware SD-WAN Orchestrator, also called Velocloud Orchestrator (VCO), with IP address 24.17.0.53. There are two VMware SD-WAN Gateway, also called Velocloud Gateway (VCG), they are vcg-40-sfpg01 (24.11.0.54) and vcg-40-sfpg02 (24.11.0.55).<\/p>\n\n\n\n

There is a VMware Edge, also called Velocloud Edge (VCE), the name of this VCE is Edge-1. Edge-1 is connected to two Internet links on GE3 and GE4 with IP address 98.1.2.19 and 184.1.2.27 (In test 5.1, this IP is changed to 184.1.2.30) respectively. There are two Internet links for verify different situation with different characteristics of each Internet link. The LAN facing side of Edge-1 is GE2 with IP address 192.168.200.254. There is a client machine named Client100 with IP address 192.168.200.100 to generate traffic to test the local Internet breakout.<\/p>\n\n\n\n

There are also two servers namely wordpress05.lab.local (43.254.254.14) and wordpress06.lab.local (24.12.0.14). These two servers are hosting some services, such as WordPress, iperf3 and SSH server.<\/p>\n\n\n\n

In the Edge-1, Internet link on GE3 is intentionally configured as 5M (both up and down), while the Internet link on GE4 is intentionally configured as 10M (both up and down). The Internet links are configured to relatively small bandwidth because that will allow saturate the link easier during the test of load balancing or aggregation situation. The follow is the screen capture of the Monitor –> Overview of the Edge-1:<\/p>\n\n\n\n

\"\"
Figure 2<\/figcaption><\/figure>\n\n\n\n

The software version<\/h3>\n\n\n\n

VCO: 4.0.1
VCG: R401-20201124-GA-53090-64ce7e02d2
VCE: R401-20201110-GA-MGMT-IP-e3fe6a0725<\/p>\n\n\n\n

About the GREEN, YELLOW, RED threshold for the WAN link<\/h3>\n\n\n\n

The VMware SD-WAN is an overlay tunnel technology, this overlay tunnel is sometimes called Dynamic Multipath Optimization (DMPO), the tunnels are being checked continuously for the latency, jitter and packet loss. There are pre-defined threshold to classify the link as \u201cGREEN<\/span>\u201d, \u201cYELLOW<\/span>\u201d or \u201cRED<\/span>\u201d based on different value of latency, jitter and packet loss. The pre-defined threshold can be found in this article: https:\/\/kb.vmware.com\/s\/article\/2733094<\/a><\/p>\n\n\n\n

The follow table is a re-cap of the threshold value for the reader\u2019s reference:<\/p>\n\n\n\n

\"\"
Figure 3<\/figcaption><\/figure>\n\n\n\n

In this post, the Internet link will describe as \u201cGREEN\u201d, \u201cYELLOW\u201d or \u201cRED\u201d, the matrix of different grade of the Internet link quality can be referred to Figure 3 above.<\/p>\n\n\n\n

Some housekeeping before going into the tests and conclusions<\/h3>\n\n\n\n

In usual situation, the SD-WAN Edge gets assigned with at least two SD-WAN Gateways. In this lab test, the SD-WAN Edge, Edge-1, is specifically configured with Partner Gateway, such that only a single SD-WAN Gateway is assigned. This is because when introducing packet loss during the test, if there are multiple Gateways (which result of multiple tunnels), the packet loss will spread randomly to different tunnels so some tunnels get more packet loss while some tunnels get less packet loss. This will make setting the WAN link to the desired color more difficult. As a result, in this post, the Edge-1 is only configured with a single SD-WAN Gateway except in Test 5.1. In all the lab tests expect Test 5.1 in this post, the Edge-1 determines the packet loss, jitter and latency by measuring these three parameters with the SD-WAN Gateway, vcg-40-sfpg02 (24.11.0.55), over the overlay tunnel.<\/p>\n\n\n\n

Local Internet breakout behavior<\/h3>\n\n\n\n

There are a few common questions about local Internet breakout, which will be addressed in this post:<\/p>\n\n\n\n

  1. When use \u201cAuto\u201d link steering, will the VCE select the better link? Such as automatically select the GREEN link over RED link?<\/strong><\/li>
  2. When use \u201cAuto\u201d link steering, and every links are GREEN, which link will be picked?<\/strong><\/li>
  3. Is there any load balancing or link aggregation possible for local Internet breakout with multiple Internet WAN links?<\/strong><\/li>
  4. What happen for link steering \u201cPreferred\u201d, \u201cAvailable\u201d and \u201cMandatory\u201d options?<\/strong><\/li>
  5. Is there any source IP persistence in \u201cAuto\u201d link steering?<\/strong><\/li><\/ol>\n\n\n\n\n\n\n\n

    When use \u201cAuto\u201d link steering, will the VCE select the better link? Such as automatically select the GREEN link over RED link?<\/h3>\n\n\n\n

    Edge Configuration for Test 1.1 to Test 1.4<\/h4>\n\n\n\n

    The Edge-1 is configured with local Internet breakout, such that all the UDP traffic destinated to Internet will be marked as High Priority and Realtime Class, while all other traffic (TCP and ICMP) destinated to Internet will be marked as Normal Priority and Transactional Class. In addition, the link steering is configured to \u201cAuto\u201d<\/p>\n\n\n\n

    The following screen capture shows the business policy configuration:<\/p>\n\n\n\n

    \"\"
    Figure 4<\/figcaption><\/figure>\n\n\n\n

    Test 1.1 \u2013 GE3 (98.1.2.19) is green while GE4 (184.1.2.27) is YELLOW due to packet loss<\/h4>\n\n\n\n

    This part will test if the Edge-1 will pick local Internet breakout with the GREEN link if the other one is YELLOW or RED. The test is based on web based traffic, which is TCP. TCP traffic is marked as Normal Priority and Transactional Class. The first test is based on packet loss, such that GE4 \u2013 184.1.2.27 is yellow due to 2% packet loss, while GE3 \u2013 98.1.2.19 with no packet loss. The follow table illustrate the link condition:<\/p>\n\n\n\n

    \"\"
    Figure 5<\/figcaption><\/figure>\n\n\n\n

    NOTE*: The lab environment is not able to make a steady exact 2% packet loss, so the value of packet loss will have some derivation. However, the link color (GREEN, YELLOW, RED) will be ensured falling into the expected color level.<\/p>\n\n\n\n

    The test is by using Client100 (192.168.200.100) browser to access wordpress05.lab.local (43.254.254.14), and the result is by checking the wordpress05 access log to confirm which WAN link is being selected. Before accessing the wordpress05, confirm the link status and color by \u201cdebug.py –dec\u201d in the Edge-1:<\/p>\n\n\n\n

    \"\"
    Figure 6<\/figcaption><\/figure>\n\n\n\n

    Confirmed the \u201cGE4 \u2013 184.1.2.27\u201d is YELLOW due to packet loss (for transactional).<\/p>\n\n\n\n

    The following is the web server access log from wordpress05:<\/p>\n\n\n\n

    \"\"
    Figure 7<\/figcaption><\/figure>\n\n\n\n

    From the wordpress05 access log, the Edge-1 is picking 98.1.2.19, that is \u201cGE3 \u2013 98.1.2.19\u201d as the link for local Internet breakout. This means, Edge-1 selected the GREEN link over YELLOW link for local Internet breakout.<\/p>\n\n\n\n

    Test 1.2 – GE3 (98.1.2.19) is YELLOW due to packet loss while GE4 (184.1.2.27) is GREEN<\/h4>\n\n\n\n

    To make sure Edge-1 picking \u201cGE3 \u2013 98.1.2.19\u201d is due to \u201cGE4 \u2013 184.1.2.27\u201d is YELLOW, let\u2019s swap the condition between two links. In this Test 1.2, make \u201cGE3 \u2013 98.1.2.19\u201d YELLOW by 2% packet loss, and \u201cGE4 \u2013 184.1.2.27\u201d GREEN with no packet loss. The follow table illustrate the link condition in the Test 1.2:<\/p>\n\n\n\n

    \"\"
    Figure 8<\/figcaption><\/figure>\n\n\n\n

    Before accessing wordpress05.lab.local by Client100 (192.168.200.100), check the WAN link status by \u201cdebug.py –path\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 9<\/figcaption><\/figure>\n\n\n\n

    The following is the web server access log from wordpress05 in Test 1.2:<\/p>\n\n\n\n

    \"\"
    Figure 10<\/figcaption><\/figure>\n\n\n\n

    From the access log, Edge-1 picked \u201cGE4 \u2013 184.1.2.27\u201d for local Internet breakout. This means Edge-1 again picks the GREEN \u201cGE4 \u2013 184.1.2.27\u201d instead of YELLOW \u201cGE3 \u2013 98.1.2.19\u201d. From Test 1.1 and Test 1.2, here can conclude Edge-1 picks the GREEN WAN link over the YELLOW WAN link for local Internet breakout, where the YELLOW is result of packet loss.<\/p>\n\n\n\n

    Test 1.3 – GE3 (98.1.2.19) is green while GE4 (184.1.2.27) is YELLOW due to latency<\/h4>\n\n\n\n

    In Test 1.1 and Test 1.2, the WAN link goes to YELLOW because of packet loss. In Test 1.3 and Test 1.4, will test the WAN link goes to YELLOW because of latency. In Test 1.3, \u201cGE4 \u2013 184.1.2.27\u201d is manipulated to become YELLOW by having a latency of 60ms. The follow table illustrate the link condition in the Test 1.3:<\/p>\n\n\n\n

    \"\"
    Figure 11<\/figcaption><\/figure>\n\n\n\n

    Before using the Client100 (192.168.200.100) to access wordpress05.lab.local, confirm the link status by \u201cdebug.py –dec\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 12<\/figcaption><\/figure>\n\n\n\n

    The debug output confirmed \u201cGE4 \u2013 184.1.2.27\u201d is yellow due to latency.<\/p>\n\n\n\n

    The following is the web server access log from wordpress05 in Test 1.3:<\/p>\n\n\n\n

    \"\"
    Figure 13<\/figcaption><\/figure>\n\n\n\n

    From the access log, Edge-1 picks \u201cGE3 \u2013 98.1.2.19\u201d for local Internet breakout. This means Edge-1 picked the GREEN \u201cGE3 \u2013 98.1.2.19\u201d over YELLOW \u201cGE4 \u2013 184.1.2.27\u201d, where the YELLOW is due to 60ms latency.<\/p>\n\n\n\n

    Test 1.4 – GE3 (98.1.2.19) is YELLOW due to latency while GE4 (184.1.2.27) is GREEN<\/h4>\n\n\n\n

    To ensure the Test 1.3 result is not because of other factors, will swap the two WAN links latency condition in Test 1.4. That is \u201cGE3 \u2013 98.1.2.19\u201d becomes yellow with 60ms, while \u201cGE4 \u2013 184.1.2.27\u201d is green. The follow table illustrate the link condition in the Test 1.4:<\/p>\n\n\n\n

    \"\"
    Figure 14<\/figcaption><\/figure>\n\n\n\n

    Before using the Client100 (192.168.200.100) to access wordpress05.lab.local, confirm the link status by \u201cdebug.py –dec\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 15<\/figcaption><\/figure>\n\n\n\n

    The debug output confirmed \u201cGE3 \u2013 98.1.2.19\u201d is yellow due to latency.<\/p>\n\n\n\n

    The following is the web server access log from wordpress05 in Test 1.4:<\/p>\n\n\n\n

    \"\"
    Figure 16<\/figcaption><\/figure>\n\n\n\n

    From the access log, Edge-1 picked \u201cGE4 \u2013 184.1.2.27\u201d for local Internet breakout. This means Edge-1 picked the GREEN \u201cGE4 \u2013 184.1.2.27\u201d over YELLOW \u201cGE3 \u2013 98.1.2.19\u201d, where the YELLOW is due to 60ms latency.<\/p>\n\n\n\n

    Conclusion of Test 1.1 to Test 1.4<\/h4>\n\n\n\n

    The conclusion is:<\/p>\n\n\n\n

    For internet Local breakout, the SD-WAN Edge picks the GREEN WAN link over YELLOW\/RED WAN link when the link steering is configured as Auto.<\/em><\/strong><\/p>\n\n\n\n

    NOTE: The test of having WAN link in RED is not repeated here. I have verified the condition of picking GREEN over RED but will not have the detail here to make the post too long.<\/p>\n\n\n\n\n\n\n\n

    When use \u201cAuto\u201d link steering, and every links are GREEN, which link will be picked?<\/h3>\n\n\n\n

    This section focuses on the situation when both Internet WAN links are GREEN with link steering configured as Auto.<\/p>\n\n\n\n

    Edge Configuration for Test 2.1 to Test 2.3<\/h4>\n\n\n\n

    Let\u2019s review the Edge-1 business policy configuration. The Edge-1 is configured with local Internet breakout, such that all the UDP traffic destinated to Internet will be marked as High Priority and Realtime Class, while all other traffic (TCP and ICMP) destinated to Internet will be marked as Normal Priority and Transactional Class. In addition, the link steering is configured to \u201cAuto\u201d. The follow screen capture is the business policy configuration:<\/p>\n\n\n\n

    \"\"
    Figure 17<\/figcaption><\/figure>\n\n\n\n

    Test 2.1 – GE3 (98.1.2.19) and GE4 (184.1.2.27) are both GREEN, and having minimal latency\/jitter\/packet loss, GE4 (184.1.2.27) configured with larger bandwidth<\/h4>\n\n\n\n

    In Test 2.1, both GE3 (98.1.2.19) and GE4 (184.1.2.27) are GREEN. No latency\/jitter\/packet loss are added to the links.<\/p>\n\n\n\n

    Let\u2019s check the WAN link condition under the \u201cMonitor –> Overview\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 18<\/figcaption><\/figure>\n\n\n\n

    From the Edge-1 overview page, both links is having 0ms latency, 0ms jitter and 0% packet loss. And \u201cGE4 \u2013 184.1.2.27\u201d is configured with 10Mpbs\/10Mbps, while \u201cGE3 \u2013 98.1.2.19\u201d is configured with 5Mbps\/5Mbps.<\/p>\n\n\n\n

    The following is the \u201cdebug.py –dec\u201d output to confirm both links are GREEN:<\/p>\n\n\n\n

    \"\"
    Figure 19<\/figcaption><\/figure>\n\n\n\n

    Now, from the Client100 (192.168.200.100) access the wordpress05.lab.local to see which link does the Edge-1 picked for local breakout. By checking the \u201cList Active Flows\u201d, verified it is taking the local Internet breakout as the route is \u201cDirect to Cloud\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 20<\/figcaption><\/figure>\n\n\n\n

    The following is the wordpress05 access log:<\/p>\n\n\n\n

    \"\"
    Figure 21<\/figcaption><\/figure>\n\n\n\n

    From the access log, \u201cGE4 \u2013 184.1.2.27\u201d is the WAN links picked by Edge-1 for local Internet breakout. Since both WAN links are having zero latency\/jitter\/loss, the only difference is the configured bandwidth. \u201cGE4 \u2013 184.1.2.27\u201d is with 10Mbps\/10Mbps while \u201cGE3 \u2013 98.1.2.19\u201d is with 5Mbps\/5Mpbs. With the test result, it seems when both WAN links are GREEN, the one with larger bandwidth is selected. The following test, Test 2.2, will make \u201cGE3 \u2013 98.1.2.19\u201d with larger bandwidth to confirm this is the behavior.<\/p>\n\n\n\n

    Test 2.2 – GE3 (98.1.2.19) and GE4 (184.1.2.27) are both GREEN, and having minimal latency\/jitter\/packet loss, GE3 (98.1.2.19) configured with larger bandwidth<\/h4>\n\n\n\n

    In order to confirm the link steering \u201cAuto\u201d picks the link with the largest bandwidth when all the links are GREEN, the \u201cGE3 — (98.1.2.19)\u201d configured bandwidth will be modified to 12Mbps\/12Mpbs. And the latency\/jitter\/packet loss are remains 0 for both WAN links:<\/p>\n\n\n\n

    \"\"
    Figure 22<\/figcaption><\/figure>\n\n\n\n

    The following is the \u201cdebug.py –dec\u201d output to confirm both links are GREEN<\/p>\n\n\n\n

    \"\"
    Figure 23<\/figcaption><\/figure>\n\n\n\n

    At this point, from the Client100 (192.168.200.100) access the wordpress05.lab.local to see which link does the Edge-1 picked for local breakout. By checking the \u201cList Active Flows\u201d, verified it is taking the local Internet breakout as the route is \u201cDirect to Cloud\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 24<\/figcaption><\/figure>\n\n\n\n

    The following is the wordpress05 access log:<\/p>\n\n\n\n

    \"\"
    Figure 25<\/figcaption><\/figure>\n\n\n\n

    From the access log, \u201cGE3 — (98.1.2.19)\u201d is the WAN link picked by Edge-1 for local Internet breakout. During Test 2.2, the latency\/jitter\/packet loss are zero for both links. And \u201cGE3 \u2013 98.1.2.19\u201d is 12Mbps\/12Mbps and \u201cGE4 \u2013 184.1.2.27\u201d is 10Mbps\/10Mbps. With the result of Test 2.1 and Test 2.2, the conclusion is when both Internet WAN links are GREEN, the SD-WAN Edge will select the link with the highest bandwidth.<\/p>\n\n\n\n

    *At this point, the 12Mbps\/12Mbps of \u201cGE3 \u2013 98.1.2.19\u201d is reverted to 5Mbps\/5Mpbs.<\/p>\n\n\n\n

    Test 2.3 – GE3 (98.1.2.19) and GE4 (184.1.2.27) are both GREEN, with GE4 with higher latency<\/h4>\n\n\n\n

    In Test 2.3, the purpose is to check when both links are GREEN, will the SD-WAN Edge compare the latency\/jitter\/packet loss within the GREEN range. Since \u201cGE4 \u2013 184.1.2.27\u201d is with 10Mbps\/10Mbps while \u201cGE3 \u2013 98.1.2.19\u201d is with 5Mbps\/5Mpbs, \u201cGE4 \u2013 184.1.2.27\u201d will be picked according to Test 2.1. In this Test 2.3, \u201cGE4 \u2013 184.1.2.27\u201d will get added latency of 22ms, that is the \u201cGE4 \u2013 184.1.2.27\u201d is still GREEN but the latency is 22ms which is higher than 0ms latency of \u201cGE3 \u2013 98.1.2.19\u201d. The following diagram shows the WAN link conditions in Test 2.3:<\/p>\n\n\n\n

    \"\"
    Figure 26<\/figcaption><\/figure>\n\n\n\n

    Let\u2019s take a look on the Edge-1 Monitor –> Overview page below:<\/p>\n\n\n\n

    \"\"
    Figure 27<\/figcaption><\/figure>\n\n\n\n

    The Edge-1 Overview page allows us to confirm the Internet WAN links\u2019 condition, especially pay attention to \u201cGE3 \u2013 98.1.2.19\u201d with 22ms (up), 21ms (down) latency and \u201cGE4 \u2013 184.1.2.27\u201d with 0ms latency.<\/p>\n\n\n\n

    The following is the \u201cdebug.py –dec\u201d output to confirm both links are GREEN. However, \u201cGE3 \u2013 98.1.2.19\u201d with 22ms (TX) and 21ms (RX) latency, while \u201cGE4 \u2013 184.1.2.27\u201d with 0ms latency:<\/p>\n\n\n\n

    \"\"
    Figure 28<\/figcaption><\/figure>\n\n\n\n

    At this point, from the Client100 (192.168.200.100) access the wordpress05.lab.local to see which link does the Edge-1 picked for local breakout. By checking the \u201cList Active Flows\u201d, verified it is taking the local Internet breakout as the route is \u201cDirect to Cloud\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 29<\/figcaption><\/figure>\n\n\n\n

    The following is the wordpress05 access log:<\/p>\n\n\n\n

    \"\"
    Figure 30<\/figcaption><\/figure>\n\n\n\n

    From the access log, \u201cGE4 \u2013 184.1.2.27\u201d is the WAN link picked by Edge-1 for local Internet breakout. The result shows that, when SD-WAN Edge select the WAN link, it firstly screen out the link(s) by the color (GREEN, YELLOW or RED), however, what is the latency\/jitter\/loss difference between two links with the same color is not a selection criteria. In this example, although \u201cGE4 \u2013 184.1.2.27\u201d is having 22ms\/21ms latency compare with \u201cGE3 \u2013 98.1.2.19\u201d is having 0ms latency, the 22ms is still fall into the GREEN range. Since both links are GREEN, the latency difference is not used for link selection. The next selection criteria is the bandwidth. As a result, \u201cGE4 \u2013 184.1.2.27\u201d is selected as it gets 10Mbps which is larger than 5Mbps of \u201cGE3 \u2013 98.1.2.19\u201d.<\/p>\n\n\n\n

    Conclusion of Test 2.1 to Test 2.3<\/h4>\n\n\n\n

    The conclusion is:<\/p>\n\n\n\n

    For internet Local breakout, with link steering Auto, when the SD-WAN Edge comes with multiple WAN links are GREEN, the WAN link with the highest bandwidth* is being selected. In addition, when picking the WAN link with the highest bandwidth, the SD-WAN Edge does not care if the WAN link is very GREEN (such as 0ms latency) or the WAN link is barely GREEN (such as 22ms latency). As long as the WAN link is GREEN, that link is entitled to be selected based on the bandwidth.<\/em><\/strong><\/p>\n\n\n\n

    *bandwidth is either the configured bandwidth or auto detected bandwidth.<\/em><\/strong><\/p>\n\n\n\n

    Note: upcoming Test 3.1 to Test 3.3 to retrieve more detail about what bandwidth is actually being considered for link selection.<\/em><\/strong><\/p>\n\n\n\n\n\n\n\n

    Is there any load balancing or link aggregation possible for local Internet breakout with multiple Internet WAN links?<\/h3>\n\n\n\n

    When the SD-WAN Edge connected with two Internet WAN links, one of the typical questions pop up is \u201cCan the SD-WAN Edge perform any sort of load balancing or aggregation between the two Internet WAN links when doing local Internet breakout?\u201d. Before going to the answer of this question, there are some technical background information worth to mention.<\/p>\n\n\n\n

    Firstly, when SD-WAN Edge is responsible for local Internet breakout, and if the SD-WAN Edge interface is assigned with public IP address, typically the SD-WAN Edge will perform source NAT (or precisely PAT). The NAT is configurable option called \u201cNAT Direct Traffic\u201d under the interface setting, the following screen capture is from GE3 of Edge-1, which \u201cNAT Direct Traffic\u201d is enabled:<\/p>\n\n\n\n

    \"\"
    Figure 31<\/figcaption><\/figure>\n\n\n\n

    The Edge-1 is having \u201cNAT Direct Traffic\u201d enabled for both \u201cGE3 \u2013 98.1.2.19\u201d and \u201cGE4 \u2013 184.1.2.27\u201d. As a result, when the client machines access Internet, the client machine\u2019s source IP address will get PAT to either 98.1.2.19 or 184.1.2.27 depends or whether GE3 or GE4 is selected. Although SD-WAN Edge is a \u201cPer-Packet\u201d device, the local Internet breakout situation is not applicable for \u201cPer-Packet\u201d load balancing across multiple links when NAT is enabled. The \u201cPer-Packet\u201d load balancing happens when both ends are SD-WAN device so traffic send\/receive over the WAN links are encapsulated in the SD-WAN overlay tunnel. When local Internet breakout with NAT, there is no overlay tunnel between the SD-WAN Edge and the destination. For example, when the client machine access web site https:\/\/www.vmware.com<\/a>, and say GE4 is selected for the corresponding traffic flow. That means the web server www.vmware.com will see TCP connection(s) from GE4 IP address 184.1.2.27. The SD-WAN Edge must keep using GE4 IP address 184.1.2.27 for existing flows to www.vmware.com<\/a>, it cannot switch the NATTed IP to GE3 98.1.2.19 in the middle of existing flows bound to GE4 IP address 184.1.2.27, that will break the flows and confuse the web server www.vmware.com<\/a>. As a result, it is not possible to have a single flow get load balanced across multiple links for local Internet breakout when NAT is enabled.<\/p>\n\n\n\n

    Further to the above, that means the only possible load balancing is per flow load balancing in the situation for local Internet breakout when NAT is enabled. In upcoming section and tests will discover is there any flow based load balancing will occur, if yes, under what conditions flow based load balancing will occur.<\/p>\n\n\n\n

    Edge Configuration for Test 3.1 to Test 3.x<\/h4>\n\n\n\n

    The business policy is still remaining the same as the previous test. That is the link steering is configured as auto. The follow diagram re-cap the business policy configuration:<\/p>\n\n\n\n

    \"\"
    Figure 32<\/figcaption><\/figure>\n\n\n\n

    During Test 3.1-3.3, there will not no extra latency\/jitter\/packet loss introduced to GE3 (98.1.2.19) and GE4 (184.1.2.27). The configured bandwidth of GE3 (98.1.2.19) is 5Mbps\/5Mbps, the configured bandwidth of GE4 (184.1.2.27) is 10Mbps\/10Mbps. The following figure summarize these two Internet WAN links status:<\/p>\n\n\n\n

    \"\"
    Figure 33<\/figcaption><\/figure>\n\n\n\n

    The following figure is the Edge-1 Overview reflecting the Internet WAN links status:<\/p>\n\n\n\n

    \"\"
    Figure 34<\/figcaption><\/figure>\n\n\n\n

    Test 3.1 – GE3 (98.1.2.19) and GE4 (184.1.2.27) are both GREEN, when higher bandwidth link upstream is utilized<\/h4>\n\n\n\n

    In this Test 3.1, the focus is on the WAN link upstream utilization. The primary objective of Test 3.1 is to check if the higher bandwidth link GE4 (184.1.2.27) is fully utilized on upstream direction, will the Edge-1 perform some sort of load balancing, such as let any new flow to utilize the other GREEN link with lower bandwidth, that is GE3 (98.1.2.19).<\/p>\n\n\n\n

    In this Test 3.1, wordpress05 (43.254.254.14) and wordpress06 (24.12.0.14) are having iperf3 (https:\/\/github.com\/esnet\/iperf) running as server mode on port 5201. The iperf3 will be used to let the particular WAN link fully utilized.<\/p>\n\n\n\n

    To start with, the Client100 (192.168.200.100) will acting as an iperf3 client by injecting upload traffic destinated to wordpress05 (43.254.254.14) for 600s with this command \u201c\/usr\/local\/bin\/iperf3 -c 43.254.254.14 -t 600\u201d<\/p>\n\n\n\n

    Client100 start iperf3 for upstream traffic generation:<\/p>\n\n\n\n

    \"\"
    Figure 35<\/figcaption><\/figure>\n\n\n\n

    In wordpress05 (43.254.254.14), iperf3 server starts to receive the traffic. From the screen capture below, it shows the connection is from 184.1.2.27:20002, that means GE4 (184.1.2.27) is selected by the Edge-1 for this iperf3 session:<\/p>\n\n\n\n

    \"\"
    Figure 36<\/figcaption><\/figure>\n\n\n\n

    It is expected to select GE4 (184.1.2.27) for this flow, from section 2, we understand that when both WAN links are GREEN, the one with higher bandwidth is selected.<\/p>\n\n\n\n

    The transport live monitoring of Edge-1 shows the GE4 (184.1.2.27) WAN link upstream side is fully utilized:<\/p>\n\n\n\n

    \"\"
    Figure 37<\/figcaption><\/figure>\n\n\n\n

    While the iperf3 to wordpress05 (43.254.254.14) is still running, start another iperf3 to generate upstream traffic to wordpress06 (24.12.0.14) from Client100:<\/p>\n\n\n\n

    \"\"
    Figure 38<\/figcaption><\/figure>\n\n\n\n

    In the iperf3 server side, the screen capture below shows the flow is coming from 184.1.2.27:20002, this means Edge-1 select GE4 (184.1.2.27) as the WAN link to reach wordpress06 (24.12.0.14) iperf3 server:<\/p>\n\n\n\n

    \"\"
    Figure 39<\/figcaption><\/figure>\n\n\n\n

    The transport live monitoring of Edge-1 shows only GE4 (184.1.2.27) is being used, GE3 (98.1.2.19) is not being used:<\/p>\n\n\n\n

    \"\"
    Figure 40<\/figcaption><\/figure>\n\n\n\n

    To further confirm, while the iperf3 is still running, open a browser at Client100 (192.168.200.100) and access the web service at wordpress05 (43.254.254.14). The follow is the access log at wordpress05 (43.254.254.14):<\/p>\n\n\n\n

    \"\"
    Figure 41<\/figcaption><\/figure>\n\n\n\n

    From the web access log, the traffic is from 184.1.2.27, that mean Edge-1 selected GE4 (184.1.2.27) as the WAN link for the web access flow from Client100 (192.168.200.100), during the time GE4 (184.1.2.27) upstream bandwidth is fully utilized.<\/p>\n\n\n\n

    In this Test 3.1, the result shows the upstream bandwidth fully utilized does not trigger any new flow to utilize the other Internet WAN link. That means when both Internet WAN links are GREEN, and then the larger bandwidth link upstream is fully utilized, the SD-WAN Edge continue to select this larger bandwidth link for new flows going Internet local breakout.<\/p>\n\n\n\n

    Test 3.2 – GE3 (98.1.2.19) and GE4 (184.1.2.27) are both GREEN, when higher bandwidth link downstream is utilized<\/h4>\n\n\n\n

    Test 3.2 is similar to Test 3.1 but the focus will be on the WAN link downstream utilization instead of upstream. In this Test 3.2, wordpress05 (43.254.254.14) and wordpress06 (24.12.0.14) are having iperf3 (https:\/\/github.com\/esnet\/iperf) running as server mode on port 5201. The iperf3 will be used to let the WAN link fully utilized.<\/p>\n\n\n\n

    To start with, the Client100 (192.168.200.100) will acting as an iperf3 client by injecting download traffic destinated to wordpress05 (43.254.254.14) for 600s with this command \u201c\/usr\/local\/bin\/iperf3 -c 43.254.254.14 -R -t 600\u201d. The \u201c-R\u201d parameter means reverse which will make the traffic going downstream towards the client.<\/p>\n\n\n\n

    \"\"
    Figure 42<\/figcaption><\/figure>\n\n\n\n

    At the iperf3 server side, the screen capture shows the connection is from 184.1.2.27:20002:<\/p>\n\n\n\n

    \"\"
    Figure 43<\/figcaption><\/figure>\n\n\n\n

    Edge-1 selected GE4 (184.1.2.27) for this iperf3 flow from Client100 (192.168.200.100) to wordpress05 (43.254.254.14). Selecting GE4 (184.1.2.27) is expected because both WAN links are GREEN and GE4 (184.1.2.27) is having larger bandwidth (10Mbps > 5 Mbps).<\/p>\n\n\n\n

    The transport live monitoring of Edge-1 shows the iperf traffic is downstream and using GE4 (184.1.2.27):<\/p>\n\n\n\n

    \"\"
    Figure 44<\/figcaption><\/figure>\n\n\n\n

    To check the downstream utilization will affect how the Edge-1 select which WAN link for new flow for local Internet breakout, at this point, that is the iperf3 between Client100 (192.168.200.100) and wordpress05 (43.254.254.14) are still running and generating downstream traffic for GE4 (184.1.2.27), Client100 (192.168.200.100) start a new iperf3 to wordpress06 (24.12.0.14) by \u201c\/usr\/local\/bin\/iperf3 -c 24.12.0.14 -R -t 180\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 45<\/figcaption><\/figure>\n\n\n\n

    On the iperf3 server, wordpress06 (24.12.0.14), the screen capture below shows the connection is from 98.1.2.19:20002:<\/p>\n\n\n\n

    \"\"
    Figure 46<\/figcaption><\/figure>\n\n\n\n

    This result is different from the upstream test in Test 3.1, as Edge-1 picked GE3 (98.1.2.19) for the new iperf3 flow when the GE4 (184.1.2.27) downstream bandwidth is utilized.<\/p>\n\n\n\n

    Let\u2019s check the transport monitoring page to confirm:<\/p>\n\n\n\n

    \"\"
    Figure 47<\/figcaption><\/figure>\n\n\n\n

    At the transport live monitoring page, now both GE4 (184.1.2.27) and GE3 (98.1.2.19) are fully utilized. This result demonstrated, when the larger bandwidth link downstream bandwidth is fully utilized, the SD-WAN will select the remaining GREEN WAN link for new traffic flow that is local Internet breakout. To further confirm, an additional test is conducted here. The additional test is when iperf3 between Client100 (192.168.200.100) and wordpress05 (43.254.254.14) is running, such that GE4 (184.1.2.27) downstream is fully utilized. But there is no traffic for GE3 (98.1.2.19), that is iperf3 between Client100 (192.168.200.100) and wordpress06 (24.12.0.14) is not running. With this situation, Client100 use a web browser to access the wordpress05 web service, the following is the web access log of wordpress05:<\/p>\n\n\n\n

    \"\"
    Figure 48<\/figcaption><\/figure>\n\n\n\n

    From the wordpress05 web access log, the Edge-1 selected GE3 (98.1.2.19) for the new log from Client100 (192.168.200.100) access wordpress05 web service. The conclusion of Test 3.2 is, when both Internet WAN links are GREEN, and the WAN link with larger bandwidth is fully utilized in the downstream direction, SD-WAN Edge will select the remaining GREEN WAN link for local Internet breakout for newly created flow.<\/p>\n\n\n\n

    Test 3.3 – GE3 (98.1.2.19) and GE4 (184.1.2.27) are both GREEN, the higher bandwidth link downstream is partly utilized<\/h4>\n\n\n\n

    From Test 3.1 and Test 3.2, for local Internet breakout, to have the SD-WAN Edge selects the other GREEN WAN link (that is not the one with the largest bandwidth) for flow based load balancing, the larger bandwidth link needs to have downstream traffic which consuming the downstream bandwidth. Test 3.3 will continue to test when the WAN link are downstream utilized but specifically not make the downstream bandwidth fully consumed.<\/p>\n\n\n\n

    The objective of Test 3.3 is to confirm if the SD-WAN Edge picks the WAN link with the largest \u201cremaining downstream bandwidth\u201d<\/em>. Let\u2019s take some examples to explain the meaning of \u201cremaining downstream bandwidth\u201d<\/em> and also the expected WAN link selection.<\/p>\n\n\n\n

    Test 3.3.1 – GE3 (98.1.2.19) and GE4 (184.1.2.27) are both GREEN, GE4 (184.1.2.27) configured with 10Mbps and consumed 3.2Mbps, GE3 (98.1.2.19) configured with 5Mbps with 0Mbps consumed<\/h5>\n\n\n\n

    In Test 3.3.1, all bandwidth mentioned are downstream bandwidth unless specified as upstream.<\/p>\n\n\n\n

    The formula of calculating the \u201cremaining downstream bandwidth\u201d is as follow:<\/p>\n\n\n\n

    remaining downstream bandwidth = configure\/discovered downstream bandwidth –\u00a0 consumed downstream bandwidth<\/strong><\/p>\n\n\n\n

    In Test 3.3.1:<\/p>\n\n\n\n

    GE4 (184.1.2.27) configured downstream bandwidth = 10Mbps<\/p>\n\n\n\n

    GE4 (184.1.2.27) consumed downstream bandwidth = 3.2Mbps<\/p>\n\n\n\n

    GE4 (184.1.2.27) remaining downstream bandwidth = GE4 (184.1.2.27) configured downstream bandwidth – GE4 (184.1.2.27) consumed downstream bandwidth<\/p>\n\n\n\n

    GE4 (184.1.2.27) remaining downstream bandwidth = 10Mbps – 3.2Mpbs = 6.8Mbps<\/p>\n\n\n\n

    GE3 (98.1.2.19) configured downstream bandwidth = 5Mbps<\/p>\n\n\n\n

    GE3 (98.1.2.19) consumed downstream bandwidth = 0Mbps<\/p>\n\n\n\n

    GE3 (98.1.2.19) remaining downstream bandwidth = GE3 (98.1.2.19) configured downstream bandwidth – GE3 (98.1.2.19) consumed downstream bandwidth<\/p>\n\n\n\n

    GE3 (98.1.2.19) remaining downstream bandwidth = 5bps – 0Mpbs = 5Mbps<\/p>\n\n\n\n

    Since 6.8Mbps > 5Mbps, that is GE4 (184.1.2.27) remaining downstream bandwidth > GE3 (98.1.2.19) remaining downstream bandwidth, the expectation is under this situation, Edge-1 will select GE4 (184.1.2.27) for new flow which is Internet local breakout<\/p>\n\n\n\n

    To run the test to check the expectation is correct or not, Client100 (192.168.200.100) starts an iperf3 flow with destination to wordpress05 (43.254.254.14), the iperf3 command is \u201c\/usr\/local\/bin\/iperf3 -c 43.254.254.14 -R -b 3M -t 600\u201d. The \u201c-b 3M\u201d is to let iperf3 limit the download speed at around 3Mbps.<\/p>\n\n\n\n

    \"\"
    Figure 49<\/figcaption><\/figure>\n\n\n\n

    The iperf3 server screen capture shows the connection is from 184.1.2.27:20002:<\/p>\n\n\n\n

    \"\"
    Figure 50<\/figcaption><\/figure>\n\n\n\n

    This means Edge-1 selected GE4 (184.1.2.27) as the WAN link for this iperf3 flow, this is expected as there is no traffic on both links when the flow start. The transport live monitoring confirmed the iperf3 is consuming about 3.26Mbps downstream bandwidth for the GE4 (184.1.2.27):<\/p>\n\n\n\n

    \"\"
    Figure 51<\/figcaption><\/figure>\n\n\n\n

    While the iperf3 is running, that is 3.26Mbps downstream is consumed in GE4 (184.1.2.27), at Client100 (192.168.200.100) open a browser to access the web service at wordpress05 (43.254.254.14). Here is the web service access log:<\/p>\n\n\n\n

    \"\"
    Figure 52<\/figcaption><\/figure>\n\n\n\n

    From the wordpress05 (43.254.254.14) access log, Edge-1 selected GE4 (184.1.2.27) for the newly created web access flow towards wordpress05 (43.254.254.14). This result match the expectation because GE4 (184.1.2.27) remaining downstream bandwidth is 10Mbps – 3.26Mbps = 6.74Mbps, while is larger than 5Mbps of GE3 (98.1.2.19).<\/p>\n\n\n\n

    Test 3.3.2 – GE3 (98.1.2.19) and GE4 (184.1.2.27) are both GREEN, GE4 (184.1.2.27) configured with 10Mbps and consumed 6.4Mbps, GE3 (98.1.2.19) configured with 5Mbps with 0Mbps consumed<\/h5>\n\n\n\n

    The difference of Test 3.3.2 compare with Test 3.3.1 is the GE4 (184.1.2.27) will get consumed more downstream bandwidth. In Test 3.3.2:<\/p>\n\n\n\n

    GE4 (184.1.2.27) configured downstream bandwidth = 10Mbps<\/p>\n\n\n\n

    GE4 (184.1.2.27) consumed downstream bandwidth = 6.4Mbps<\/p>\n\n\n\n

    GE4 (184.1.2.27) remaining downstream bandwidth = GE4 (184.1.2.27) configured downstream bandwidth – GE4 (184.1.2.27) consumed downstream bandwidth<\/p>\n\n\n\n

    GE4 (184.1.2.27) remaining downstream bandwidth = 10Mbps – 6.4Mpbs = 3.6Mbps<\/p>\n\n\n\n

    GE3 (98.1.2.19) configured downstream bandwidth = 5Mbps<\/p>\n\n\n\n

    GE3 (98.1.2.19) consumed downstream bandwidth = 0Mbps<\/p>\n\n\n\n

    GE3 (98.1.2.19) remaining downstream bandwidth = GE3 (98.1.2.19) configured downstream bandwidth – GE3 (98.1.2.19) consumed downstream bandwidth<\/p>\n\n\n\n

    GE3 (98.1.2.19) remaining downstream bandwidth = 5bps – 0Mpbs = 5Mbps<\/p>\n\n\n\n

    Since 5Mbps > 3.6Mbps, that is GE3 (98.1.2.19) remaining downstream bandwidth > GE4 (184.1.2.27) remaining downstream bandwidth, the expectation is under this situation, Edge-1 will select GE3 (98.1.2.19)  for new flow which is Internet local breakout<\/p>\n\n\n\n

    To run the test to check the expectation is correct or not, Client100 (192.168.200.100) starts an iperf3 flow with destination to wordpress05 (43.254.254.14), the iperf3 command is \u201c\/usr\/local\/bin\/iperf3 -c 43.254.254.14 -R -b 6M -t 600\u201d. The \u201c-b 6M\u201d is to let iperf3 limit the download speed at around 6Mbps:<\/p>\n\n\n\n

    \"\"
    Figure 53<\/figcaption><\/figure>\n\n\n\n

    The following is the iperf3 screen capture:<\/p>\n\n\n\n

    \"\"
    Figure 54<\/figcaption><\/figure>\n\n\n\n

    From the iperf3 server screen capture, the connection is from 184.1.2.27:20002, that is GE4 (184.1.2.27). This is expected because when the iperf3 flow starts, there is no traffic at both GE3 (98.1.2.19) and GE4 (184.1.2.27). GE4 (184.1.2.27) is selected because it has the larger remaining downstream bandwidth. The transport live monitoring below shows, with the iperf3 running, GE4 (184.1.2.27) gets consumed about 6.4Mbps bandwidth:<\/p>\n\n\n\n

    \"\"
    Figure 55<\/figcaption><\/figure>\n\n\n\n

    When the iperf3 is running, now GE3 (98.1.2.19) is having a larger remaining downstream bandwidth because 5Mbps > 3.6Mbps (10-6.4=3.6). While the iperf3 is running, in Client100 (192.168.200.100), open a web browser and access the web service of wordpress05 (43.254.254.14). The following is the web service access log from wordpress05:<\/p>\n\n\n\n

    \"\"
    Figure 56<\/figcaption><\/figure>\n\n\n\n

    From the wordpress05 web access log, the request comes from 98.1.2.19 while means Edge-1 selected GE3 (98.1.2.19) as the Internet WAN link for new local Internet breakout flow. This confirmed the expectation is correct because when the web access flow is created, the iperf3 is still running. The iperf3 consumed about 6.4Mbps bandwidth of GE4 (184.1.2.27), this makes GE3 (98.1.2.19) have a larger remaining downstream bandwidth, so the Edge-1 selects GE3 (98.1.2.19) for new local Internet breakout flow.<\/p>\n\n\n\n

    Conclusion of Test 3.1 to Test 3.3<\/h5>\n\n\n\n

    The conclusion is:<\/p>\n\n\n\n

    For new flow that is local Internet breakout, with link steering Auto, when the SD-WAN Edge comes with multiple WAN links are GREEN, the WAN link with the highest remaining downstream bandwidth is being selected. The \u201cremaining downstream bandwidth\u201d is calculated by:<\/em><\/strong><\/p>\n\n\n\n

    Remaining downstream bandwidth = configure\/discovered downstream bandwidth – consumed downstream bandwidth<\/span><\/strong><\/p>\n\n\n\n

    Configured\/discovered upstream bandwidth and consumed upstream bandwidth is not considered when select the WAN link for new flow that is local Internet breakout.<\/em><\/strong><\/p>\n\n\n\n

    In addition, the above conclusion applies to traffic in any service classes (Real Time, Transactional, Bulk).<\/em><\/strong><\/p>\n\n\n\n\n\n\n\n

    What happen for link steering \u201cPreferred\u201d, \u201cAvailable\u201d and \u201cMandatory\u201d options?<\/h3>\n\n\n\n

    At this point, we understand the SD-WAN Edge selects the Internet WAN link with the highest remaining downstream bandwidth for new flow that needs local Internet breakout, with link steering Auto. In Test 4.1 to Test 4.3, the objective is to see how different link steering setting will behave. During Test 4.1-4.3, there will be no extra latency\/jitter\/packet loss introduced to GE3 (98.1.2.19) and GE4 (184.1.2.27). The configured bandwidth of GE3 (98.1.2.19) is 5Mbps\/5Mbps, the configured bandwidth of GE4 (184.1.2.27) is 10Mbps\/10Mbps. The following figure summarize these two Internet WAN links status:<\/p>\n\n\n\n

    \"\"
    Figure 57<\/figcaption><\/figure>\n\n\n\n

    The following is the overview monitoring page of Edge-1 showing the WAN link status:<\/p>\n\n\n\n

    \"\"
    Figure 58<\/figcaption><\/figure>\n\n\n\n
    Test 4.1 – GE3 (98.1.2.19, 5Mbps) and GE4 (184.1.2.27, 10Mbps) are both GREEN, link steering prefers GE3 (98.1.2.19, 5Mbps)<\/h5>\n\n\n\n

    Since both WAN links are GREEN, without any traffic, GE4 (184.1.2.27) will be selected as it has a higher bandwidth than GE3 (98.1.2.19). As a result, business policies of prefer GE3 (98.1.2.19) are configured as follow to test if the business policies are effective or not:<\/p>\n\n\n\n

    \"\"
    Figure 59<\/figcaption><\/figure>\n\n\n\n

    Test 4.1 starts with iperf3 from Client100 (192.168.200.100) starting an iperf3 to wordpress05 (43.254.254.14) to generate downstream traffic. The iperf3 command is \u201c\/usr\/local\/bin\/iperf3 -c 43.254.254.14 -R -t 600\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 60<\/figcaption><\/figure>\n\n\n\n

    The following is the screen capture of the output from the iperf3 server side:<\/p>\n\n\n\n

    \"\"
    Figure 61<\/figcaption><\/figure>\n\n\n\n

    The iperf3 server shows the connections is from 98.1.2.19:20002, that is GE3 (98.1.2.19). This means the business policy \u201cPrefer-GE3-Other\u201d is taking effect to make the Edge-1 selects GE3 (98.1.2.19) without considering the remaining downstream bandwidth. Let\u2019s check the transport live monitoring page:<\/p>\n\n\n\n

    \"\"
    Figure 62<\/figcaption><\/figure>\n\n\n\n

    The transport live monitoring confirms the iperf3 is running on GE3 (98.1.2.19) and make the GE3 (98.1.2.19) fully utilized in the downstream direction. While the iperf3 is running, in Client100 (192.168.200.100), open a web browser to access the web service at wordpress05 (43.254.254.14). The following is the web service access log:<\/p>\n\n\n\n

    \"\"
    Figure 63<\/figcaption><\/figure>\n\n\n\n

    From the wordpress05 web access log, the request is coming from IP address 98.1.2.19, this means when GE3 (98.1.2.19) fully utilized in the downstream direction, Edge-1 still selects GE3 (98.1.2.19) for new flow matching business policy Prefer-GE3-Other to honor the Preferred link steering setting.<\/p>\n\n\n\n

    The following the output of List Active Flows to wordpress05 (43.254.254.14) from Edge-1 for reference:<\/p>\n\n\n\n

    \"\"
    Figure 64<\/figcaption><\/figure>\n\n\n\n

    The list active flows output confirms both web access and iperf3 traffic match the Prefer-GE3-Other business policy, and the Link Policy is Preferred.<\/p>\n\n\n\n

    With the tests down so far, link steering \u201cPreferred\u201d will make the SD-WAN Edge selects the preferred WAN link for new local Internet breakout flow, without considering the remaining downstream bandwidth.<\/p>\n\n\n\n

    Since the link steering is \u201cPreferred\u201d, the WAN link color (GREEN\/YELLOW\/RED) should still be honored for link selection. Let\u2019s perform a test such that packet loss is introduced to GE3 (98.1.2.19) such that the GE3 (98.1.2.19) will be RED for video\/voice, YELLOW for transactional. The following is the \u201cdebug.py –dec\u201d to verify the WAN link\u2019s color:<\/p>\n\n\n\n

    \"\"
    Figure 65<\/figcaption><\/figure>\n\n\n\n

    With the GE3 (98.1.2.19) is RED for video\/voice, YELLOW for transactional. At the Client100 (192.168.200.100), open a web browser to access the web service at wordpress05 (43.254.254.14). The following is the wordpress05 access log:<\/p>\n\n\n\n

    \"\"
    Figure 66<\/figcaption><\/figure>\n\n\n\n

    From the wordpress05 (43.254.254.14) web access log, the request is from 184.1.2.27, that is Edge-1 selected GE4 (184.1.2.27) for the web request flow when GE3 (98.1.2.19) having packet loss such that it is RED for video\/voice, YELLOW for transactional.<\/p>\n\n\n\n

    Packet loss is removed after this test to let both WAN links having 0% packet loss.<\/p>\n\n\n\n

    NOTE: The primary objective of Test 4.1 is to check if the link steering \u201cPreferred\u201d will prefer the preferred WAN link even if the preferred WAN link remaining downstream bandwidth is not the highest (that is SD-WAN Edge does not consider remaining downstream bandwidth). However, reader might wonder in local Internet breakout with link steering Preferred, does the SD-WAN Edge consider the WAN link quality SLA? Thus, here provided an example of introducing packet loss to demonstrate, if the preferred WAN link cannot meet the SLA (which is packet loss in this example), SD-WAN Edge does select other available GREEN link for local Internet breakout. However, Test 4.1 is not intended to study in detail every scenario about \u201cWAN link quality SLA is not met\u201d. (If time allow, I will create another post on this topic.)<\/p>\n\n\n\n

    Test 4.2 – GE3 (98.1.2.19, 5Mbps) and GE4 (184.1.2.27, 10Mbps) are both GREEN, link steering available for GE3 (98.1.2.19, 5Mbps)<\/h5>\n\n\n\n

    With link steering setting as \u201cAvailable\u201d for a particular WAN link, the expectation is that \u201cAvailable\u201d WAN link is always used unless that WAN link goes down. In Test 4.2, the business policy is adjusted to have link steering \u201cAvailable\u201d for WAN link GE3 (98.1.2.19, 5Mbps). Refer to the following screen capture for the business policy configuration in Test 4.2:<\/p>\n\n\n\n

    \"\"
    Figure 67<\/figcaption><\/figure>\n\n\n\n

    In Test 4.2, there is business policy \u201cAvailable-GE3-UDP\u201d which will catch UDP traffic going to Internet, with link steering setting as Available GE3 (98.1.2.19). Business policy \u201cAvailable-GE3-Other\u201d will catch any Non-UDP traffic going to Internet, with link steering setting as Available GE3 (98.1.2.19).<\/p>\n\n\n\n

    Test 4.2 starts with Client100 (192.168.200.100) start an iperf3 to wordpress05 (43.254.254.14), this iperf3 is responsible to generate downstream traffic. The command of the iperf3 is \u201c\/usr\/local\/bin\/iperf3 -c 43.254.254.14 -R -t 600\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 68<\/figcaption><\/figure>\n\n\n\n

    Let\u2019s check the wordpress05 (43.254.254.14) iperf3 server status:<\/p>\n\n\n\n

    \"\"
    Figure 69<\/figcaption><\/figure>\n\n\n\n

    From the iperf3 server output, the connection is from 98.1.2.19:20004, that means Edge-1 selected GE3 (98.1.2.19) for the iperf3 flow from Client100 (192.168.200.100). This match the expectation as the link steering setting is now \u201cAvailable\u201d GE3 (98.1.2.19), all new flows should be using GE3 (98.1.2.19) unless GE3 (98.1.2.19) is down.<\/p>\n\n\n\n

    Let\u2019s check the transport live monitoring page:<\/p>\n\n\n\n

    \"\"
    Figure 70<\/figcaption><\/figure>\n\n\n\n

    From the live monitoring, GE3 (98.1.2.19) downstream is fully utilized by the iperf3 traffic. With the iperf3 still running, at Client100 (192.168.200.100), open a web browser to access the web service at wordpress05 (43.254.254.14). The following is the web access log at wordpress05:<\/p>\n\n\n\n

    \"\"
    Figure 71<\/figcaption><\/figure>\n\n\n\n

    From the web access log, the request is coming from IP address 98.1.2.19, that is Edge-1 selected GE3 (98.1.2.19) for the web request flow. Since the GE3 (98.1.2.19) downstream bandwidth is consumed by the iperf3, this confirmed the link steering \u201cAvailable\u201d configuration will make the SD-WAN Edge always use the \u201cAvailable\u201d GE3 (98.1.2.19) link, no matter the GE3 (98.1.2.19) downstream bandwidth is consumed or not. The following is the list active flows output to confirm the flow is hitting the expected business policy, which is \u201cAvailable-GE3-Other\u201d. Also the Link Policy is Available:<\/p>\n\n\n\n

    \"\"
    Figure 72<\/figcaption><\/figure>\n\n\n\n

    The last test in Test 4.2 is to disconnect GE3 (98.1.2.19). When GE3 (98.1.2.19) is disconnected, that is GE3 (98.1.2.19) is down, the expectation is any new flow should switch over to use other available WAN link, in this case is GE4 (184.1.2.27). Let\u2019s disconnect GE3 (98.1.2.19), the following is the Edge-1 overview page after disconnected GE3:<\/p>\n\n\n\n

    \"\"
    Figure 73<\/figcaption><\/figure>\n\n\n\n

    The GE3 (98.1.2.19) link status is red. At this condition, open a web browser at Client100 (192.168.200.100) and access web service at wordpress05 (43.254.254.14). Client100 (192.168.200.100)\u00a0 is able to access the web service and the following is the web access log from wordpress05:<\/p>\n\n\n\n

    \"\"
    Figure 74<\/figcaption><\/figure>\n\n\n\n

    From wordpress05 web access log, the request is coming from IP address 184.1.2.27. This means Edge-1 selected GE4 (184.1.2.27) for new flow when GE3 (98.1.2.19) is link. This matches the expectation because link steering \u201cAvailable\u201d meaning using the configured \u201cAvailable\u201d link unless that particular link is down. Now, since the configured \u201cAvailable\u201d link GE3 is down, Edge-1 will pick other WAN links (GE4 in this example) available for new flow.<\/p>\n\n\n\n

    *GE3 (98.1.2.19) is re-connected after this test, so in the upcoming Test 4.3, GE3 (98.1.2.19) is started with connected status.<\/p>\n\n\n\n

    Test 4.3 – GE3 (98.1.2.19, 5Mbps) and GE4 (184.1.2.27, 10Mbps) are both GREEN, link steering mandatory for GE3 (98.1.2.19, 5Mbps)<\/h5>\n\n\n\n

    With link steering setting as \u201cMandatory\u201d for a particular WAN link, the expectation is that \u201cMandatory\u201d WAN link is always used. In Test 4.3, the business policy is adjusted to have link steering \u201cMandatory\u201d for WAN link GE3 (98.1.2.19, 5Mbps). Refer to the following screen capture for the business policy configuration in Test 4.3:<\/p>\n\n\n\n

    \"\"
    Figure 75<\/figcaption><\/figure>\n\n\n\n

    In Test 4.3, there is business policy \u201cMandatory-GE3-UDP\u201d which will catch UDP traffic going to Internet, with link steering setting as Mandatory GE3 (98.1.2.19). Business policy \u201cMandatory-GE3-Other\u201d will catch any Non-UDP traffic going to Internet, with link steering setting as Mandatory GE3 (98.1.2.19).<\/p>\n\n\n\n

    Test 4.3 starts with Client100 (192.168.200.100) start an iperf3 to wordpress05 (43.254.254.14), this iperf3 is responsible to generate downstream traffic. The command of the iperf3 is \u201c\/usr\/local\/bin\/iperf3 -c 43.254.254.14 -R -t 600\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 76<\/figcaption><\/figure>\n\n\n\n

    The following screen capture shows the iperf3 server side at wordpress05 (43.254.254.14):<\/p>\n\n\n\n

    \"\"
    Figure 77<\/figcaption><\/figure>\n\n\n\n

    From the iperf3 server output, the connection is from 98.1.2.19:20002. This means Edge-1 selected GE3 (98.1.2.19) for this iperf3 flow. This is matching the expectation because with mandatory GE3 (98.1.2.19) business policy will let every traffic flow using GE3 (98.1.2.19). The following is the transport live monitoring page for inspecting the WAN links average throughput:<\/p>\n\n\n\n

    \"\"
    Figure 78<\/figcaption><\/figure>\n\n\n\n

    The live monitoring shows the downstream bandwidth of GE3 (98.1.2.19) is being fully utilized by the iperf3 traffic. While the iperf3 is running, in Client100 (192.168.200.100), open a browser to access the web service at wordpress05 (43.254.254.14). The following is the web access log from wordpress05:<\/p>\n\n\n\n

    \"\"
    Figure 79<\/figcaption><\/figure>\n\n\n\n

    The wordpress05 (43.254.254.14) web access log shows the request is from IP address 98.1.2.19. This means Edge-1 selected GE3 (98.1.2.19) for the web access flows. This match the expectation as the business policy is configured to be mandatory using GE3 (98.1.2.19). The following is the output of live active flows:<\/p>\n\n\n\n

    \"\"
    Figure 80<\/figcaption><\/figure>\n\n\n\n

    The output of list active flows confirmed the iperf3 and web access flow is matching business policy \u201cMandatory-GE3-Other\u201d with the effective Link Policy Mandatory.<\/p>\n\n\n\n

    With a link steering Mandatory at GE3 (98.1.2.19), in the situation of GE3 (98.1.2.19) went down, the Internet access traffic is expected to be dropped. There should be no failover to other WAN links for link steering Mandatory. What link steering Mandatory means is the SD-WAN Edge must use that mandatory WAN link, if that particular WAN link is down, the traffic will be dropped. Let\u2019s disconnect GE3 (98.1.2.19):<\/p>\n\n\n\n

    \"\"
    Figure 81<\/figcaption><\/figure>\n\n\n\n

    With GE3 (98.1.2.19) disconnected, the overview page above shows the link status is red for GE3. Edge-1 now only having GE4 (184.1.2.27) is up. At this condition, open a browser in Client100 (192.168.200.100) and access the web service at wordpress05 (43.254.254.14). The following is the screen capture of the browser output of Client100 (192.168.200.100):<\/p>\n\n\n\n

    \"\"
    Figure 82<\/figcaption><\/figure>\n\n\n\n

    The browser ends up with \u201cThe connection has timed out\u201d, that means Client100 (192.168.200.100) is not able to access web service at wordpress05 (43.254.254.14). This confirmed when the configured mandatory WAN link is down (that is GE3 is down in this case), the traffic matching the business policy is dropped.<\/p>\n\n\n\n

    *GE3 (98.1.2.19) is re-connected after this test, so in the upcoming Test 4.3, GE3 (98.1.2.19) is started with connected status.<\/p>\n\n\n\n

    Conclusion of Test 4.1 to Test 4.3<\/h5>\n\n\n\n

    The following are the conclusion of Test 4.1 to Test 4.3.<\/p>\n\n\n\n

    For new flow that is local Internet breakout:<\/em><\/strong><\/p>\n\n\n\n

    Link Steering is Preferred<\/strong>: When the link steering is \u201cPreferred\u201d for a particular WAN link, say GE3 in this example. SD-WAN Edge will select the configured preferred WAN link, regardless what is that WAN link downstream bandwidth utilization. However, the configured preferred WAN link still has to meet the link quality (packet loss, jitter, latency) SLA. If the link quality SLA is not met and there is other GREEN link available, the SD-WAN Edge will select other GREEN link for new flow.<\/p>\n\n\n\n

    Link Steering is Available<\/strong>: When the link steering is \u201cAvailable\u201d for a particular WAN link, say GE3 in this example. SD-WAN Edge will select the configured available WAN link, regardless what is that WAN link downstream bandwidth utilization and what is the link quality (packet loss, jitter, latency). In the other words, as long as the configured available WAN link is up, this particular WAN link will be selected. When the configured available WAN link is down and there is other WAN link is up, new traffic flow will use the other WAN link which is up. For example, if GE3 is the configured available WAN link, and when GE3 goes down while GE4 is up, new traffic flow will be using GE4.<\/p>\n\n\n\n

    Link Steering is Mandatory<\/strong>: When the link steering is \u201cMandatory\u201d for a particular WAN link, say GE3 in this example, SD-WAN Edge will always select the configured mandatory WAN link for traffic flow. In the situation the configured mandatory WAN link is down, the traffic flow will get dropped. There is no failover to other WAN link when the link steering is Mandatory.<\/p>\n\n\n\n\n\n\n\n

    Is there any source IP persistence with \u201cAuto\u201d link steering?<\/h3>\n\n\n\n

    When the SD-WAN Edge is having multiple Internet WAN links, and performing local Internet breakout, there is a common question: \u201cIs there any type of client side\/source IP persistence with \u201cAuto\u201d link steering?\u201d. For example, in this post, the Edge-1 is having two Internet WAN links: GE3 (98.1.2.19) and GE4 (184.1.2.27). If the Client100 (192.168.200.100) open a browser access web service such as wordpress05 (43.254.254.14), the browser can open multiple TCP sessions which result of multiple TCP flows for Edge-1 to local Internet breakout. If the flows spread across two WAN links GE3 and GE4, the web server will see the request from the same client from different source IP address, this might confuse the web server, or a load balancer in front of the web server, and cause issue for accessing the web services.<\/p>\n\n\n\n

    If the reader takes a look at Test 3.2 and Test 3.3 (or you still remember the result), during these two tests, the link steering setting is \u201cAuto\u201d. Furthermore, the traffic flows are always initiated from the same client machine Client100 (192.168.200.100). From result of Test 3.2 and Test 3.3, we can realize with \u201cAuto\u201d link steering, the SD-WAN Edge does not have any client side\/source IP persistence consideration for selecting which WAN link to use. For example, if at time 0 the Edge-1 selected GE4 (184.1.2.27) for local Internet breakout, and then Client100 (192.168.200.100) initiate a new TCP flow at time N. At this time N, GE4 (184.1.2.27) downstream bandwidth is mostly consumed, then Edge-1 will select GE3 (98.1.2.19) for the new TCP flow. This will end up Client100(192.168.200.100) has some flows using GE4 (184.1.2.27) while some other flows using GE3 (98.1.2.19).<\/p>\n\n\n\n

    If source IP persistence is desired, a possible workaround is to use \u201cPreferred\u201d link steering instead of \u201cAuto\u201d link steering. Take a look of the following business polices configuration as an example:<\/p>\n\n\n\n

    \"\"
    Figure 83<\/figcaption><\/figure>\n\n\n\n

    For client IP address ends with even number, the traffic flow will match business policy \u201cEvenIP-Prefer-GE3-UDP\u201d or \u201cEvenIP-Prefer-GE3-Other\u201d, which will be preferably using GE3 (98.1.2.19). For client IP address ends with odd number, the traffic flow will match business policy \u201cOddIP-Prefer-GE4-UDP\u201d or \u201cOddIP-Prefer-GE4-Other\u201d, which will be preferably using GE4 (184.1.2.27). This sample configuration will be very similar to having client IP address persistence. The only exception is when the preferred link is down or unable to match the latency\/jitter\/packet loss SLA.<\/p>\n\n\n\n

    <\/p>\n\n\n\n\n\n\n\n

    What happen when the VCE is having multiple tunnels to different SD-WAN Gateways (VCG)?<\/h3>\n\n\n\n

    In the housekeeping section, it is mentioned the SD-WAN Edge (VCE) is by purpose to get assigned with a single VCG. This is because it will make manipulate the packet loss and latency value much easier. However, in production environment, the SD-WAN Edge is assigned with at least two VCG. Then in this situation, SD-WAN Edge Internet WAN link will have at least two overlay tunnels, one to each of the VCG. Since two overlay tunnels will have different latency\/jitter\/packet loss, this section will check which tunnel value will be used for the SD-WAN Edge to select WAN link for Internet Local Breakout.<\/p>\n\n\n\n

    Edge Configuration for Test 5.1<\/h5>\n\n\n\n

    Firstly, the business policy of Edge-1 is configured with link steering Auto for Internet local breakout traffic. The following is the screen capture of the business policy configuration:<\/p>\n\n\n\n

    \"\"
    Figure 84<\/figcaption><\/figure>\n\n\n\n

    In order to check when there is multiple overlay tunnels on a single Internet WAN link to different VCGs, Edge-1 will be assigned with two VCG with vcg-40-sfpg02 (24.11.0.55) as primary and vcg-40-sfpg01 (24.11.0.54) as secondary, the following screen captures from remote diagnostics and \u201cdebug.py –path\u201d shows now the Edge-1 is have two VCG assigned, which each Internet WAN link forms overlay tunnel to two VCGs:<\/p>\n\n\n\n

    \"\"
    Figure 85<\/figcaption><\/figure>\n\n\n\n

    The output of \u201cdebug.py –path\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 86<\/figcaption><\/figure>\n\n\n\n

    The \u201cdebug.py –gateway\u201d can help to confirm vcg-40-sfpg02 (24.11.0.55) is the primary gateway:<\/p>\n\n\n\n

    \"\"
    Figure 87<\/figcaption><\/figure>\n\n\n\n

    The following is the \u201cMonitor–>Overview\u201d of Edge-1, GE3 (98.1.2.19) is configured with 5Mpbs\/5Mbps while GE4 (184.1.2.30) is with 10Mbps\/10Mbps.<\/p>\n\n\n\n

    \"\"
    Figure 88<\/figcaption><\/figure>\n\n\n\n
    Test 5.1 \u2013 Introduce 4% packet loss to overlay tunnel between GE4 (184.1.2.30) and VCG vcg-40-sfpg02 (24.11.0.5)<\/h5>\n\n\n\n

    In the situation where all the overlay tunnels having latency\/jitter\/packet loss in GREEN, according from previous section \u201cWhen use \u201cAuto\u201d link steering, and every links are GREEN, which link will be picked?\u201d, we understand the Internet WAN link with larger remaining downstream bandwidth will be selected. In this case (where there is no traffic), the Internet WAN link selected will be GE4 (184.1.2.30) as the downstream bandwidth is 10Mbps which is larger than GE3 (98.1.2.19) downstream bandwidth with 5Mbps.<\/p>\n\n\n\n

    In Test 5.1, packet loss with a target of 4% will be introduced to the overlay tunnel between GE4 (184.1.2.30) and vcg-40-sfpg02 (24.11.0.55). And there will be no packet loss for overlay tunnel between GE4 (184.1.2.30) and vcg-40-sfpg01 (24.11.0.54). This is a specifically created scenario such that GE4 (184.1.2.30) is RED to primary gateway vcg-40-sfpg02 (24.11.0.55) while GREEN to secondary gateway vcg-40-sfpg01 (24.11.0.54).<\/p>\n\n\n\n

    Let\u2019s check the \u201cdebug.py –path\u201d output with this scenario:<\/p>\n\n\n\n

    \"\"
    Figure 89<\/figcaption><\/figure>\n\n\n\n

    The above screen capture shows, the introduction of packet loss is applied to the overlay tunnel we want. In the output, there are 4 overlay tunnels, three of them are without any packet loss. The only overlay tunnel with packet loss is the tunnel between (184.1.2.30) and vcg-40-sfpg02 (24.11.0.55), which having 3.08% loss at the Rx direction and 4.41% loss at the Tx direction.<\/p>\n\n\n\n

    Let\u2019s check the output of \u201cdebug.py –dec\u201d:<\/p>\n\n\n\n

    \"\"
    Figure 90<\/figcaption><\/figure>\n\n\n\n

    The output of \u201cdebug.py –dec\u201d shows overlay between (184.1.2.30) and vcg-40-sfpg02 (24.11.0.55) is RED on every type of traffic as the packet loss is larger than 3%. The output actually gives us a hint for predicting the result. Take a look on the two lines for GE4 with \u201cDest\u201d N\/A, these two lines show GE4 as the WAN link only (not with specific destination peer) is GREEN.<\/p>\n\n\n\n

    Let\u2019s generate some traffic from Client100 (192.168.200.100), firstly, access web service of wordpress05 (43.254.254.14). The following is the web access log from wordpress05 (43.254.254.14):<\/p>\n\n\n\n

    \"\"
    Figure 91<\/figcaption><\/figure>\n\n\n\n

    From the wordpress05 (43.254.254.14) access log, the source IP is 184.1.2.30, this means Edge-1 selected GE4 (184.1.2.30) for local Internet breakout when link steering is Auto.<\/p>\n\n\n\n

    Let\u2019s give a try to generate traffic by iperf3 from Client100 (192.168.200.100) to wordpress05 (43.254.254.14). The following is the screen capture from the iperf3 client:<\/p>\n\n\n\n

    \"\"
    Figure 92<\/figcaption><\/figure>\n\n\n\n

    The following screen capture is the iperf3 server at wordpress05 (43.254.254.14):<\/p>\n\n\n\n

    \"\"
    Figure 93<\/figcaption><\/figure>\n\n\n\n

    With the screen capture from the iperf3 server, it is showing the connection is from 184.1.2.30:20006. This means Edge-1 selected GE4 (184.1.2.30) for local Internet breakout when link steering is Auto.<\/p>\n\n\n\n

    The following is the screen capture of the \u201cList Active Flows\u201d for the completeness of the test.<\/p>\n\n\n\n

    \"\"
    Figure 94<\/figcaption><\/figure>\n\n\n\n

    With the above test result, the conclusion is, when SD-WAN Edge is having multiple overlay tunnels to different VCGs in a WAN link. Only the best quality overlay tunnel is used to determine that particular WAN link is GREEN\/YELLOW\/RED. Take the above test as an example, GE4 (184.1.2.30) is having two overlay tunnels, one to vcg-40-sfpg02 (24.11.0.55) which is RED. And there is a another overlay tunnel to vcg-40-sfpg01 (24.11.0.54) which is GREEN. In this scenario, the GE4 (184.1.2.30) is consider as GREEN as the best quality overlay tunnel is GREEN.<\/p>\n\n\n\n

    Note: I believe the idea behind this is, if the best quality overlay tunnel is poor, such as YELLOW\/RED, then it is very likely this WAN link is with poor quality at that point of time. That\u2019s why only the best quality overlay tunnel is used to determine the WAN link\u2019s quality.<\/span><\/p>\n\n\n\n\n\n\n\n

    Final conclusion for Internet local breakout with two Internet WAN links<\/h3>\n\n\n\n

    When link steering is \u201cAuto\u201d:<\/strong><\/span><\/h5>\n\n\n\n

    For Internet local breakout new flow, the SD-WAN Edge selects the GREEN WAN link over YELLOW\/RED link. In the situation where multiple GREEN WAN links are available, SD-WAN Edge selects the WAN link with the highest remaining downstream bandwidth. The \u201cremaining downstream bandwidth\u201d is calculated by:<\/strong><\/p>\n\n\n\n

    Remaining downstream bandwidth = configure\/discovered downstream bandwidth – consumed downstream bandwidth<\/em><\/strong><\/p>\n\n\n\n

    In addition, when picking the WAN link with the highest remaining downstream bandwidth, the SD-WAN Edge does not care if the WAN link is very GREEN (such as 0ms latency) or the WAN link is barely GREEN (such as 22ms latency). As long as the WAN link is GREEN, that link is entitled to be selected based on the remaining downstream bandwidth.<\/strong><\/p>\n\n\n\n

    Upstream bandwidth consumption is not in the link selection consideration. There is no source IP address persistence (for link steering \u201cAuto\u201d).<\/strong><\/p>\n\n\n\n

    When link steering is \u201cPreferred\u201d:<\/span><\/strong><\/h5>\n\n\n\n

    For Internet local breakout new flow, SD-WAN Edge will select the configured preferred WAN link, regardless what is that WAN link downstream bandwidth utilization. However, the configured preferred WAN link still needs to meet the link quality (packet loss, jitter, latency) SLA. If the link quality SLA is not met and there is other GREEN link available, the SD-WAN Edge will select other GREEN link for new flow. In addition, if the configured preferred WAN link is down, the SD-WAN Edge will select other available link for new flow.<\/strong><\/p>\n\n\n\n

    Note: What exactly conditions for meeting link quality (packet loss, jitter, latency) SLA is not covered in this post.<\/strong><\/p>\n\n\n\n

    When link steering is \u201cMandatory\u201d:<\/span><\/h5>\n\n\n\n

    For Internet local breakout new flow, SD-WAN Edge always select the configured mandatory WAN link for traffic flow. In the situation the configured mandatory WAN link is down, the traffic flow will get dropped. There is no failover to other WAN link when the link steering is Mandatory.<\/strong><\/p>\n\n\n\n

    NOTE: When the Internet WAN link is having multiple tunnels to different SD-WAN Gateway (VCG):<\/span><\/h6>\n\n\n\n
    Only the best quality overlay tunnel is used to determine that particular WAN link is GREEN\/YELLOW\/RED.<\/h6>\n\n\n\n

    End of post “Local Internet Breakout with two Internet Links – VMware SD-WAN by Velocloud”<\/p>\n","protected":false},"excerpt":{"rendered":"

    Background For users behind the VMware SD-WAN Edge (that is user on the LAN side), one of the options for access Internet is local Internet breakout. When the SD-WAN Edge connected to more than one Internet link, questions like \u201cwhich link will be picked for local breakout?\u201d, \u201ccan the SD-WAN Edge perform traffic load balance?\u201d […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"templates\/template-fullwidth.php","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[10,5],"tags":[],"class_list":["post-248","post","type-post","status-publish","format-standard","hentry","category-local-breakout","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/248","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=248"}],"version-history":[{"count":73,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/248\/revisions"}],"predecessor-version":[{"id":424,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/248\/revisions\/424"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=248"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=248"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=248"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}