{"id":483,"date":"2021-10-13T06:12:00","date_gmt":"2021-10-13T06:12:00","guid":{"rendered":"https:\/\/www.sdwan2.com\/?p=483"},"modified":"2021-10-13T08:21:25","modified_gmt":"2021-10-13T08:21:25","slug":"vmware-sd-wan-ebgp-with-azure-route-server","status":"publish","type":"post","link":"https:\/\/www.sdwan2.com\/index.php\/2021\/10\/13\/vmware-sd-wan-ebgp-with-azure-route-server\/","title":{"rendered":"VMware SD-WAN eBGP with Azure Route Server"},"content":{"rendered":"\n

Objective<\/h3>\n\n\n\n

The objective of this post is to document how to connect site(s) with VMware SD-WAN Edge to Azure VNet(s), where in the Azure side there is a transit VNet with virtual edge forming eBGP with Azure Route Server (https:\/\/docs.microsoft.com\/en-us\/azure\/route-server\/overview). Let’s take a look at the following figures which show the “before” and “after”.<\/p>\n\n\n\n

\"\"
Figure 1 – Before deploying virtual edge and Route Server<\/figcaption><\/figure>\n\n\n\n

Figure 1 shows there are two VNets, namely SG-VNET1 (10.210.0.0\/17) and SG-VNET2 (10.210.218.17). In SG-VNET1, there is a Linux VM called VNet1-Linux with IP address 10.210.0.4. In SG-VNET2, there is a Linux VM called VNet2-Linux with IP address 10.210.128.4. There is a SD-WAN Edge called RT-Spoke1 with LAN subnet 10.11.1.0\/24, and let’s call this 10.11.1.0\/24 location Spoke1 Site. As shown in the diagram, at this point, the VNets and Spoke1 Site are not connected. The next figure shows the end state this post would achieve.<\/p>\n\n\n\n

\"\"
Figure 2 -After deploy virtual edge and Route Server<\/figcaption><\/figure>\n\n\n\n

In order to let Spoke1 Site connects to the two VNet, a transit VNet called SG-VNet-Transit is created. And then Route Server will be created in SG-VNet-Transit. Furthermore, a couple of virtual SD-WAN Edges (Azure-SG-Edge1 and Azure-SG-Edge2) are deployed in SG-VNet-Transit, this pair of SD-WAN Edges will form eBGP with the Route Server, they are also the “hub” for RT-Spoke1. The SG-VNet-Transit VNet will add VNet peering (https:\/\/docs.microsoft.com\/en-us\/azure\/virtual-network\/virtual-network-peering-overview) with SG-VNET1 and SG-VNET2. The VNet peering will allow the SG-VNET1 (10.210.0.0\/17) and SG-VNET2 (10.210.128.0\/17) address space advertised from the Route Server to Azure-SG-Edge1 and Azure-SG-Edge2. The VNet peering also allows the routes Route Server learnt from the virtual edge (which is 10.11.1.0\/24) to be available at SG-VNET1 and SG-VNET2.<\/p>\n\n\n\n

The reason of having two virtual edges in the transit VNet is for redundancy. In this post, the Azure-SG-Edge1 will be acting as a primary edge while Azure-SG-Edge2 is a standby. The method used to achieve primary and standby is AS Path prepend at BGP peer configuration of Azure-SG-Edge2.<\/p>\n\n\n\n

The upcoming sections will show the step-by-step how to of deploying the virtual edges and Route Server. <\/p>\n\n\n\n\n\n\n\n

1. Create Transit VNet<\/h3>\n\n\n\n
\"\"
Figure 3 – Create Transit VNet SG-Transit-VNet<\/figcaption><\/figure>\n\n\n\n

The first step is to select the Resource Group and Region for the VNet, in this example, the region is Southeast-Asia. The name of the VNet is SG-Transit-VNet. Then click “Next: IP Addresses”:<\/p>\n\n\n\n

<\/p>\n\n\n\n

\"\"
Figure 4 – Define the address space, and also create two subnets SG-Transit-Private and SG-Transit-Public<\/figcaption><\/figure>\n\n\n\n

In the second screen of creating a VNet, we need to supply the address space, in this example, the address space in 10.209.0.0\/23. Two subnets will be created as well, they are SG-Transit-Private (10.209.0.0\/25) and SG-Transit-Public (10.209.0.128\/25).<\/p>\n\n\n\n\n\n\n\n

2. Create a pair of virtual edges (Azure-SG-Edge1 and Azure-SG-Edge2)<\/h3>\n\n\n\n

2.1 Create the SD-WAN Edge in the VCO<\/h4>\n\n\n\n

The major purpose of this post to show how the SD-WAN Edges and form eBGP with the Route Server to provide connectivity, but not showing the detail step-by-step of how to deploy SD-WAN Edges in Azure environment. Thus, only key steps will be shown.<\/p>\n\n\n\n

The following figure shows the interface setting of virtual edge Azure-SG-Edge1:<\/p>\n\n\n\n

\"\"
Figure 5 – interface setting of virtual edge<\/figcaption><\/figure>\n\n\n\n

The GE1 interface will be WAN facing while GE2 interface will be LAN facing. As a result, the GE1 and GE2 both needs to be converted from switch port to routed port. GE1 will be using DHCP and Auto Detect overlay. GE2 will be using static IP (because it will form BGP peer) with overlay disabled. The following diagram shows the GE2 configuration:<\/p>\n\n\n\n

\"\"
Figure 6 – GE2 configuration, WAN overlay is disabled<\/figcaption><\/figure>\n\n\n\n

2.2 Create the SD-WAN Edge in the Azure portal<\/h4>\n\n\n\n

After the virtual edge Azure-SG-Edge1 is created in the VCO, we can create the virtual edge in the Azure portal. The recommended way is to use Azure Resource Manager (ARM) template to create the VMware SD-WAN virtual edge in Azure environment. The template used in this post can be downloaded from here: https:\/\/code.vmware.com\/samples\/7633\/vmware-sd-wan-azure-resource-manager-template-2-nic-only<\/a><\/p>\n\n\n\n

To create the virtual edge in Azure, search by keyword “deploy” in Azure portal, and select “Deploy a custom template”. Then select “Build your own template in the editor”. There is an option called “Load file”, click on that and upload the template downloaded from the URL in previous paragraph. When the upload is completed, select “Save”. After that, some information are required to be filled:<\/p>\n\n\n\n

\"\"
Figure 7 – Information to fill in the template for virtual edge creation<\/figcaption><\/figure>\n\n\n\n

Since the VNet SG-Transit-VNet is already created, select “existing” for the “Virtual Network New Or Existing”. The public subnet is the Internet side, that is the WAN side. The private subnet is the LAN side, which is the interface forming eBGP with the Route Server. The public subnet (SG-Transit-Public) and private subnet (SG-Transit-Private) are already created when the VNet is being created, fill back the corresponding information here. Finally, the Edge GE2LANIP is the GE2 IP address, it needs to be match the IP address configured in the VCO and also fall under the private subnet, in this example, the IP address is 10.209.0.10.<\/p>\n\n\n\n

After filling all the necessary information, click “Review + create”. After the review, click “Create”. The virtual edge will be created and automatically reach out to the VCO for activation. The entire process usually takes about 3-5 minutes. If everything goes as expected, the VCO monitoring page of the virtual edge should will show the Link Status as green and the bandwidth detected successfully. Here is the Azure-SG-Edge1 status after it is being created:<\/p>\n\n\n\n

\"\"
Figure 8 – Azure-SG-Edge1 status after being created<\/figcaption><\/figure>\n\n\n\n

The creation of Azure-SG-Edge2 is very similar to Azure-SG-Edge1, the process will not be repeated here.<\/p>\n\n\n\n\n\n\n\n

3. Assign Azure-SG-Edge1 and Azure-SG-Edge2 as hub for Spoke1 site<\/h3>\n\n\n\n

To assign Azure-SG-Edge1 and Azure-SG-Edge2 as the hub for RT-Spoke1 (which is the SD-WAN Edge at Spoke1 site), in the VCO, go to “Configure –> Profiles”, select the profile of RT-Spoke1. In Cloud VPN, enable Branch to Hubs and select Azure-SG-Edge1 and Azure-SG-Edge2 as the hub. The following screen capture shows the corresponding profile settings:<\/p>\n\n\n\n

\"\"
Figure 9 – Assign Azure-SG-Edge1 and Azure-SG-Edge2 as hub in the spoke profile<\/figcaption><\/figure>\n\n\n\n\n\n\n\n

4. Create Route Server Subnet and Route Server<\/h3>\n\n\n\n

4.1 Create Route Server Subnet<\/h4>\n\n\n\n

A “Route Server Subnet” is mandatory for the Route Server. In this post, the Route Server Subnet is created prior to the creation of Route Server. Please note the Route Server Subnet is mandatory to have a name called “RouteServerSubnet”, any other name will not be accepted. This is the corresponding screen capture of creating Route Server Subnet:<\/p>\n\n\n\n

\"\"
Figure 10 – Create Route Server Subnet<\/figcaption><\/figure>\n\n\n\n

Figure 10 shows how to create RouteServerSubnet, in this example, the address range for this subnet is 10.209.1.0\/25<\/p>\n\n\n\n

4.2 Create Route Server<\/h4>\n\n\n\n

To create route server, search Route Server in Azure portal and click on Route Server. When arrived in the Route Server menu, click on “Create new route server”. The following is the Route Server creation screen used in this post:<\/p>\n\n\n\n

\"\"
Figure 11 – Create Route Server<\/figcaption><\/figure>\n\n\n\n

We need to select the region where the Route Server will be located, in this post, the Route Server named SG-Transit-RouteServer is located at Southeast Asia (Singapore). The Route Server will be placed in VNet SG-Transit-VNet. The subnet is the RouteServerSubnet created in the previous step. A public IP address is mandatory for Route Server, the option of create a new public IP address was selected. After every field is filled, click “Review + create”, and then click “Create”, the Route Server will be created. From my experience, it took around 15-20 minutes for the Route Server get created.<\/p>\n\n\n\n\n\n\n\n

5. Configure BGP on Route Server<\/h3>\n\n\n\n

After the SG-Transit-RouteServer is created, we need to configure the eBGP peer. Click on the Route Server SG-Transit-RouteServer and then click on Peers on the left hand side menu. On the Peers page, click “Add”. In this post, since there are two SD-WAN Edges, thus we need to add two eBGP peers. The first eBGP peer with Azure-SG-Edge1 (AS number 65123 and IP address 10.209.0.10) is as follow:<\/p>\n\n\n\n

\"\"
Figure 12 – Configure eBGP peer with Azure-SG-Edge1<\/figcaption><\/figure>\n\n\n\n

The second eBGP peer is with Azure-SG-Edge2 (AS number 65123 and IP address 10.209.0.11) is as follow:<\/p>\n\n\n\n

\"\"
Figure 13 – Configure eBGP peer with Azure-SG-Edge2<\/figcaption><\/figure>\n\n\n\n

After these two eBGP peers are created successfully, the two peers will be listed like this:<\/p>\n\n\n\n

\"\"
Figure 14 – Two BGP peers created successfully in the Route Server<\/figcaption><\/figure>\n\n\n\n\n\n\n\n

6. Configure BGP on SD-WAN Edge<\/h3>\n\n\n\n

Before we can configure the BGP on the SD-WAN Edge, we need to obtain the Route Server peer IP address and AS Number. To obtain those information, click “Overview” of the corresponding Route Server. For example:<\/p>\n\n\n\n

\"\"
Figure 15 – Overview of SG-Transit-RouteServer<\/figcaption><\/figure>\n\n\n\n

From the overview session, we can find the AS Number is 65515 and peer IP addresses are 10.209.1.4 and 10.209.1.5. Then, we will configure the eBGP peer of the SD-WAN Edge, the following is the configuration for Azure-SG-Edge1:<\/p>\n\n\n\n

\"\"
Figure 16 – BGP configuration of Azure-SG-Edge1<\/figcaption><\/figure>\n\n\n\n

The Azure-SG-Edge1 local AS number is 65123, there are two eBGP peers of IP address 10.209.1.4 and 10.209.1.5, both peer are having AS number 65515. The Keep Alive and Hold Timers are adjusted to 10 and 30 respectively to allow a faster fail over in the situation of the primary SD-WAN Edge goes down.<\/p>\n\n\n\n

The max-hop cannot be the default value of 1 because the peer is not in the same subnet, in this post a value of 3 for max-hop is used. This brings up another problem, there is no route for the SD-WAN Edge to reach the eBGP peer (route server). As a result, a static route is required:<\/p>\n\n\n\n

\"\"
Figure 17 – Static route to allow the SD-WAN Edge to reach the Route Server to form eBGP<\/figcaption><\/figure>\n\n\n\n

Azure-SG-Edge2 also requires the same static route. Let’s take a look a the Azure-SG-Edge2 BGP configuration:<\/p>\n\n\n\n

\"\"
Figure 18 – BGP configuration of Azure-SG-Edge2<\/figcaption><\/figure>\n\n\n\n

Since Azure-SG-Edge2 is intended to be a standby of Azure-SG-Edge1, there are AS number prepend configured at both inbound and outbound direction. With the AS number prepend, Azure-SG-Edge2 will be less preferred from the Route Server and also the spoke sites. Other than the AS number prepend, the BGP configuration of Azure-SG-Edge2 is the same as Azure-SG-Edge1.<\/p>\n\n\n\n\n\n\n\n

7. Configure VNet Peering<\/h3>\n\n\n\n

Without VNet Peering, the Route Server SG-Transit-RouteServer will not accept or advertise the address space of other VNet(s). Thus, the next task is to create peering between SG-VNET1 and SG-Transit-VNet. To do this, go to the “Peerings” of SG-VNET1 and click “Add”. When add the peering, the configuration is divided to “This virtual network” and “Remote virtual network”. Let’s look at the Remote virtual network configuration first:<\/p>\n\n\n\n

\"\"
Figure 19 – SG-VNET1 peering: Remote virtual network<\/figcaption><\/figure>\n\n\n\n

Since the purpose is to peer with SG-Transit-VNet, SG-Transit-VNet is selected as the VNet under Remote virtual network. As we want two ways communication between SG-VNET1 and SG-Transit-VNet, the “Traffic to remote virtual network” and “Traffic forwarded from remote virtual network” setting will be configured as “Allow”, which is the default setting. More importantly, under “Virtual network gateway or Route Server”, the options to select is “Use this virtual network’s gateway or Route Server” instead of None. This is because the Route Server is at SG-Transit-VNet, with this setting, virtual machines in SG-VNet1 can utilize the Route Server advertised routes to reach the spoke sites. Now, let’s go back to the configuration of “This virtual network”:<\/p>\n\n\n\n

\"\"
Figure 20 – SG-VNET1 peering, this virtual network<\/figcaption><\/figure>\n\n\n\n

The option needs to pay attention is “Virtual network gateway or Route Server”, “Use the remote virtual network’s gateway or Route Server” needs to be selected instead of None. Again, this is to allow virtual machines in SG-VNET1 can utilize the Route Server advertised routes to reach the spoke sites. Finally, click “Add” to create peering between SG-VNET1 and SG-Transit-VNet. <\/p>\n\n\n\n

SG-VNet2 also needs to create peering with SG-Transit-VNet. Since the peering configuration for SG-VNET2 is the same as SG-VNET1, it will not be repeated here.<\/p>\n\n\n\n\n\n\n\n

8. Verification of the routing configuration<\/h3>\n\n\n\n

The configurations are completed, it is time to verify the configurations are done correctly or not.<\/p>\n\n\n\n

8.1 Verify the effective route on Azure VNets<\/h4>\n\n\n\n

Let’s check the effective route on Azure VNets SG-VNET1 and SG-VNET2, this will help to confirm can the SG-VNET1 and SG-VNET2 learnt the Spoke1 site LAN subnet 10.11.1.0\/24. To do this, go to virtual machine, VNet1-Linux1, select Networking, and then click on the Network Interface, which is called vnet1-linux1589 in this example. Once arrived at the network interface, click “Effective routes” on the left side menu:<\/p>\n\n\n\n

\"\"
Figure 21 – Effective routes of Linux machine VNet1-Linux1<\/figcaption><\/figure>\n\n\n\n

The third row is the route of 10.11.1.0\/24, with source “Virtual network gateway” and next hop 10.209.0.10. Actually, 10.209.0.10 is the GE2 IP address of the SD-WAN Edge Azure-SG-Edge1. This confirms the SG-VNET1 route table is able to learnt the Spoke1 site LAN prefix 10.11.1.0\/24 successfully (from the VNet peering and Route Server).<\/p>\n\n\n\n

The same procedure needs to be done at SG-VNET2 virtual machine VNet2-Linux1:<\/p>\n\n\n\n

\"\"
Figure 22 – Effective routes of Linux machine VNet2-Linux1<\/figcaption><\/figure>\n\n\n\n

Again, the third row is 10.11.1.0\/24 which is the Spoke1 site LAN prefix. The above confirmed, in both VNet, they can learn the 10.11.1.0\/24 from the Route Server SG-Transit-RouteServer through the VNet peering. The next hop of 10.209.0.10 shows the Route Server will not change the next hop when advertise the routes, the next hop is maintained as 10.209.0.10 which is the Azure-SG-Edge1 GE2 IP address.<\/p>\n\n\n\n

8.2 Verify the BGP status at Azure-SG-Edge1 and Azure-SG-Edge2<\/h4>\n\n\n\n

Let’s check the BGP status of Azure-SG-Edge1:<\/p>\n\n\n\n

\"\"
Figure 23 – Show BGP summary of Azure-SG-Edge1<\/figcaption><\/figure>\n\n\n\n
\"\"
Figure 24 – Show BGP table of Azure-SG-Edge1<\/figcaption><\/figure>\n\n\n\n

From the “Show BGP Summary” output, Azure-SG-Edge1 established eBGP peer with 10.209.1.4 and 10.209.1.5 (they are the IP address of the Route Server) successfully. The “Show BGP Table” output confirms Azure-SG-Edge1 is able to learnt routes 10.210.0.0\/17 (SG-VNET1 address space) and 10.210.128.0\/17 (SG-VNET2 address space).<\/p>\n\n\n\n

Let’s take a look at Azure-SG-Edge2:<\/p>\n\n\n\n

\"\"
Figure 25 – Show BGP Summary of Azure-SG-Edge2<\/figcaption><\/figure>\n\n\n\n
\"\"
Figure 26 – Show BGP Table of Azure-SG-Edge2<\/figcaption><\/figure>\n\n\n\n

Azure-SG-Edge2 also forms two eBGP peers with 10.209.1.4 and 10.209.1.5. It also learns 10.210.0.0\/17 and 10.210.128.0\/17 from the Route Server, the difference is the AS Path gets two additional AS number 65123. This confirms the AS number prepend is working at Azure-SG-Edge2.<\/p>\n\n\n\n

Next, take a look at the route table of the Spoke1 site, that is SD-WAN Edge RT-Spoke1:<\/p>\n\n\n\n

\"\"
Figure 27 – Route Table of RT-Spoke1<\/figcaption><\/figure>\n\n\n\n

Pay attention to the two routes 10.210.0.0\/17 and 10.210.128.0\/17. Both routes are learnt from both Azure-SG-Edge1 and Azure-SG-Edge2. The route on top is with next hop Azure-SG-Edge1 (when multiple same prefix routes appear in the route table, the one on top is the one being used), this is because the AS Path information is carried over. As a result, RT-Spoke1 will prefer Azure-SG-Edge1 as next hop instead of Azure-SG-Edge2, it is because the AS Path is shorter when the route is learnt from Azure-SG-Edge1.<\/p>\n\n\n\n\n\n\n\n

9. Ping Test and SD-WAN Edge failover test<\/h3>\n\n\n\n

Let’s start ping from the Win10-A2 (10.11.1.11) to the two Linux machines VNet1-Linux1 (10.210.0.4) and VNet2-Linux1 (10.210.128.4):<\/p>\n\n\n\n

\"\"
Figure 27 – Ping from Win10-A2 (10.11.1.11) to VNet1-Linux1 (10.210.0.4) and VNet2-Linux1 (10.210.128.4) <\/figcaption><\/figure>\n\n\n\n

The ping test verified the Windows machine at the Spoke1 site is able to reach the virtual machine in both SG-VNET1 and SG-VNET2.<\/p>\n\n\n\n

Next, let’s try a fail over test of the virtual edge. In the test, the Azure-SG-Edge1 will get powered off and will observe can the continuous ping traffic switch over to Azure-SG-Edge2:<\/p>\n\n\n\n

\"\"
Figure 28 – Failover from Azure-SG-Edge1 to Azure-SG-Edge2<\/figcaption><\/figure>\n\n\n\n

The Win10-A2 issues continuous ping to 10.210.0.4 and 10.210.128.4. The first “Request timed out” is the time Azure-SG-Edge1 gets powered off. From the above screen capture, there are about 5-6 ping loss for traffic to get resume by using Azure-SG-Edge2. The time for failover is around 30 seconds. This test confirms in the case of Azure-SG-Edge1 (the primary hub) goes down, Azure-SG-Edge2 will be able to pick up the traffic automatically.<\/p>\n\n\n\n

The effective routes from VNet1-Linux1 after the failover is like this:<\/p>\n\n\n\n

\"\"
Figure 29 – Effective routes of VNet1-Linux1<\/figcaption><\/figure>\n\n\n\n

By looking at the next hop of 10.11.1.0\/24, the next hop is now 10.209.0.11 (Azure-SG-Edge2) instead of 10.209.0.10 (Azure-SG-Edge1). This further confirmed the failover from Azure-SG-Edge1 to Azure-SG-Edge2 is successful.<\/p>\n\n\n\n

10. Conclusion<\/h3>\n\n\n\n

This post demonstrates how to utilize VMware SD-WAN Edge and Azure Route Server to connect the sites with SD-WAN to multiple Azure VNets. The post ends here.<\/p>\n","protected":false},"excerpt":{"rendered":"

Objective The objective of this post is to document how to connect site(s) with VMware SD-WAN Edge to Azure VNet(s), where in the Azure side there is a transit VNet with virtual edge forming eBGP with Azure Route Server (https:\/\/docs.microsoft.com\/en-us\/azure\/route-server\/overview). Let’s take a look at the following figures which show the “before” and “after”. Figure […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[11,5],"tags":[],"class_list":["post-483","post","type-post","status-publish","format-standard","hentry","category-public-cloud","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/483","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=483"}],"version-history":[{"count":42,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/483\/revisions"}],"predecessor-version":[{"id":560,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/483\/revisions\/560"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=483"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=483"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=483"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}