{"id":671,"date":"2023-07-05T05:56:06","date_gmt":"2023-07-05T05:56:06","guid":{"rendered":"https:\/\/www.sdwan2.com\/?p=671"},"modified":"2023-07-05T05:56:06","modified_gmt":"2023-07-05T05:56:06","slug":"flow-visibility-vmware-sd-wan","status":"publish","type":"post","link":"https:\/\/www.sdwan2.com\/index.php\/2023\/07\/05\/flow-visibility-vmware-sd-wan\/","title":{"rendered":"Flow Visibility (VMware SD-WAN)"},"content":{"rendered":"\n

Background<\/h2>\n\n\n\n

In VMware SD-WAN version 5.1, a new feature called Flow Visibility is introduced. The following is the feature description in the 5.1 release note (https:\/\/docs.vmware.com\/en\/VMware-SASE\/5.1.0\/rn\/vmware-sase-510-release-notes\/index.html#What%20Is%20in%20The%20Release%20Notes-New%20SD-WAN%20Features<\/a>):<\/p>\n\n\n\n

\n

Flow Visibility<\/strong><\/p>\n\n\n\n

In previous releases, the Orchestrator UI only displays aggregated flow information and statistics individually from the lens of Application<\/strong>, Source<\/strong>, or Destination<\/strong> and does not combine all this information on one screen to provide a single, end-to-end view. As a result, monitoring, troubleshooting, and reporting are hindered by the lack of detailed visibility of each flow.<\/p>\n\n\n\n

With Release 5.1.0, the New Orchestrator UI includes a \u201cFlows\u201d tab that displays the consolidated data for each traffic flow. The Orchestrator UI displays each flow\u2019s key parameters in a single view.  In addition, the Flow Visibility<\/strong> feature allows customers to view historical flow data, filter data based on matched parameters, and download end-to-end flow statistics.<\/p>\n<\/blockquote>\n\n\n\n

In this post, some tests are being perform to check how this Flow Visibility work.<\/p>\n\n\n\n

Test Environment and Topology<\/h2>\n\n\n\n

Version<\/h3>\n\n\n\n

The following are the software version used in the tests for this post:<\/p>\n\n\n\n

VCO: R5105-20230611-0336-GA-a7c4c61840
VCG: R5103-20230621-GA-9d8588f02e
VCE: R5102-20230310-GA-6ee0ca8b2d<\/p>\n\n\n\n

The following diagram shows the test topology:<\/p>\n\n\n\n

\"\"
Diagram 1: The test topology<\/figcaption><\/figure>\n\n\n\n

In this lab environment, everything is simulated including the Internet. This is a hub and spoke topology where the hub site (Left-Hub-1) is having a LAN subnet 10.230.3.0\/24, while the spoke site (Left-Spoke-1) is having a LAN subnet 10.2.1.0\/24. On the hub site, there is a iperf server (Joe-LinuxC-L2) with IP address 10.230.3.230. On the spoke site, there is a iperf client (Joe-LinuxC-L1) with IP address 10.2.1.111. These two VMs are used to generate iperf traffic so flows are being created in the flows tab of the VCO.<\/p>\n\n\n\n

WAN status and business policy<\/h2>\n\n\n\n

Let’s take a look on the spoke and hub site WAN status:<\/p>\n\n\n\n

\"\"
Diagram 2: Left-Spoke-1 WAN status<\/figcaption><\/figure>\n\n\n\n
\"\"
Diagram 3: Left-Hub-1 WAN status<\/figcaption><\/figure>\n\n\n\n

The following are the business policy configured, basically the link steering is configured to auto:<\/p>\n\n\n\n

\"\"
Diagram 4: Left-Spoke-1 Policy<\/figcaption><\/figure>\n\n\n\n
\"\"
Diagram 5: Left-Hub-1 Policy<\/figcaption><\/figure>\n\n\n\n

The following is the list path output of Left-Spoke-1 to show the tunnel status with Left-Hub-1:<\/p>\n\n\n\n

\"\"
Diagram 6: List Path output of Left-Spoke-1 with peer Left-Hub-1<\/figcaption><\/figure>\n\n\n\n

The purpose of showing the WAN status and business policy is to confirm both the Internet and MPLS tunnel between Left-Spoke-1 and Left-Hub-1 are able to establish. In addition, the business policy is configured with link steering auto.<\/p>\n\n\n\n\n\n\n\n

Flow Visibility Tests<\/h2>\n\n\n\n

Test 1: iperf with a single data flow<\/h3>\n\n\n\n

The iperf command being used is<\/p>\n\n\n\n

\/usr\/local\/bin\/iperf3 -c 10.230.1.230 -w 128k -b 2M -t 20<\/code><\/pre>\n\n\n\n
Screen capture of the iperf3 client output:<\/p>\n\n\n\n
\"\"
Diagram 7: iperf3 client command in test1<\/figcaption><\/figure>\n\n\n\n
While the iperf3 is running, packet capture was done on the iperf client machine (10.2.1.111). The following is the conversation of the packet capture:<\/p>\n\n\n\n
\"\"
Diagram 8: packet capture conversations at iperf client machine during test1<\/figcaption><\/figure>\n\n\n\n
The packet capture shows us there are two TCP stream 10.2.1.111:40072 <–> 10.230.1.230:5201 and 10.2.1.111:40086 <–> 10.230.1.230:5201. The one with source port 40072 should be the iperf control connection while the data are being transmit at the one with source port 40086.<\/p>\n\n\n\n
Now, let’s take a look of what this become in the “Flows” tab in the VCO:<\/p>\n\n\n\n
\"\"
Diagram 9: Flows output generate by iperf3 in test1 (the left part)<\/figcaption><\/figure>\n\n\n\n
\"\"
Diagram 10: Flows output generate by iperf3 in test1 (the right part)<\/figcaption><\/figure>\n\n\n\n
Firstly, because the flows contain a lot of information, the output is too width to display in a single screen capture, two screen captures are used to show the entire flow information. To make reading the flow easier, the related flow generate by the iperf test is marked with red square bracket (sorry for my bad drawing). There are a few things caught my attentions:<\/p>\n\n\n\n
    \n
  1. There is no “source port” column.<\/li>\n\n\n\n
  2. There are two row here and seems there are two rows because some traffic is using GE3 (Internet) and some traffic is using GE4 (MPLS).<\/li>\n\n\n\n
  3. The duration of the flows are always a 5-minute interval, in this example is 1:44:59pm to 1:49:59pm, while the iperf ran only for 20 seconds.<\/li>\n\n\n\n
  4. There is a next hop showing the next hop SD-WAN Edge which is Left-Hub-1. And there is a route type of “Branch To Branch”.<\/li>\n<\/ol>\n\n\n\n
    For the no “source port” column, it is by design. Actually this comes back to the release note mentioned “consolidated data for each traffic flow”, that is how information are consolidated? It is understandable if every flow are recorded in the VCO, that will consume a lot of disk space. That’s why some sort of consolidation is required. It looks like if multiple flows from the same source IP (even with different source port) and visiting the same destination IP and port, they are grouped in the same “flow entry”. In the next section, will carry a iperf with multiple flows to confirm.<\/p>\n\n\n\n
    Since the DMPO is per packet, it is possible to have the same flow utilizing multiple WAN (that is the link column). From this test, when the flow spread across multiple WAN, GE3 and GE4, the flow are showing separately. That is in the flow visibility, each WAN link needs to count individually to display the data.<\/p>\n\n\n\n
    Regarding the start and stop time, the iperf3 traffic actually happens between 1:47:14pm to 1:47:34pm. And the flows show in the VCO is between 1:44:59pm to 1:49:59pm. We can realize the flows are always with a 5 minute interval in the VCO. This aligns with the current monitoring of SD-WAN Edge, the statistic collection is with a granularity of 5 minutes. That means the flow visibility is not to record the start and stop of individual flow, it records down flows fall into the 5-minutes interval, consolidate them as long as source IP, destination IP, destination port are the same.<\/p>\n\n\n\n
    So far, we are looking at the “Flows” at the spoke site Left-Spoke-1, let’s take a look at the hub site Left-Hub-1:<\/p>\n\n\n\n
    \"\"
    Diagram 11: Flows output generate by iperf3 in test1 in Left-Hub-1 (left part)<\/figcaption><\/figure>\n\n\n\n
    \"\"
    Diagram 12:  Flows output generate by iperf3 in test1 in Left-Hub-1 (right part)<\/figcaption><\/figure>\n\n\n\n
    I believe the first thing caught our attention is the destination port is 5201. The iperf3 server is 10.230.1.230 listening at port 5201. If the flow is talking about source IP 10.230.1.230 with destination IP 10.2.1.111, then technically the destination port is not 5201, the destination port should be 40072 and 40086 from the packet capture. So why the flow data is displayed like this in the hub site Left-Hub-1?<\/p>\n\n\n\n
    The flow does not indicate it is “LAN to WAN” or “WAN to LAN”. Or in VMware SD-WAN’s term, is the flow locally initiated or initiated by the peer. I think VMware needs to enhance the flow visibility to include this to make the flow information making more sense.<\/p>\n\n\n\n
     Let’s check the “list active flow” in the Left-Hub-1 remote diagnostic to check:<\/p>\n\n\n\n
    \"\"
    Diagram 13: List active flows at Left-Hub-1<\/figcaption><\/figure>\n\n\n\n
    From List Active Flows at Left-Hub-1:<\/p>\n\n\n\n
      \n
    • Source IP: 10.2.1.111<\/li>\n\n\n\n
    • Source Port: 40072\/40086<\/li>\n\n\n\n
    • Destination IP: 10.230.1.230<\/li>\n\n\n\n
    • Destination Port: 5201<\/li>\n<\/ul>\n\n\n\n
      From the “Flows” tab for Left-Hub-1 (flow visibility):<\/p>\n\n\n\n
        \n
      • Source IP: 10.230.1.230<\/li>\n\n\n\n
      • Source Port: N\/A<\/li>\n\n\n\n
      • Destination IP: 10.2.1.111<\/li>\n\n\n\n
      • Destination Port: 5201<\/li>\n<\/ul>\n\n\n\n
        Here explain some of the confusions. In the VMware SD-WAN world, it is always the client side Edge, that is Left-Spoke-1 initiated the flow, and the server side Edge, that is Left-Hub-1, accept the flow. This is the reason when viewing the flow from Left-Hub-1, the business policy is “Refer Policy on Peer Edge Device”. That means the flow information is sort of “synchronize” from the Left-Spoke-1 to the Left-Hub-1, so this particular flow are with destination port 5201. And interestingly, the “Flows” tab (flow visibility) is always show “LAN to WAN” so it adjusted the source IP to 10.230.1.230 and destination IP to 10.2.1.111. Personally, this is quite confusing, I suggest VMware needs to at least add the flow is “locally initiated” or “peer initiated” in the “Flows” tab (flow visibility) to make the information more complete.<\/p>\n\n\n\n\n\n\n\n
        Test 2: iperf3 with simultaneous data flow<\/h3>\n\n\n\n
        The test2 is to have iperf3 with 5 data flows running in parallel, this is used to confirm is that multiple flow with different source ports but same source IP, destination IP, destination port are being combined.<\/p>\n\n\n\n
        The iperf3 command being used is:<\/p>\n\n\n\n
        \/usr\/local\/bin\/iperf3 -c 10.230.1.230 -w 128k -P 5 -b 5M -t 30<\/code><\/pre>\n\n\n\n
        The iperf3 client screen capture:<\/p>\n\n\n\n
        \"\"
        Diagram 14: iperf3 client command in test2<\/figcaption><\/figure>\n\n\n\n
        While the iperf3 is running, packet capture was done on the iperf client machine (10.2.1.111). The following is the conversation of the packet capture during test2:<\/p>\n\n\n\n
        \"\"
        Diagram 15: packet capture conversations at iperf client machine during test2<\/figcaption><\/figure>\n\n\n\n
        From the packet capture, there are 6 flows as follow:<\/p>\n\n\n\n
          \n
        • 10.2.1.111:35362 <—> 10.230.1.230:5201 (iperf3 control connection)<\/li>\n\n\n\n
        • 10.2.1.111:35368 <—> 10.230.1.230:5201<\/li>\n\n\n\n
        • 10.2.1.111:35378 <—> 10.230.1.230:5201<\/li>\n\n\n\n
        • 10.2.1.111:35390 <—> 10.230.1.230:5201<\/li>\n\n\n\n
        • 10.2.1.111:45072 <—> 10.230.1.230:5201<\/li>\n\n\n\n
        • 10.2.1.111:45082 <—> 10.230.1.230:5201<\/li>\n<\/ul>\n\n\n\n
          Let’s check the “Flows” tab (flow visibility) for Left-Spoke-1:<\/p>\n\n\n\n
          \"\"
          Diagram 16: Flows output generate by iperf3 in test2 in Left-Spoke-1 (left part)<\/figcaption><\/figure>\n\n\n\n
          \"\"
          Diagram 17: Flows output generate by iperf3 in test2 in Left-Spoke-1 (right part)<\/figcaption><\/figure>\n\n\n\n
          Again, the output is too width so the output is separated to two screen capture. Red square bracket also added to indicated the flow related to iperf3 generated under test2. The related flows only comes with two rows. It is because one row is for WAN link of GE3 while the other row is for WAN link of GE4. From this test2, we can confirm the source port is not in the consideration to listing as a separate flow in this flow visibility. That is there are in total of 6 flows, because the source IP, destination IP and destination port are the same, even the source port are different, these 6 flows are consolidated to two entries in this case due to there are two WAN links. If there is only single WAN link, we expect the flows will consolidate to a single entry.<\/p>\n\n\n\n\n\n\n\n
          Test 3: iperf3 running longer than 5-minutes<\/h3>\n\n\n\n
          In this test3, I would like to run a single iperf3 for, say, 12 minutes. The reason is this flow will spread across multiple 5-minutes interval. The expectation is the flow will be like “chopped” to multiple 5 minutes flow data. Let’s see.<\/p>\n\n\n\n
          The iperf3 command being used is:<\/p>\n\n\n\n
          \/usr\/local\/bin\/iperf3 -c 10.230.1.230 -w 128k -b 2M -t 720<\/code><\/pre>\n\n\n\n
          The iperf3 client screen capture:<\/p>\n\n\n\n
          \"\"
          Diagram 18: iperf3 client command in test3<\/figcaption><\/figure>\n\n\n\n
          While the iperf3 is running, packet capture was done on the iperf client machine (10.2.1.111). The following is the conversation of the packet capture during test3:<\/p>\n\n\n\n
          \"\"
          Diagram 19: packet capture conversations at iperf client machine during test3<\/figcaption><\/figure>\n\n\n\n
          From the packet capture, there are 2 flows as follow:<\/p>\n\n\n\n
          10.2.1.111:38682 <—> 10.230.1.230:5201 (iperf3 control connection)<\/p>\n\n\n\n
          10.2.1.111:38694 <—> 10.230.1.230:5201<\/p>\n\n\n\n
          Let’s check the “Flows” tab (flow visibility) for Left-Spoke-1:<\/p>\n\n\n\n
          \"\"
          Diagram 20: Flows output generate by iperf3 in test3 in Left-Spoke-1 (left part)<\/figcaption><\/figure>\n\n\n\n
          \"\"
          Diagram 21: Flows output generate by iperf3 in test3 in Left-Spoke-1 (right part)<\/figcaption><\/figure>\n\n\n\n
          From the packet capture the two iperf3 flows start at 09:43:57am and end at 09:55:57am. This 12 minutes duration cross the follow four 5-minutes interval (the flow 5-minutes interval):<\/p>\n\n\n\n
            \n
          • 09:39:58am to 09:33:58am<\/li>\n\n\n\n
          • 09:44:58am to 09:49:59am<\/li>\n\n\n\n
          • 09:49:59am to 09:54:59am<\/li>\n\n\n\n
          • 09:54:59am to 09:59:59am<\/li>\n<\/ul>\n\n\n\n
            That’s why from the “Flows” tab, the two flows actually generate 8 rows of data (spread across 4 interval and there are 2 WAN links, so 4×2 = 8).<\/p>\n\n\n\n\n\n\n\n
            Conclusion of Flow Visibility<\/h2>\n\n\n\n
            From the tests conducted above, here is the conclusion:<\/p>\n\n\n\n
              \n
            1. The “Flow Visibility row (let’s call them “Flows”)” consolidate the flows with a 5-minutes interval. That means the start time and end time of the “Flows” is not the flow creation and removal time, it is the particular interval the SD-WAN Edge consolidate the flow data.<\/li>\n\n\n\n
            2. The “Flows” are consolidate into a single row when:\n
                \n
              • The flows are having the same source IP address, destination IP address, destination port and utilizing the same WAN link interface. And these flows are happening within the same 5-minutes interval.<\/li>\n\n\n\n
              • Source port is not considered, different source ports can consolidate to the same “Flow Visibility row” as long as the above 4 parameters are the same.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
              • The “Flows” does not indicate it is initiated locally or it is initiated by the peer.<\/li>\n\n\n\n
              • When the “Flows” are initiated by the peer, the destination port is confusing because the destination port is actually from the peer’s point of view. For example, in the above test, in the hub site (Left-Hub-1) where the iperf3 server located, the “Flows” shows the destination port is 5201. This is because the destination port is extracted from the peer’s point of view (that is extracted from the spoke site (Left-Spoke-1)). Further to this, personally, we need enhancement from VMware to cater this problem.<\/li>\n<\/ol>\n\n\n\n
                Final note: I do not mean the Flow Visibility is bad, the Flow Visibility is a good improvement and can help in a lot of troubleshooting situations. Just it is needs to further polish.<\/p>\n\n\n\n
                End of post<\/mark><\/p>\n","protected":false},"excerpt":{"rendered":"
                Background In VMware SD-WAN version 5.1, a new feature called Flow Visibility is introduced. The following is the feature description in the 5.1 release note (https:\/\/docs.vmware.com\/en\/VMware-SASE\/5.1.0\/rn\/vmware-sase-510-release-notes\/index.html#What%20Is%20in%20The%20Release%20Notes-New%20SD-WAN%20Features): Flow Visibility In previous releases, the Orchestrator UI only displays aggregated flow information and statistics individually from the lens of Application, Source, or Destination and does not combine all this information on one […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-671","post","type-post","status-publish","format-standard","hentry","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/671","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=671"}],"version-history":[{"count":49,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/671\/revisions"}],"predecessor-version":[{"id":747,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/671\/revisions\/747"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=671"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=671"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=671"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}