{"id":574,"date":"2022-03-19T08:32:26","date_gmt":"2022-03-19T08:32:26","guid":{"rendered":"https:\/\/www.sdwan2.com\/?p=574"},"modified":"2022-03-25T06:18:43","modified_gmt":"2022-03-25T06:18:43","slug":"grafana-dashboard-for-sd-wan-gateway-vcg","status":"publish","type":"post","link":"https:\/\/www.sdwan2.com\/index.php\/2022\/03\/19\/grafana-dashboard-for-sd-wan-gateway-vcg\/","title":{"rendered":"Grafana Dashboard for SD-WAN Gateway (VCG)"},"content":{"rendered":"\n

Objective<\/h2>\n\n\n\n

A sample Grafana dashboard for VMware SD-WAN Gateway (VCG) is included in this post. Next, this post will explain how Telegraf, Prometheus and Grafana work together in the VCG monitoring. Finally, step-by-step guide is included at the end.<\/p>\n\n\n\n

Sample Grafana VCG Dashboard for Download<\/h2>\n\n\n\n

If you are only interested in getting the sample Grafana dashboard for VCG, here you go: <\/p>\n\n\n\n

VCG-Dashboard-2022Mar19-2<\/a>Download<\/a><\/div>\n\n\n\n

How Telegraf, Prometheus and Grafana work together in VCG Monitoring?<\/h2>\n\n\n\n

In version 4.3, VCG added the support of monitoring by Telegraf, with VCG specific counters (such as number of tunnels on the VCG). The official document is Monitor Gateways using Telegraf<\/a>. You can also find the complete list of supported counters in Supported Counters<\/a><\/p>\n\n\n\n

If you have no idea what are Telegraf, Prometheus and Grafana, I will use the following analogy. One of the classic tools to monitor network device is MRTG, that is there is a monitoring station running MRTG polls network device by SNMP to get metric such as throughput. The SNMP provides raw data and MRTG consolidate and then represent the data as graph (so the raw data become more human readable). In the “Telegraf, Prometheus and Grafana” scenario, you can consider Telegraf doing the job of SNMP, while Prometheus and Grafana doing the job of MRTG.<\/p>\n\n\n\n

The following block diagram shows how each components talk to each other (this is also the demo environment I am going to use in the step-by-step configuration):<\/p>\n\n\n\n

\"\"
Figure 1 – Block Diagram<\/figcaption><\/figure>\n\n\n\n\n\n\n\n

Enable Telegraf on VCG<\/h2>\n\n\n\n

You can find the official guide here Monitor Gateways using Telegraf<\/a><\/p>\n\n\n\n

The following is how I enable Telegraf on my demo environment.<\/p>\n\n\n\n

Step 1: Add the user telegraf to sudo group<\/h4>\n\n\n\n

This is because Telegraf needs to run \/opt\/vc\/bin\/dispcnt command in \/etc\/telegraf\/vcg_metrics.sh. The command to add user telegraf to sudo is<\/p>\n\n\n\n

sudo usermod -G sudo telegraf\n<\/code><\/pre>\n\n\n\n
Sample screen capture from my demo environment:<\/p>\n\n\n\n
\"\"
Figure 2 – Add user telegraf to sudo group<\/figcaption><\/figure><\/div>\n\n\n\n
Step 2: Edit the \/etc\/telegraf\/telegraf.conf<\/h4>\n\n\n\n
By default, telegraf.conf is not exporting any metric. To make Telegraf export the metric in Prometheus format, ucomment the three lines below “## Export metrics to prometheus”. The following screen capture shows the change being made in my demo environment:<\/p>\n\n\n\n
\"\"
Figure 3 – Uncomment three lines below “Export metrics to prometheus”<\/figcaption><\/figure><\/div>\n\n\n\n
The timeout for each command to complete needs to configured as 10s. In later release, the timeout is already configured as 10s, if this is the case, no change is required. If the timeout is 5s, uncomment and change the timeout value to 10s. The following screen capture shows the change being made in my demo environment:<\/p>\n\n\n\n
\"\"
Figure 4 – Uncomment and change the timeout value to 10s<\/figcaption><\/figure><\/div>\n\n\n\n
Step 3: Restart the Telegraf<\/h4>\n\n\n\n
After the configuration change in Telegraf, restart the Telegraf service to make the change effective. The command to restart the service is<\/p>\n\n\n\n
sudo systemctl restart telegraf\n\n<\/code><\/pre>\n\n\n\n
To check status of the telegraf service:<\/p>\n\n\n\n
sudo systemctl status telegraf\n<\/code><\/pre>\n\n\n\n
The following screen capture shows restart and checking the status of telegraf:<\/p>\n\n\n\n
\"\"
Figure 5 – restart and check telegraf status<\/figcaption><\/figure><\/div>\n\n\n\n
Step 4: Adjust the iptables<\/h4>\n\n\n\n
By default, Telegraf is listening on port 9273, waiting for the Prometheus client to visit. The VCG iptables by default block every source IP to access port 9273. Thus, the iptables rule need to modify to allow Prometheus client to connect to port 9273. The command is:<\/p>\n\n\n\n
sudo iptables -I INPUT -p tcp -m tcp --source <Prometheus Client IP><\/em> --dport 9273 -m comment --comment \"prometheus\" -j ACCEPT<\/code><\/pre>\n\n\n\n
In my demo environment, the Prometheus machine IP address is 192.168.252.26, the following screen capture shows the corresponding iptables command:<\/p>\n\n\n\n
\"\"
Figure 6 – adjust iptables rule to allow the Prometheus client to connect to Telegraf (port 9273)<\/figcaption><\/figure><\/div>\n\n\n\n
*Reminder: The iptables rule will not persist after reboot. You can edit \/etc\/iptables\/rules.v4 to include this new iptables rule to make the change survive after reboot (or use any other methods you can comfortable with)<\/em><\/p>\n\n\n\n
Step 5: Verification of Telegraf<\/h4>\n\n\n\n
To verify if the Telegraf is working or not, at the Prometheus client, issue this curl command:<\/p>\n\n\n\n
curl http:\/\/<VCG IP address><\/em>:9273\/metrics\n<\/code><\/pre>\n\n\n\n
The example screen capture shows Prometheus client get metrics from a VCG with IP address 24.11.0.54:<\/p>\n\n\n\n
\"\"
Figure 7 – Output of the curl request and response to Telegraf (only the first few lines are show as the output is very long)<\/figcaption><\/figure><\/div>\n\n\n\n
If everything is working, the curl will get response. And in the response there will are a list of metrics with prefix VCG_ like the above screen capture.<\/p>\n\n\n\n
It is worth to mention the Telegraf log file is at \/var\/log\/telegraf.conf. By looking into this log file, we notice the metrics are obtained every 60 seconds:<\/p>\n\n\n\n
\"\"
Figure 8 – Telegraf log file with Telegraf running on VCG<\/figcaption><\/figure><\/div>\n\n\n\n\n\n\n\n
Installation of Prometheus and Grafana<\/h2>\n\n\n\n
The installation is not the focus of this article. Refer to the official web site for the installation instruction:<\/p>\n\n\n\n
Installation of Prometheus: https:\/\/prometheus.io\/docs\/prometheus\/latest\/installation\/<\/a><\/p>\n\n\n\n
Installation of Grafana: \u200bhttps:\/\/grafana.com\/docs\/grafana\/latest\/installation\/debian\/<\/a><\/p>\n\n\n\n
Configure and Verify Prometheus<\/h2>\n\n\n\n
Step 1: Add new job at Prometheus configuration file<\/h4>\n\n\n\n
My demo system is Ubuntu 20.04, the Prometheus configuration file is at \/etc\/prometheus\/prometheus.yml<\/p>\n\n\n\n
To make the Prometheus collect metrics from the Telegraf agent in VCG, edit the configuration file by adding new job. In this example, a new job called “SF-VCG1” where the VCG IP address is 24.11.0.54. To do this, append the text below to the end of the Prometheus configuration file:<\/p>\n\n\n\n
- job_name: 'SF-VCG1' \u000b   \n  scrape_interval: 60s \u000b   \n  static_configs: \u000b      \n    - targets: ['24.11.0.54:9273']\n<\/code><\/pre>\n\n\n\n
If you need to monitor multiple VCGs, you can add multiple jobs.<\/p>\n\n\n\n
Step 2: Restart Prometheus<\/h4>\n\n\n\n
Prometheus needs to be restarted for the new configuration to take effect. To restart Prometheus:<\/p>\n\n\n\n
sudo systemctl restart prometheus\n<\/code><\/pre>\n\n\n\n
To check the Prometheus status:<\/p>\n\n\n\n
sudo systemctl status prometheus\n<\/code><\/pre>\n\n\n\n
Sample screen capture from my demo environment<\/p>\n\n\n\n
\"\"
Figure 9 – Restart Prometheus and checking the Prometheus status<\/figcaption><\/figure><\/div>\n\n\n\n
Step 3: Verify the Prometheus is able to get metrics from Telegraf at VCG<\/h4>\n\n\n\n
To verify is the setting is working or not, open a browser and then access port 9090 of the Prometheus. In my demo environment, I access http:\/\/192.168.252.26:9090:<\/p>\n\n\n\n
\"\"
Figure 10 – Access Prometheus web interface<\/figcaption><\/figure><\/div>\n\n\n\n
At the Prometheus web interface, click on the drop down menu “- insert metric at cursor -“, if there are metrics with prefix VCG_, the setup should be working. To confirm, pick one of those metrics and click “Execute” to check if the value is available or not. The following screen capture shows successfully query the value of the metric number_of_flows_value<\/p>\n\n\n\n
\"\"
Figure 11 – Get the value of metric number_of_flows_value<\/figcaption><\/figure><\/div>\n\n\n\n
This concludes the Prometheus is working properly.<\/p>\n\n\n\n\n\n\n\n
About the sample Grafana dashboard for VCG<\/h2>\n\n\n\n
Before jumping to how to configure the Grafana, I would like to point out couple things about sample dashboard provided in this post (the json you download from the first page)<\/p>\n\n\n\n
The supported counters is such a long list, I picked a few counters which I believe most people would like to have them available in the dashboard. The following are the counters included in the dashboard:<\/p>\n\n\n\n
- mem_available (Memory available)\n- Sum of all cpu_usage (Total CPU usage in percentage)\n- num_paths_PATH_STABLE, num_paths_UNSTABLE, num_paths_QUIET, num_paths_INITIAL-\n- number_of_edges<\/span>\n<\/span>- active_NAT_entries\n- number_of_flows\n- number_of_routes\n- tx_bytes and rx_bytes (Per Enterprise, represent as 5 minutes average throughout)\n- tx_packets and rx_packets (Per Enterprise, represent as 5 minutes average packet rate)<\/span><\/pre>\n\n\n\n
If any counters you want but it is not in the dashboard, free feel to add them by yourself. <\/p>\n\n\n\n
Secondly, the dashboard utilize variable to allow monitor multiple VCGs and each VCG is an individual job. At the top of the dashboard, there is an drop-down selection “Select VCG” to allow select one of the VCG in the Prometheus job list to read the corresponding metrics. If the panel is intended for “Per Enterprise”, there is another drop-down selection for you to select the corresponding Enterprise.<\/p>\n\n\n\n
Finally, if you find anything is not correct in the dashboard, please either leave a comment or ping me via linkedin, I will try my best to fix the dashboard. <\/p>\n\n\n\n
Configure Grafana<\/h2>\n\n\n\n
Step 1: Add data source<\/h4>\n\n\n\n
Open the browser, visit port 3000 of the Grafana to access the Grafana GUI.<\/p>\n\n\n\n
The following series of screen capture demonstrate how to add the Prometheus as the data source:<\/p>\n\n\n\n
\"\"
Figure 12 – Click configuration and click “Add data source”<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
Figure 13- Select Prometheus under Time series databases<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
Figure 14 – Configure a name for the data source. The URL is the Prometheus which configured to collect the VCG metrics<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
Figure 15 – Change the scrape interval to 60s and then click “Save & test”<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
Figure 16 – The expected result is get a green tick saying “Data source is working”<\/figcaption><\/figure><\/div>\n\n\n\n
At this point the data source is added successfully<\/p>\n\n\n\n
Step 2: Import the dashboard<\/h4>\n\n\n\n
The following screen captures demonstrate how to import the dashboard:<\/p>\n\n\n\n
\"\"
Figure 17 – Go to Dashboards and click import<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
Figure 18 – Upload the json template you downloaded from the first page<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
Figure 19 – Name the dashboard and select the data source<\/figcaption><\/figure><\/div>\n\n\n\n
At this point, the import of the dashboard is completed, and the dashboard is ready.<\/p>\n\n\n\n\n\n\n\n
The Dashboard and Panels<\/h2>\n\n\n\n
After the import of the dashboard, you should see a dashboard like the following (the dashboard is long, the screen capture only show the 3 graphs at the top).<\/p>\n\n\n\n
\"\"
Figure 20 – How the dashboard looks like after import<\/figcaption><\/figure><\/div>\n\n\n\n
Let’s have couple more samples of individual panel. The first one is the number of flows for a VCG called SF-VCG1<\/p>\n\n\n\n
\"\"
Figure 21 – Number of flows of VCG named SF-VCG1 (Last 3 hours)<\/figcaption><\/figure><\/div>\n\n\n\n
The second one is average throughput of a particular Enterprise<\/p>\n\n\n\n
\"\"
Figure 22 – Average throughput (per 5 mins) in VCG named SF-VCG1 for the Enterprise named Customer-T3 (Last 3 hours)<\/figcaption><\/figure><\/div>\n\n\n\n
–THE END–<\/p>\n","protected":false},"excerpt":{"rendered":"
Objective A sample Grafana dashboard for VMware SD-WAN Gateway (VCG) is included in this post. Next, this post will explain how Telegraf, Prometheus and Grafana work together in the VCG monitoring. Finally, step-by-step guide is included at the end. Sample Grafana VCG Dashboard for Download If you are only interested in getting the sample Grafana […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"zakra_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[9,5],"tags":[],"class_list":["post-574","post","type-post","status-publish","format-standard","hentry","category-partner-gateway","category-velocloud"],"_links":{"self":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/574","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/comments?post=574"}],"version-history":[{"count":36,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/574\/revisions"}],"predecessor-version":[{"id":651,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/posts\/574\/revisions\/651"}],"wp:attachment":[{"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/media?parent=574"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/categories?post=574"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sdwan2.com\/index.php\/wp-json\/wp\/v2\/tags?post=574"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}