How to use ovc-trace to troubleshoot logical flows
By Vikrant
September 20, 2017
In case of OVN, we are talking about logical flows, it’s difficult to go through the flows manually to understand the working, ovn developers understand this pain, and they have created utility ovn-trace to simulate the traffic flow for virtual packets.
ovn-trace can be used to trace both L2 and L3 behavior of traffic.
In case of L2, it requires inport, source MAC and destination MAC to trace the flow. In case of L3, it requires inport, source MAC, source ip address, source MAC and destination MAC to trace the flow.
I am using ovs-ovn version 2.7 which is having some issues with L3 flow tracing, I brought this issue on ovs-discuss list and came to know that these are known issues which are fixed in latest version of ovs-ovn i.e 2.8
Example : L2 logical flow trace.
To show this example, spawned two instances using same network but these instances are running on different compute nodes.
[root@controller ~(keystone_admin)]# nova list --fields name,status,host,networks
+--------------------------------------+---------------+--------+----------+--------------------------------------+
| ID | Name | Status | Host | Networks |
+--------------------------------------+---------------+--------+----------+--------------------------------------+
| 69736780-e0cc-46d4-a1f7-f0fac7e1cf54 | testinstance1 | ACTIVE | compute1 | internal1=10.10.10.4, 192.168.122.54 |
| 278b5a14-8ae6-4e91-870e-35f6230ed48a | testinstance2 | ACTIVE | compute2 | internal1=10.10.10.10 |
+--------------------------------------+---------------+--------+----------+--------------------------------------+
[root@controller ~(keystone_admin)]# ovn-nbctl show
switch 0d413d9c-7f23-4ace-9a8a-29817b3b33b5 (neutron-89113f8b-bc01-46b1-84fb-edd5d606879c)
port 397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e
addresses: ["router"]
port 84645ee6-8efa-435e-b93a-73cc173364ba
addresses: ["fa:16:3e:ef:50:3e 10.10.10.10"]
port 0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1
addresses: ["fa:16:3e:55:52:80 10.10.10.4"]
switch 1ec08997-0899-40d1-9b74-0a25ef476c00 (neutron-e411bbe8-e169-4268-b2bf-d5959d9d7260)
port provnet-e411bbe8-e169-4268-b2bf-d5959d9d7260
addresses: ["unknown"]
port b95e9ae7-5c91-4037-8d2c-660d4af00974
addresses: ["router"]
router 7418a4e7-abff-4af7-85f5-6eea2ede9bea (neutron-67dc2e78-e109-4dac-acce-b71b2c944dc1)
port lrp-b95e9ae7-5c91-4037-8d2c-660d4af00974
mac: "fa:16:3e:52:20:7c"
networks: ["192.168.122.50/24"]
port lrp-397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e
mac: "fa:16:3e:87:28:40"
networks: ["10.10.10.1/24"]
Tracing the traffic from “10.10.10.4” to “10.10.10.10”. I am using the MAC addresses of the port along with the inport information of the port.
Here input port is for the instance “10.10.10.4”. It’s showing the output port “0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1” automatically in egress output section. If you look closely it’s determining the destination port in ingress rule itself.
[root@controller ~(keystone_admin)]# ovn-trace 0d413d9c-7f23-4ace-9a8a-29817b3b33b5 'inport=="84645ee6-8efa-435e-b93a-73cc173364ba" && eth.src == fa:16:3e:ef:50:3e && eth.dst == fa:16:3e:55:52:80'
# reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:ef:50:3e,dl_dst=fa:16:3e:55:52:80,dl_type=0x0000
ingress(dp="neutron-89113f8b-bc01-46b1-84fb-edd5d606879c", inport="84645ee6-8efa-435e-b93a-73cc173364ba")
---------------------------------------------------------------------------------------------------------
0. ls_in_port_sec_l2 (ovn-northd.c:2979): inport == "84645ee6-8efa-435e-b93a-73cc173364ba" && eth.src == {fa:16:3e:ef:50:3e}, priority 50, uuid 1ebfeeab
next;
13. ls_in_l2_lkup (ovn-northd.c:3274): eth.dst == fa:16:3e:55:52:80, priority 50, uuid 8b07b8bf
outport = "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1";
output;
egress(dp="neutron-89113f8b-bc01-46b1-84fb-edd5d606879c", inport="84645ee6-8efa-435e-b93a-73cc173364ba", outport="0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1")
--------------------------------------------------------------------------------------------------------------------------------------------------------
8. ls_out_port_sec_l2 (ovn-northd.c:3399): outport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.dst == {fa:16:3e:55:52:80}, priority 50, uuid 09012b8e
output;
/* output to "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1", type "" */
Example : L3 logical flow trace
Created another network, attach that network to router as port, spawn new instance using that network.
[root@controller ~(keystone_admin)]# nova list --fields name,status,host,networks
+--------------------------------------+---------------+--------+----------+--------------------------------------+
| ID | Name | Status | Host | Networks |
+--------------------------------------+---------------+--------+----------+--------------------------------------+
| 69736780-e0cc-46d4-a1f7-f0fac7e1cf54 | testinstance1 | ACTIVE | compute1 | internal1=10.10.10.4, 192.168.122.54 |
| 278b5a14-8ae6-4e91-870e-35f6230ed48a | testinstance2 | ACTIVE | compute2 | internal1=10.10.10.10 |
| 8683b0c2-6685-4aff-9549-c69311b57238 | testinstance3 | ACTIVE | compute2 | internal2=10.10.11.4 |
+--------------------------------------+---------------+--------+----------+--------------------------------------+
[root@controller ~(keystone_admin)]# ovn-nbctl show
switch 0d413d9c-7f23-4ace-9a8a-29817b3b33b5 (neutron-89113f8b-bc01-46b1-84fb-edd5d606879c)
port 397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e
addresses: ["router"]
port 84645ee6-8efa-435e-b93a-73cc173364ba
addresses: ["fa:16:3e:ef:50:3e 10.10.10.10"]
port 0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1
addresses: ["fa:16:3e:55:52:80 10.10.10.4"]
switch f12c50d5-1dad-4e68-9c04-89d4732946a2 (neutron-7a2cf4c3-1476-4a86-8757-8102ec511362)
port 8db628d6-cf39-4166-bae6-715e71e5a6f5
addresses: ["router"]
port df575f1c-9282-4a94-a490-3e570ca02429
addresses: ["fa:16:3e:12:69:18 10.10.11.4"]
switch 1ec08997-0899-40d1-9b74-0a25ef476c00 (neutron-e411bbe8-e169-4268-b2bf-d5959d9d7260)
port provnet-e411bbe8-e169-4268-b2bf-d5959d9d7260
addresses: ["unknown"]
port b95e9ae7-5c91-4037-8d2c-660d4af00974
addresses: ["router"]
router 7418a4e7-abff-4af7-85f5-6eea2ede9bea (neutron-67dc2e78-e109-4dac-acce-b71b2c944dc1)
port lrp-b95e9ae7-5c91-4037-8d2c-660d4af00974
mac: "fa:16:3e:52:20:7c"
networks: ["192.168.122.50/24"]
port lrp-8db628d6-cf39-4166-bae6-715e71e5a6f5
mac: "fa:16:3e:27:66:8f"
networks: ["10.10.11.1/24"]
port lrp-397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e
mac: "fa:16:3e:87:28:40"
networks: ["10.10.10.1/24"]
Tracing the traffic from “10.10.11.4” and “10.10.10.4” both instances are running on different compute nodes. As I indicated earlier, version which I am using is having bug due to which it’s not able to trace the logical flow successfully.
[root@controller ~(keystone_admin)]# ovn-trace f12c50d5-1dad-4e68-9c04-89d4732946a2 'inport=="df575f1c-9282-4a94-a490-3e570ca02429" && eth.src == fa:16:3e:12:69:18 && ip4.src == 10.10.11.4 && eth.d st == fa:16:3e:55:52:80 && ip4.dst == 10.10.10.4 && ip.ttl == 32'
# ip,reg14=0x2,vlan_tci=0x0000,dl_src=fa:16:3e:12:69:18,dl_dst=fa:16:3e:55:52:80,nw_src=10.10.11.4,nw_dst=10.10.10.4,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=32
ingress(dp="neutron-7a2cf4c3-1476-4a86-8757-8102ec511362", inport="df575f1c-9282-4a94-a490-3e570ca02429")
---------------------------------------------------------------------------------------------------------
0. ls_in_port_sec_l2 (ovn-northd.c:2979): inport == "df575f1c-9282-4a94-a490-3e570ca02429" && eth.src == {fa:16:3e:12:69:18}, priority 50, uuid 0de9048a
next;
1. ls_in_port_sec_ip (ovn-northd.c:2113): inport == "df575f1c-9282-4a94-a490-3e570ca02429" && eth.src == fa:16:3e:12:69:18 && ip4.src == {10.10.11.4}, priority 90, uuid 8d1c26a6
next;
3. ls_in_pre_acl (ovn-northd.c:2397): ip, priority 100, uuid 1b768b1b
reg0[0] = 1;
next;
5. ls_in_pre_stateful (ovn-northd.c:2515): reg0[0] == 1, priority 100, uuid 39f7b20b
ct_next;
*** ct_* actions not implemented
This Bug is also applicable when both instances are running on same compute node like testinstance2 and testinstance3
[root@controller ~(keystone_admin)]# ovn-trace f12c50d5-1dad-4e68-9c04-89d4732946a2 'inport=="df575f1c-9282-4a94-a490-3e570ca02429" && eth.src == fa:16:3e:12:69:18 && ip4.src == 10.10.11.4 && eth.d st == fa:16:3e:ef:50:3e && ip4.dst == 10.10.10.10 && ip.ttl == 32'
# ip,reg14=0x2,vlan_tci=0x0000,dl_src=fa:16:3e:12:69:18,dl_dst=fa:16:3e:ef:50:3e,nw_src=10.10.11.4,nw_dst=10.10.10.10,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=32
ingress(dp="neutron-7a2cf4c3-1476-4a86-8757-8102ec511362", inport="df575f1c-9282-4a94-a490-3e570ca02429")
---------------------------------------------------------------------------------------------------------
0. ls_in_port_sec_l2 (ovn-northd.c:2979): inport == "df575f1c-9282-4a94-a490-3e570ca02429" && eth.src == {fa:16:3e:12:69:18}, priority 50, uuid 0de9048a
next;
1. ls_in_port_sec_ip (ovn-northd.c:2113): inport == "df575f1c-9282-4a94-a490-3e570ca02429" && eth.src == fa:16:3e:12:69:18 && ip4.src == {10.10.11.4}, priority 90, uuid 8d1c26a6
next;
3. ls_in_pre_acl (ovn-northd.c:2397): ip, priority 100, uuid 1b768b1b
reg0[0] = 1;
next;
5. ls_in_pre_stateful (ovn-northd.c:2515): reg0[0] == 1, priority 100, uuid 39f7b20b
ct_next;
*** ct_* actions not implemented