How openstack DVR works
By Vikrant
May 7, 2017
In this article I am going to explain the DVR traffic flow. Distributed virtual routing (DVR) helps to reduce the number of hops through which packet need to traverse before reaching the final destination.
I have deployed DVR based setup using 1 controller and 2 compute nodes with the help of Red Hat openstack director. Templates used for deployment can be found at.
After deployment, created two internal and external networks. Connected each pair using neutron router.
neutron net-create internal1
neutron subnet-create --name internal1 internal1 10.10.20.0/24
neutron router-create router1
neutron router-interface-add router1 internal1
neutron router-gateway-set router1 external1
neutron net-create --provider:network_type vlan --provider:physical_network datacentre --provider:segmentation_id 10 --router:external True --name external1
neutron subnet-create --name external1 --gateway 10.11.48.254 --allocation-pool start=10.11.48.100,end=10.11.48.150 --disable-dhcp external1 10.11.48.0/24
In the default security group, rules are added to allow ssh and ICMP traffic.
neutron security-group-rule-create 08a7aa44-41c1-4701-8376-3a7b90f60736 --direction ingress --ethertype IPv4 --port-range-min 22 --port-range-max 22 --protocol tcp
neutron security-group-rule-create 08a7aa44-41c1-4701-8376-3a7b90f60736 --direction ingress --ethertype IPv4 --protocol icmp
Spawned two instances. Attached floating ip to one of the instance. Second instance contains only private IP address. i kept the scenario like this because I want to show two kind of traffic flows:
- N/S for instance without floating ip.
- N/S for instance with floating ip.
Common info for both scenarios.
As we want to do the packet capture at all the hops coming in the path hence we need to create OVS dummy ports to collect the capture from OVS bridges as explained in Red Hat solution.
Here is the sample for creating dummy port in br-tun ovs bridge. Same command by changing names can be used for br-ex and br-int.
ip link add name br-tun-snooper0 type dummy
ip link set dev br-tun-snooper0 up
ovs-vsctl add-port br-tun br-tun-snooper0
ovs-vsctl -- set Bridge br-tun mirrors=@m \
-- --id=@br-tun-snooper0 get Port br-tun-snooper0 \
-- --id=@br-tun get Port br-tun \
-- --id=@m create Mirror name=mymirror1 \
select-dst-port=@br-tun \
select-src-port=@br-tun \
output-port=@br-tun-snooper0 \
select_all=1
First, I will be covering N/S for instance without floating ip.
In this case my instance is running on compute-0. Following are instance details:
Instance name: test02 Private ip : 10.10.10.8 Mac address : fa:16:3e:f1:f8:e1
Compute Node:
- Let’s check the namespaces created on controller and compute node after spawning an instance without floating ip.
On controller node: SNAT is used to perform NAT operatioin for instances without floating ip.
[root@overcloud-controller-0 ~]# ip netns list
snat-d463a28e-e1f5-48f3-991a-6f90980490e1
qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1
qdhcp-d7554f84-c7f2-4a3b-9331-f9df22043d6a
On compute node: Default gateway of internal network is present in qrouter namespace.
[root@overcloud-controller-0 ~]# ip netns list
qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1
- After creating dummy interfaces for OVS bridges br-int and br-tun. I used following commands to capture the tcpdump on compute node on which instance is running.
tcpdump -s0 -i tap006715b9-82 -w /tmp/$(hostname)_tap006715b9-82.pcap &
tcpdump -s0 -i qvo006715b9-82 -w /tmp/$(hostname)_qvo006715b9-82.pcacp &
ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i qr-b6129458-2e -w /tmp/$(hostname)_qr-b6129458-2e.pcap &
tcpdump -s0 -i br-int-snooper0 -w /tmp/$(hostname)_br-int-snooper.pcap &
tcpdump -s0 -i br-tun-snooper0 -w /tmp/$(hostname)_br-tun-snooper.pcap &
tcpdump -s0 -i vlan50 -w /tmp/$(hostname)_vlan50.pcap &
Similary on controller node, after creating dummy interfaces for br-int, br-tun and br-ex, used following commands to capture tcpdump.
ip netns exec snat-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i sg-ff19cae5-3a -w /tmp/$(hostname)_sg-ff19cae5-3a.pcap &
ip netns exec snat-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i qg-ef46e09a-0b -w /tmp/$(hostname)_qg-ef46e09a-0b.pcap &
ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i qr-b6129458-2e -w /tmp/$(hostname)_qr-b6129458-2e.pcap &
tcpdump -s0 -i br-int-snooper0 -w /tmp/$(hostname)_br-int-snooper.pcap &
tcpdump -s0 -i br-ex-snooper0 -w /tmp/$(hostname)_br-ex-snooper.pcap &
tcpdump -s0 -i br-tun-snooper0 -w /tmp/$(hostname)_br-tun-snooper.pcap &
tcpdump -s0 -i vlan50 -w /tmp/$(hostname)_vlan50.pcap &
Try to ping 8.8.8.8 from inside the instance.
- As the instance doesn’t know the MAC address of 8.8.8.8 hence it’s taking default GW MAC address as destination MAC.
# tshark -tad -n -r overcloud-compute-0.localdomain_tap006715b9-82.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
22 fa:16:3e:c0:66:cd,fa:16:3e:f1:f8:e1 10.10.10.8 8.8.8.8
22 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5 8.8.8.8 10.10.10.8
- Traffic remains same on qvo interface of br-int bridge.
# tshark -tad -n -r overcloud-compute-0.localdomain_qvo006715b9-82.pcacp -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
22 fa:16:3e:c0:66:cd,fa:16:3e:f1:f8:e1 10.10.10.8 8.8.8.8
22 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5 8.8.8.8 10.10.10.8
- As packet is going from br-int to qr namespace and then coming back again into br-int namespace with destination MAC address hence we are seeing more information from br-int-snooper dummy interface packet capture.
# tshark -tad -n -r overcloud-compute-0.localdomain_br-int-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
22 fa:16:3e:42:5a:c5,fa:16:3e:c0:66:cd 10.10.10.8 8.8.8.8
22 fa:16:3e:c0:66:cd,fa:16:3e:f1:f8:e1 10.10.10.8 8.8.8.8
22 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5 8.8.8.8 10.10.10.8
This transformation of MAC address happens with the help of static MAC entry present in qrouter namespace present on compute node.
# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip neigh | grep '10.10.10.9'
10.10.10.9 dev qr-b6129458-2e lladdr fa:16:3e:42:5a:c5 PERMANENT
IP address 10.10.10.9 is present sg interface of SNAT namespace.
- From br-int packet goes into br-tun through patch cable and notice the different of MAC address. Instance source MAC address “fa:16:3e:f1:f8:e1” is replaced with “fa:16:3e:42:5a:c5” and destination MAC address is changed from “fa:16:3e:c0:66:cd” to “fa:16:3f:ec:d7:ad”.
# tshark -tad -n -r overcloud-compute-0.localdomain_br-tun-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
22 fa:16:3e:42:5a:c5,fa:16:3f:ec:d7:ad 10.10.10.8 8.8.8.8
22 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5 8.8.8.8 10.10.10.8
- On vlan interface, tunnel endpoint addresses are seen.
# tshark -tad -n -r overcloud-compute-0.localdomain_vlan50.pcap -p udp -T fields -e eth.addr -e ip.src -e ip.dst | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
20 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be 192.168.123.25 192.168.123.24
20 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1 192.168.123.24 192.168.123.25
Let’s check the ip address of vlan50 interface from both controller and compute node.
[root@overcloud-compute-0 ~]# ip a show vlan50
10: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 0a:d1:5b:87:f8:e1 brd ff:ff:ff:ff:ff:ff
inet 192.168.123.24/24 brd 192.168.123.255 scope global vlan50
valid_lft forever preferred_lft forever
inet6 fe80::8d1:5bff:fe87:f8e1/64 scope link
valid_lft forever preferred_lft forever
[root@overcloud-controller-0 tmp]# ip a show vlan50
8: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether e6:36:e1:cc:b7:be brd ff:ff:ff:ff:ff:ff
inet 192.168.123.25/24 brd 192.168.123.255 scope global vlan50
valid_lft forever preferred_lft forever
inet6 fe80::e436:e1ff:fecc:b7be/64 scope link
valid_lft forever preferred_lft forever
- Decoding the encapsulated vxlan traffic using vxlan decoder. Vxlan identifier it has taken is ‘93’.
# tshark -tad -n -r overcloud-compute-0.localdomain_vlan50.pcap -p -d udp.port==51929,vxlan -d udp.port==4789,vxlan -T fields -e eth.addr -e ip.src -e ip.dst -e vxlan.vni | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
1 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be
20 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be,fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5 192.168.123.25,8.8.8.8 192.168.123.24,10.10.10.8 93
1 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1
20 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1,fa:16:3e:42:5a:c5,fa:16:3f:ec:d7:ad 192.168.123.24,10.10.10.8 192.168.123.25,8.8.8.8 93
Controller node:
- Same vxlan traffic reflected on vlan50 interface of controller node.
# tshark -tad -n -r overcloud-controller-0.localdomain_vlan50.pcap -p udp -T fields -e eth.addr -e ip.src -e ip.dst | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
10 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be 192.168.123.25 192.168.123.24
10 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1 192.168.123.24 192.168.123.25
Again did the verification using vxlan decoder.
# tshark -tad -n -r overcloud-controller-0.localdomain_vlan50.pcap -p -d udp.port==51929,vxlan -d udp.port==4789,vxlan -T fields -e eth.addr -e ip.src -e ip.dst -e vxlan.vni | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
10 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be,fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5 192.168.123.25,8.8.8.8 192.168.123.24,10.10.10.8 93
10 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1,fa:16:3e:42:5a:c5,fa:16:3f:ec:d7:ad 192.168.123.24,10.10.10.8 192.168.123.25,8.8.8.8 93
- capture on br-int interface is showing translation of vlan ids from “1” to “2”. Along with translation of source ip address from “10.10.10.8” to “10.11.49.108”.
# tshark -tad -n -r overcloud-controller-0.localdomain_br-int-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst -e vlan.id| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
12 a6:4b:b0:6e:aa:54,fa:16:3e:24:fe:56 10.11.49.108 8.8.8.8 2
12 fa:16:3e:24:fe:56,a6:4b:b0:6e:aa:54 8.8.8.8 10.11.49.108 2
12 fa:16:3e:42:5a:c5,fa:16:3e:c0:66:cd 10.10.10.8 8.8.8.8 1
12 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5 8.8.8.8 10.10.10.8 1
This NAT of source ip address happened in snat namespace.
# tshark -tad -n -r overcloud-controller-0.localdomain_sg-ff19cae5-3a.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst -e vlan.id| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
12 fa:16:3e:42:5a:c5,fa:16:3e:c0:66:cd 10.10.10.8 8.8.8.8
12 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5 8.8.8.8 10.10.10.8
# tshark -tad -n -r overcloud-controller-0.localdomain_qg-ef46e09a-0b.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst -e vlan.id| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
12 a6:4b:b0:6e:aa:54,fa:16:3e:24:fe:56 10.11.49.108 8.8.8.8
12 fa:16:3e:24:fe:56,a6:4b:b0:6e:aa:54 8.8.8.8 10.11.49.108
- On br-ex, traffic reported to external world is reported. Noticed the changed vlan id to 11 from internal vlan id 2.
# tshark -tad -n -r overcloud-controller-0.localdomain_br-ex-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst -e vlan.id| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
12 a6:4b:b0:6e:aa:54,fa:16:3e:24:fe:56 10.11.49.108 8.8.8.8 11
12 fa:16:3e:24:fe:56,a6:4b:b0:6e:aa:54 8.8.8.8 10.11.49.108 11
All this magic of vxlan to vlan, and from internal vlan to external vlan happens with the help of flow rules.
# ovs-ofctl dump-flows br-int | grep mod_vlan_vid:2
cookie=0x84d7e40d9e25542b, duration=11259.279s, table=0, n_packets=755, n_bytes=73782, idle_age=4568, priority=3,in_port=1,dl_vlan=11 actions=mod_vlan_vid:2,NORMAL
# ovs-ofctl dump-flows br-ex | grep mod_vlan_vid:11
cookie=0x809f5e4c110018a5, duration=11288.680s, table=2, n_packets=747, n_bytes=70206, idle_age=4597, priority=4,in_port=3,dl_vlan=2 actions=mod_vlan_vid:11,NORMAL
# ovs-ofctl dump-flows br-tun | grep -i strip
cookie=0xaf99cd17f7d44000, duration=11363.093s, table=20, n_packets=18, n_bytes=1826, idle_age=10215, priority=2,dl_vlan=1,dl_dst=fa:16:3e:c4:f4:96 actions=strip_vlan,load:0x5d->NXM_NX_TUN_ID[],output:2
cookie=0xaf99cd17f7d44000, duration=9759.072s, table=20, n_packets=722, n_bytes=69895, idle_age=4703, priority=2,dl_vlan=1,dl_dst=fa:16:3e:f1:f8:e1 actions=strip_vlan,load:0x5d->NXM_NX_TUN_ID[],output:3
cookie=0xaf99cd17f7d44000, duration=10774.209s, table=22, n_packets=3, n_bytes=262, idle_age=10761, priority=1,dl_vlan=3 actions=strip_vlan,load:0x12->NXM_NX_TUN_ID[],output:2
cookie=0xaf99cd17f7d44000, duration=9759.382s, table=22, n_packets=1, n_bytes=42, idle_age=9617, priority=1,dl_vlan=1 actions=strip_vlan,load:0x5d->NXM_NX_TUN_ID[],output:2,output:3
It’s time for N/S for instance with floating ip.
- Spawned a new instance and attached floating ip with the instance. Network namespaces remains same on controller node but one new
fipnamespace is appeared on compute node.
[root@overcloud-compute-1 ~]# ip netns list
fip-27989c01-09f9-457a-a750-febd553473ba
qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1
- Let’s have a look what is available inside the namespaces.
[root@overcloud-compute-1 ~]# ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: fpr-d463a28e-e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether ca:6e:00:40:bc:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.106.115/31 scope global fpr-d463a28e-e
valid_lft forever preferred_lft forever
inet6 fe80::c86e:ff:fe40:bc0c/64 scope link
valid_lft forever preferred_lft forever
20: fg-cef4e483-b5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:e4:08:05 brd ff:ff:ff:ff:ff:ff
inet 10.11.49.109/24 brd 10.11.49.255 scope global fg-cef4e483-b5
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fee4:805/64 scope link
valid_lft forever preferred_lft forever
[root@overcloud-compute-1 ~]# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: rfp-d463a28e-e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 6e:cc:5e:a5:05:36 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.106.114/31 scope global rfp-d463a28e-e
valid_lft forever preferred_lft forever
inet6 fe80::6ccc:5eff:fea5:536/64 scope link
valid_lft forever preferred_lft forever
17: qr-b6129458-2e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:c0:66:cd brd ff:ff:ff:ff:ff:ff
inet 10.10.10.1/24 brd 10.10.10.255 scope global qr-b6129458-2e
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fec0:66cd/64 scope link
valid_lft forever preferred_lft forever
- As in this case traffic will not traverse through controller nodes hence all the captures are done on compute node.
ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i qr-b6129458-2e -w /tmp/$(hostname)_qr-b6129458-2e.pcap &
ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i rfp-d463a28e-e -w /tmp/$(hostname)_rfp-d463a28e-e.pcap &
ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba tcpdump -s0 -i fpr-d463a28e-e -w /tmp/$(hostname)_fpr-d463a28e-e.pcap &
ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba tcpdump -s0 -i fg-cef4e483-b5 -w /tmp/$(hostname)_fg-cef4e483-b5.pcap &
tcpdump -s0 -i tapda6c7254-26 -w /tmp/$(hostname)_tapda6c7254-26.pcap &
tcpdump -s0 -i qvoda6c7254-26 -w /tmp/$(hostname)_qvoda6c7254-26.pcacp &
tcpdump -s0 -i br-int-snooper0 -w /tmp/$(hostname)_br-int-snooper.pcap &
tcpdump -s0 -i br-ex-snooper0 -w /tmp/$(hostname)_br-ex-snooper.pcap &
tcpdump -s0 -i eth2 -w /tmp/$(hostname)_eth2.pcap &
- Traffic remains same on tap and qvo interface.
# tshark -tad -n -r overcloud-compute-1.localdomain_tapda6c7254-26.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
4 fa:16:3e:c0:66:cd,fa:16:3e:c4:f4:96 10.10.10.11 8.8.8.8
4 fa:16:3e:c4:f4:96,fa:16:3e:c0:66:cd 8.8.8.8 10.10.10.11
# tshark -tad -n -r overcloud-compute-1.localdomain_qvoda6c7254-26.pcacp -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
4 fa:16:3e:c0:66:cd,fa:16:3e:c4:f4:96 10.10.10.11 8.8.8.8
4 fa:16:3e:c4:f4:96,fa:16:3e:c0:66:cd 8.8.8.8 10.10.10.11
- Magic happened at br-int interface. Traffic is going from br-int to qrouter namespace and then from qrouter to fip namespace using patch cable which is joing fpr (fip namespace) and rfp (router namespace).
# tshark -tad -n -r overcloud-compute-1.localdomain_br-int-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
4 a6:4b:b0:6e:aa:54,fa:16:3e:e4:08:05 10.11.49.100 8.8.8.8
4 fa:16:3e:c0:66:cd,fa:16:3e:c4:f4:96 10.10.10.11 8.8.8.8
4 fa:16:3e:c4:f4:96,fa:16:3e:c0:66:cd 8.8.8.8 10.10.10.11
4 fa:16:3e:e4:08:05,a6:4b:b0:6e:aa:54 8.8.8.8 10.11.49.100
- Before packet go from qrouter to fip namespace. NAT happens on the packet to change the IP addresses.
# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 iptables -t nat -L | grep '10.11.49.100'
DNAT all -- anywhere 10.11.49.100 to:10.10.10.11
SNAT all -- 10.10.10.11 anywhere to:10.11.49.100
# tshark -tad -n -r overcloud-compute-1.localdomain_qr-b6129458-2e.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
4 fa:16:3e:c0:66:cd,fa:16:3e:c4:f4:96 10.10.10.11 8.8.8.8
4 fa:16:3e:c4:f4:96,fa:16:3e:c0:66:cd 8.8.8.8 10.10.10.11
qrouter namespace is not using default route table to send the packet to fip namespace this magic happens using non-default route tables.
# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip route
10.10.10.0/24 dev qr-b6129458-2e proto kernel scope link src 10.10.10.1
169.254.106.114/31 dev rfp-d463a28e-e proto kernel scope link src 169.254.106.114
# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
57481: from 10.10.10.11 lookup 16
168430081: from 10.10.10.1/24 lookup 168430081
# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip route list table 16
default via 169.254.106.115 dev rfp-d463a28e-e
# ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba ip route list table 2852022899
default via 10.11.49.254 dev fg-cef4e483-b5
# ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba ip neigh | grep '10.11.49.254'
10.11.49.254 dev fg-cef4e483-b5 lladdr a6:4b:b0:6e:aa:54 STALE
- External traffic is reported on other interfaces coming in path.
# tshark -tad -n -r overcloud-compute-1.localdomain_fg-cef4e483-b5.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst | sort | uniq -c
4 a6:4b:b0:6e:aa:54,fa:16:3e:e4:08:05 10.11.49.100 8.8.8.8
4 fa:16:3e:e4:08:05,a6:4b:b0:6e:aa:54 8.8.8.8 10.11.49.100
# tshark -tad -n -r overcloud-compute-1.localdomain_br-ex-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
4 a6:4b:b0:6e:aa:54,fa:16:3e:e4:08:05 10.11.49.100 8.8.8.8
4 fa:16:3e:e4:08:05,a6:4b:b0:6e:aa:54 8.8.8.8 10.11.49.100
# tshark -tad -n -r overcloud-compute-1.localdomain_eth2.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
4 a6:4b:b0:6e:aa:54,fa:16:3e:e4:08:05 10.11.49.100 8.8.8.8
4 fa:16:3e:e4:08:05,a6:4b:b0:6e:aa:54 8.8.8.8 10.11.49.100